So you're saying that this comment is unproblematic in some contexts, but in the context you're thinking of it is used to deny racism?

(With a corollary that your intended target audience is expected to understand the context and/or ask what you mean and/or stay confused?)