I think the individual and human-nature-based answer to the question that you give has better arguments for it than the opposition.

But there's an aspect of the "only whites can be racist" or "blacks can't be racist" viewpoint that you don't address. You address the issue of power, but there's a second issue of power that I think drives the attitude more strongly than does the explicit one.

Words have power.

For most people, words like "torture", "agony", "genocide"...maybe even "centipede"...bring up a flood of negative emotions, ready to tar any adjacent concepts with badness, justified or not. And of course if there is some justification or direct linkage, the emotional affect is all the greater.

Wouldn't it be great if you're advocating for some noble cause to have exclusive use of that kind of linguistic power? Maybe to fight other forms of privilege, the righteous should collect linguistic privilege for themselves?

Of course, this doesn't mean that the usage would be fair or an efficient use of language or good at establishing our common humanity or even would be helpful at improving the condition of those who are discriminated against.

But, man. Power. Can't deny the power.

So to me, whether intentional (led, perhaps, by scholars of Critical Discourse Analysis, where they pay attention to such things explicitly) or accidental (just the usual tribal feeling that "my side has all the good qualities and the other has all the bad ones", regardless of how tribe happens to be defined at the moment), that's what's mostly going on here.

As you explain, power plus prejudice is a particularly damaging combination. But that doesn't explain why a general-sounding word needs to be reserved for that exclusive situation. However, the addition of the idea of linguistic power (in this case, counter-power, I guess) at least motivates the opposite in a way that I don't think you fully considered.