Intuitively, this makes sense. But then doesn't this logic work equally well:
"My people have inflicted racism firsthand for more than 400 years. We are the only experts on enacting racism that I recognize."
Conclusion: if a white person tells someone that they acting racist, white people are the experts, so you'd better believe them.
Isn't this argument equally compelling?
(If you say: "not all white people" so the intensity is different, I'll just go, fine, FINE, "White nationalists and supremacists...so you'd better believe them.")