Doesn't the biggest problem with accusations of racism come when they are made falsely?

Here, you're assuming that (1) the party is guilty and (2) the accusation enough is sufficient for them to adequately perceive this if they didn't already.

But that's not what discussions look like. People deny that there is any racism (either: that didn't happen, or that did happen but it's not racism). The validity of the accusation is what's usually in doubt.

Furthermore, in the rare cases where there might be reason to think that the accusation of racism is indeed as harmful or more harmful than the racism itself (e.g. if it was mild and calling it out triggers widespread stereotype threat, or ruins trust within a group that already had borderline psychological safety), the issue deserves a more thoughtful response than "learn and stop it".

I don't think you really answered your own point here, especially since the context ("everyone who identifies with AI Safety") is exactly where the validity of the accusation would be in question.