Okay, you make a pretty good argument for this.

But the same argument works for people of any race who live in an area where most violent crime is committed by people of one race: it's morally and epistemically justifiable to act frightened by the people who represent the greatest actual source of danger.

So, for instance, white people who live in wealthy neighborhoods next to impoverished black neighborhoods are being prejudiced but not bigoted (and therefore not racist) when they clutch their handbags, switch to the other side of the street, and so on, especially at night, when they see black men from that neighborhood. Because, you know, in situations like that the threat to life is statistically worse than the threat to life from white people for the typical black person.

If you want to call that "not racism"...I...guess that's fine?

Stated simply, I guess the rule is: pragmatically acting to stay safer based on observation of immutable characteristics associated statistically with danger is not bigotry, it's a type of prejudice that is morally and epistemically okay.

If, however, you disagree--these wealthy white folks, despite literally being in some danger, are still acting racist--then I think you've left out an important aspect of your argument that differentiates these two cases.