The other way around is: it is good to have clear, simple words to describe common types of behavior, especially when it's harmful, and personal bigotry on the basis of race is sufficiently common, clearly defined, and harmful to be worthy of consideration of the term. "racial prejudice" = "racism" is pretty clear. It's an efficient way to use language, and has the advantage of having been widely understood for decades. (It does not exclude the cases where the prejudice is backed by power, it simply doesn't require power.)

If you strictly adhere to a definition that is gated by power, as I explained, it's hard to know where the boundaries are. That makes it a poor choice for a brief word or phrase, because it often needs to come with a long explanation about power dynamics.

If you just say, "well, white people have had a lot of power, so we'll just use it as a race-specific term now", and apply negative connotations, while forbidding the use of the term and the negative connotations to anyone else, regardless of action...guess what! Then using the term would be an application of racial prejudice (because a critical part of it is based on the race of the person independent of what they do or who they are). As hilariously ironic as it would be if the term "racist" was to be used as a racist slur--a racially prejudicial slur, sorry!--I think that irony does not adequately justify its usage in this regard.