I don't think it's fair to be unworried about any children, but my main concern with educational curricula loosely inspired and motivated by CRT (curricula which are called "CRT" in the lingo of Rufo) is the effect it may be having on black children.

The reason I am worried is because of cognitive effects like "stereotype threat" (e.g. https://www.reuters.com/article/us-maths-girls-idUSN2242207920070524), which is a type of (psychological) priming. That subset of the curriculum that focuses on historical injustice risks triggering stereotype threat, and while I accept that it's not absent when such materials aren't taught, I am not sure we know whether it's the same.

Also, a lot of historical material tends to be very black/white heavy. But if you are actually emphasizing race, not just ignoring it, this also risks sending a message to Hispanic and Asian kids that they aren't important, which adds yet more potential trouble.

Curricula that cover the successes and triumphs of blacks and those of other (diverse) racial backgrounds seem to me tactically a much safer bet when it comes to education, and it's not totally clear to me that it's a win to emphasize the race of the person rather than just mention it in passing.

You may well be right that the majority of the outcry is about white children, and perhaps the people fussing couldn't care less what happens to anyone else. Sure sounds plausible to me. But that doesn't mean that the curricula under target (regardless of how it's labeled) is actually helpful. Is it empowering? Does it foster a growth mindset?

If we're going to advance the dialog beyond "these people say no!" and "those people say yes, and the first group is bad!", this is what we need to consider.