Most people who are afraid of Critical Race Theory are are afraid of it because Christopher Rufo decided it was a sufficiently obscure and scary-sounding term that it was a good target for demonization (source: https://www.newyorker.com/news/annals-of-inquiry/how-a-conservative-activist-invented-the-conflict-over-critical-race-theory).
It's easy to demonize something that isn't well-understood. Unfortunately, it is also easy to gloss over the flaws of something that isn't well-understood when the "other side" attacks it. The enemy of your enemy is not necessarily a good friend.
Everyone should be concerned about the actual Critical Race Theory because it is an in-theory and in-practice regressive approach that aspires but fails to seek justice, due to, for instance, abandoning the ideal of fair and neutral laws.
The doctrine of the ordinaryness of racism is absolutely about targeting white people as benefiting materially or spiritually from maintaining racism. The "institution" that the ordinaryness-of-racism doctrine talks about is white people. It's disingenuous to say that it's only about institutions. I would like to know which expert claimed otherwise, and why they're claiming something that is refuted by reading either a few screens of Wikipedia or the first few pages of the most widely-used introductory text on the subject (written by important figures within CRT).
One might be able to justify making broad statements about white people on average (while accepting that the statements are not universal)--for example, it seems to be the case that a white supremacist mindset is more likely to be found among white than non-white people--but to deny that CRT makes such statements is simply dishonest.
Racism is a problem, and an informed population should know the history of the country. But CRT isn't about teaching an accurate account of history, and it absolutely does target white members of the country (whether justly or otherwise). The scholars who devised CRT were and are, for the most part, both highly intelligent and strongly driven to do good, so the intellectual and legal fruits of CRT contain insights into injustice and wisdom into some of the possible ways to proceed. However, these admirable personal qualities of its creators are insufficient to negate all the dangers of a divisive, pessimistic, and fundamentally unfair and evidence-ignoring framework.
So while CRT doesn't do, for the most part, what Rufo-brand criticisms say, what it does do is not a particularly advisable way to address historical injustices and build a society that values and supports all of its members.
We can do better than to follow the right-wing lies that have been spread to manipulate public sentiment, and we can also do better than to blindly adopt the tenets of CRT. We already had at our disposal all the tools that we needed to continue to reduce racism and other social ills; what we lack is the will and focus to deeply understand the problems and the wisdom and steadfastness to craft equitable solutions and implement them.