In Critical Race Theory, An Introduction, 3rd Ed. (Delgado and Stefancic, New York University Press, 2017), in the "Critical White Studies" section, they say

Many whites feel that [affirmative action] programs victimize them [...] The narrative behind this assumption characterizes whites as innocent [...] By contrast, many critical race theorists and social scientists hold that racism is pervasive, systemic, and deeply ingrained. If we take this perspective, then no white member of society seems quite so innocent. (Page 91.)

So I'm not sure this phantom is entirely a phantom. It could certainly be overstated, but it's not entirely imaginary if seminal figures in CRT admit a (weak) form of it.