I don't think you have the history quite correct here, though I think you are more or less on target with the current state of affairs.

A lot of the patterns of behavior associated with the label "woke" (the actual behavior, not the misleading stereotypes offered by some so-called "anti-woke" individuals) are problematic and/or counterproductive. The outlook trends towards illiberalism, anti-humanism, and embrace of psychological fragility, which I think originally provoked direct opposition from many directions, including from the kindly and thoughtful who want people to be empowered to fulfill their potentials.

However, since then, I think the actual racists have flocked to the banner of "anti-woke" because even though they and the woke have a lot of similarity in their outlook, the woke flavor is usually morally superior to their flavor. By throwing their lot in with the enemy of their enemy and then corrupting the process, they see a path to victory (perhaps). Thus, rather than merely being a description of a rejection of a flawed perspective, "anti-woke" now has its own agenda driven by the people who most closely associate with it, which are, often, bigots of various flavors.

But this doesn't make the "woke" perspective right all along. It's still deeply problematic. It just means that opposition should not be structured as "anti-woke" because the perspectives that fall under that title are now, for the most part, even worse.