I completely agree, but I think there's a gray area that you didn't point out that makes this even harder than it looks.

If we're willing to admit that allegiance to some groups is cause enough to roundly condemn an individual (e.g. the KKK), and we're willing to admit that allegiance to other groups doesn't warrant criticism simply for membership (e.g. American Humanists) even if we might find the attitudes of some members of the group problematic, we have to grapple with the issue that there are groups with a full spread of outlooks between one and the other.

If the defining characteristic of some group is based in a denial of some aspect reality, where this denial causes harmful real-world consequences, what exactly are we to assume about people who identify as part of that group? It isn't exactly bigotry to assume that they are likely--not certain, but likely--to hold the same problematic attitudes. The difference between that group and the KKK is one of degree, not of kind--unless we can define some sort of sufficiently discernable threshold that we can use to draw a qualitative distinction.

Unfortunately, denial of reality that reasonably can be expected to lead to real-world harm is absolutely mainstream now. For instance, it is difficult to be a Republican these days without endorsing the idea that the 2020 election was stolen (see, for instance, https://www.theatlantic.com/ideas/archive/2022/04/trump-voters-big-lie-stolen-election/629572/). But that incorrect idea is profoundly harmful. Not as harmful as rounding up the "bad guys" and beating or killing them, but nonetheless it isn't an attitude that can persist for long without losing our ability to function as a democracy. (This bill failed, but points out one direction the danger can come from: https://www.azleg.gov/legtext/55leg/1R/bills/HB2720P.pdf section 16-650 B.)

And you've already pointed out some of the dehumanizing rhetoric--the same type used to justify actual genocides!--that is becoming increasingly hard to distinguish from the core belief of some parts of the left. If you are part of a group that explicitly endorses dehumanizing those you disagree with, then is it actually bigotry for people to assume that you support dehumanization, and that you are supporting a trend that leads towards the worst kind of atrocities?

So...yes...what you say. Except we've gone far enough down a really problematic path that I think we need not just that but also a big heaping dose of tolerance for bad ideas that are expressed but not acted upon, in the hope that we can walk the bad ideas back and allow people to redeem themselves by deploying their good sense and compassion even if when they are called to demonstrate their loyalty to their chosen group, they display neither.