It's a decent point, but it is also the responsibility of those who want a fair and tolerant society to notice when bigotry is arising and to try to curtail it.

The impact can be larger if the target is socially vulnerable, but the corrosive effect on the thinking of the bigot (who isn't corrected) is similar, and among any large group there are going to be vulnerable people.

So although I think that a defensive "Not All Men! *gasp*!" is unhelpful and possibly counterproductive, I also think the usage should often be challenged in a principle way, namely: you are stereotyping, which leads you to think clumsily, make bad decisions, and be unjustly cruel to people.

Furthermore, adding extra antagonism between men and women is one of the stupidest things you could possibly do to make things better for women. Among those things that men are collectively good at, making their enemies miserable ranks very high.

Men and women are actually all on the same team. Adding tribal antagonism is just dumb.

Men and women should both be against men acting badly and should be able to call out those men who are acting badly. Failure to do this is itself acting badly (whether it's inability to call them out or inability to avoid calling every man out), and should itself be called out.