I can't fully agree with your thesis because I think that dialog prompted by ill-founded ideas can be a valuable way to spread knowledge and understanding. Furthermore, groups sometimes latch on to an idea that feels right but isn't fully justified, and the challenge to explain the idea can reveal the error. (However--do I really need to say this? Jewish culture perhaps more than any other already embraces this perspective.)
But I have noticed a quite shocking level of hypocrisy among some people who claim special privilege to speak on some topic while denying the equivalent privilege to Jewish people to speak about themselves, and that absolutely deserves to be called out.
And most of the time, the speaker displays no interest in listening or learning, at which point your criticism is valid in full: they are, most likely, just functioning to increase hatred. If there were equal levels of ignorant pro-Jewish and anti-Jewish speech, well, fine, but somehow it almost always seems dismissive and negative.
Still, I would amend the take-home slightly: those who can't be bothered to educate themselves before speaking at least should keep their ears and mind open. Those who can't should consider that the likely outcomes are only two: either to cause harm, or to be recognized a fool. Silence is better than either.