It does get used this way.

But in denoting it as a "racist dog whistle", people also use it as a trump card to shut down discussion of whether a particular anti-racist proposal might be unfair or counterproductive.

So the poor phrase is in for quite a beating.

How, exactly, is it fair for people to tease a neurodivergent person for using a straightforward phrase to describe in plain language a natural consequence of their neurodivergence?

That's the problem with anti-dogwhistling. If a dog-whistle is done well, you can't tell when it's being used. So by attacking it, you're bound to attack multiple innocent people, until everyone gets the message that you don't say that, at which point it's not a dog-whistle at all but rather a globally recognized tribal identification marker.