I don't think it festers nearly as much as it seems. There are more than enough neo-Nazis throughout Europe...which is to say very few (zero is enough), but so very few that they're unheard of.

It's just that in the U.S., the left thinks, or at least says, that the right is comprised entirely of white supremacists because it "energizes the base" or something like that. If you actually poll the right, the overwhelming majority of them agree with views that are not remotely supremacist.

(By typical definitions. You can always redefine things so "white supremacist" is equivalent to "has implicit bias", and then declare everyone to have implicit bias because you can measure it in some people, and the rest must be a measurement error. (And it's true that the measurements are quite noisy.))