It makes as little sense as complaints about cultural appropriation in a general way always do...which is to say, very little. The great thing about culture is that we can blend and merge it so we all end up enriched. It's only a problem if it's blatantly exploitative.

However, your explanation that it's okay because "it's a wildly racially diverse universe" is complete rubbish. In Tolkien's work, lineage is very important for standing ("power", roughly), and tends to be meticulously worked through. You can't just drop people of random races in random spots because you have to explain who their parents were, and their parents, and so on. If you do that, then you're good. But it's absolutely not, "I see a dwarf, so who cares what anyone looks like?" unless you also go, "Yeah, we're just going to trash this whole part of Tolkien's worldbuilding."

I would expect people to be upset to see hobbits of the Shire growing pineapples and bananas, too. Not because people are fruitist, but because the setting doesn't make sense if you drop plants that don't do well in the supposed climate.

Maybe Amazon actually did the lineaging carefully, but when there was a hint of complaint they should have said, "And here, we thought this through!" as a first reply, not "If you don't like it, you're racist!"

If they made it make sense, and explain it, and people still don't like it, yeah. Racist.