Are you sure this isn't just a Eurocentric view of history based in the reality that the exposure to people of different skin colors was pretty minimal until various countries in Europe become global maritime powers?

It would be pretty weird if everyone's surrounded by people with different noses, hair colors, languages, modest differences in skin color, etc etc and yet for some odd reason they were hyper-fixated on large differences in skin pigmentation.

But in places where that was a major distinguishing feature (e.g. Arab/Sub-Saharan Africa boundary), are we sure it wasn't a big deal?

Obviously it wouldn't look exactly like the European conception of race, but I'd imagine it would be clearly recognizable. "Those people with that color skin are (worse than us in some essential way)"?