It's possible to bias math tests not racially but culturally, by using unexplained cultural references that are critical for solving the problem. For instance, if you're talking about "off sides" or "under par" or "forty-love", and you don't explain what those things mean, if there are correlations between race and the culture of knowing about the respective sports, you will be mixing up the mathematical proficiency with cultural knowledge.

However, people are pretty aware of the issue these days, so it's not really a major concern. (You can often detect problems like this by examining the average spread between culture-rich and culture-free scores.)