Oh, people complain plenty about legacy students, too. Look, you're doing it!

But while I don't know about Harvard specifically, people have looked at legacy student scores and found that, actually, the accepted legacy students don't differ appreciably from the general student body in SAT score at least. (See, for instance, https://hechingerreport.org/proof-points-why-elite-colleges-cant-give-up-legacy-admissions/)

Anyway, Asians have no problem with Black people taking slots instead of Asian people--not that I've ever heard. I've only heard a call for the same standards to be applied to everyone. Harvard is continually in lawsuits over this, and every time they say that if they change their policies, the number of black people admitted will decline but not go to zero. I don't know why you're making this an all-or-none thing about racial groups. Nobody is suggesting "none". Higher-achieving Black people would still take slots instead of Asian students, and everyone's fine with that.

(For instance, affirmative action is not permitted by state law in the University of California system. The UC system has tried every way to target disadvantage instead, and pretty much nobody complains that some traditionally disadvantaged group is taking their slots. It's perceived as adequately fair.)