Indeed we should, but where is the right point to intervene?

Are you arguing that it is incorrect that ""it’s the right thing to do” is going to ring very flat to the Asian college freshmen that got rejected by all their dream schools even though their grades, test scores, and extracurriculars were well above the average admitted student"? You don't even provide any argument against the position. And I don't think you can, as long as college is considered to be academic-merit-based.

If you want to argue that college admission shouldn't be based on academic merit, and we should rework the messaging and reality from top to bottom, sure, go for it. Harvard sure won't like it, because their primary argument for why they had to use race as a criterion was that if they didn't, academic excellence would suffer (but were very coy about stating how much academic excellence would be improved if they stopped using race). But one could try to make the argument. Heck, one might even make a different argument for STEM and non-STEM fields, because STEM fields are extremely heavily reality-based: it doesn't matter why you have so much harder of a time keeping units straight than someone else; if you don't get it right people are going to die, or your probe is going to crash, or your circuit will melt, etc. etc.. There's a lot more leeway in Rhetoric and Mesoamerican History and so on.

The problem with the argument that we need to redress historical wrongs is that (1) it quite directly penalizes people who had literally nothing to do with in (and, indeed, may have themselves been on the receiving end of historical harms!), and (2) it neglects to consider where we need to act to redress historical wrongs.

There's also the questionable assumption that it even helps the people it's supposed to, as opposed to pairing people with the educational environment in which they're most likely to excel. Caltech isn't "better". Caltech is harder. It's really frickin' hard. If you're ready for the challenge, great! If not, though, you'd probably be a lot better off tackling a challenge you are ready for.

Furthermore, the difference in ability at some top schools is enough so that anyone who has any racial awareness at all is going to notice that "black students = dumb students". These days, since color-blindness is supposedly racist, that's going to mean that basically everyone gets the message. Even though this is, at that point, entirely a selection effect, is that really the implicit training that we want to be giving to people? (Heck, for that matter, "Asian students = smart students" also seems pretty questionable as a training set to give people.) But this is unavoidably what you do when you have significantly unequal entrance requirements.

But most fundamentally, prerequisites aren't just there for fun. They're there because many subjects are deep. They require a certain level of expertise before you can make further progress. If colleges know what they're doing with admissions at all (and, honestly, there's an argument that they don't, and another that they don't want to because other things are more profitable), then if some groups are underrepresented at college and there is no evidence of active discrimination in admissions, then the most obvious place to intervene is before admissions so that the people applying have appropriate prerequisite skills. For example, special programs in high school to prepare people for college, with admission to those on the basis of need-to-redress-historical-wrongs given adequate apparent potential for the program to be worth a try. Not enough apparent potential? Figure out the prerequisites, and intervene earlier.

To me, affirmative action in practice has mostly, especially recently, worked out to be a fundamental abrogation of our responsibilities as a society to provide an enriching upbringing for all our children, trying to paper over that abject failure by getting the "right answer" even though the setup was all wrong, and then patting ourselves on the back and telling ourselves how wonderful we are for supporting equity instead of shamefacedly admitting that we first mucked up badly in giving opportunities for our children to develop their potential and then were blatantly unfair to make it look like we hadn't.

There might be some special cases where late intervention is needed and the best way to go, even beyond taking individual hardships into account (which is still allowed after SFFA vs Harvard). But overall it looks like a bad solution, and if a solution that bad is the best available, it needs very much better arguments to that effect than I've yet seen.