Harvard's legal defense against the affirmative action lawsuit is based around your characterization above being badly wrong.

Their defense is that if they use anything other than affirmative action to increase the number of minority students, academic excellence and other things that they care about will suffer. Oral arguments (search for "simulation D"): https://www.supremecourt.gov/oral_arguments/argument_transcripts/2022/20-1199_6537.pdf.

Or read the actual briefs, where they say that their projection for the non-affirmative action alternative: "Simulation D would require Harvard to admit 17% fewer students receiving the highest academic ratings." (https://www.supremecourt.gov/DocketPDF/20/20-1199/179362/20210517111311678_20-1199%20Brief%20in%20Opposition.pdf)

If there was lots of headroom like you describe, this simply wouldn't be true as-perceived-by-Harvard.

Indeed, Harvard's expert analysis reveals that the class analyzed for the purposes of the lawsuit had only 76% of students that had an academic excellence rating of 1 or 2. (https://projects.iq.harvard.edu/files/diverse-education/files/expert_report_-_2017-12-15_dr._david_card_expert_report_updated_confid_desigs_redacted.pdf, page 111-112, exhibit 38).

If you want to argue that Harvard's own standards of academic excellence are not meaningful, despite them arguing to the Supreme Court that they are, well, sure, have a go.

But they don't agree with you.

(They also do not reveal how much their academic excellence would be elevated if they did not use race, but the last time they revealed SAT scores categorized by race, there was about one standard deviation difference between races.)