Quoting "Mark Twain" at me doesn't necessarily determine which of us would be brought down by arguing with the other, you know. (Aside: it wasn't Twain. https://marktwainstudies.com/the-apocryphal-twain-never-argue-with-stupid-people-they-will-drag-you-down-to-their-level-and-beat-you-with-experience/)
My point is that Harvard argues that there's an important distinction. Harvard argued so in their defense of using affirmative action.
Now, you might be arguing that Harvard is itself wrong. If I had access to Harvard's numbers, which I don't, because they don't make them public, then I'd be happy to argue that they're wrong if they're wrong, or that you're wrong if their data shows that instead.
If you can find data on graduating GPA vs. entering race and/or GPA, please share--otherwise you're every bit as much in the dark as I am, and your claim that the incoming students are not materially measurably objectively different in terms of outcome is even more speculative than my claim that it likely matters.
My (implicit) claim that it likely does is based on data that in general incoming scores predict outgoing ones (with incoming GPA generally the strongest predictor, e.g. http://patrickmalley.com/blog/2015/1/29/predicting-college-success is of many showing similar trends), and with Harvard accepting traditionally disadvantaged minorities with substantially lower SAT scores (https://www.thecrimson.com/article/2018/10/22/asian-american-admit-sat-scores/) and overall academic ratings (https://www.economist.com/graphic-detail/2019/10/04/does-harvard-accept-its-cleverest-applicants) than the average.
Are Harvard admissions are in the "we have piles and piles of equally good candidates; we're tripping all over them and can't tell them apart" regime? That they claim not (again, they may be wrong, but they're in a better position to know than you or I are, sans data) is evidenced by their arguments that they tried their best to remove race as an explicit factor, trying 13 different schemes to get an acceptably racially diverse class, and every single one unacceptably compromised the quality of the incoming class.
Now, maybe Harvard was arguing, "You know what, our incoming quality isn't actually related to academic performance. We just like to keep it higher-seeming than we need to so we can boast how awesome we are." Given Harvard's proclivities, this is...possible. And you could see why they wouldn't want to go around shouting about that.
Nonetheless, the relevant data to support your claim is absent. I don't have the data either and can't directly support the alternative. As such, we have to rely on less-conclusive suggestive evidence, which is exactly what I've been doing.
So, no, it's not a straw man at all. Harvard's arguments provide circumstantial but telling evidence about the broader point. I readily admit it's not conclusive, but being inconclusive doesn't automatically render something a straw man. It's the same question.