The high rates of incarceration must be a huge problem, but alas, I think that alone isn't enough to explain the effect. So even if we enact comprehensive criminal justice reform (and we should, immediately!), there will still be major problems.

I can't find good data on how many "black men always leave their partners", and so on. I can find data on children, though, so we can use that as a proxy. One quarter of black children have a parent who has ever been in jail, and "only" 10% have a parent currently in jail): https://www.epi.org/publication/mass-incarceration-and-childrens-outcomes/. The numbers are a little old, but as far as I know nothing has changed radically in the last few years.

Now, this is incredibly awful, and a sizable fraction of the incarcerations are unjust (and unequally applied, even in the cases where it would be just if it were applied equally, which it's not).

But 75% of white children (and 85% of Asian children) are in families with two married parents, while only 36% of black children are. If you add cohabitation, the gulf remains huge: 81% vs 44%. See, for instance, https://ifstudies.org/blog/the-majority-of-us-children-still-live-in-two-parent-families

If you take every single person out of prison, and assume that they will automatically become committed, present parents, that still only raises 44% to 54%. (The white numbers barely budge.) Even if you assume any jail time ever automatically makes it impossible to be part of a family, and you could change that so nobody went to jail ever, it still can't be the whole thing. (19% of white kids without two parents, 30% of black ones.)

So, yes, incarceration is very important, but it's not the whole story...it's probably not even the majority of the story. What's the rest? I haven't got any great evidence. Everything I could check didn't pan out as being clearly the whole rest of the story. Poverty or economic disadvantage? Urban vs rural residence? Culture? Government incentives? Absorption of negative stereotypes? Everything together reinforcing each other?

What you learned is part of the truth, but it isn't the whole thing. If the other factors can be identified, maybe there are ways to improve the situation there, too.

I completely agree with your point about not stereotyping a whole category of people because of a few bad apples...heck, even if it's mostly bad apples. Writing off everyone is no way to treat the people in the group who are wonderful and/or trying hard even if they're a minority in that group, let alone if they're actually the majority. It can only makes things worse--if someone gets pigeonholed and dismissed or denigrated, how long before they decide, "Why do I even bother trying?"

So I agree with your main point--I think it is your main point? To generalize slightly: let's elevate people who are doing well, not shoot them down in the careless crossfire of stereotyping. Hold accountable only those who have something to account for.