Yes to this, but no to the reason: this varies widely (sometimes it's power, sometimes insecurity, sometimes fervent belief, etc.).

And why do we object, anyway? The problem to me seems mostly to be a mismatch between different people's moral intuitions, and between reality and people's conceptions.

You pointed out some of the latter in your story, but the first, I think, is a bigger issue, despite being less obvious. For instance, if you have high-intensity moral intuitions about care/harm, but you don't have much of a moral intuition about freedom, then it makes perfect sense to be alarmed when you see a large age difference between people. There are, truthfully, ways in which it could be genuinely exploitative (whether the two people had passed our legal borders between "responsible adult" and not--we have to keep fully in mind that this is mostly a social construct, and not particularly well-founded in the neuroscience of maturing brains). So, one person's care/harm sensors go UH-OH! and their freedom sensors go "meh whatever"--and they object. Someone else's freedom sensors go WHOA! and their care/harm sensors go "yeah I don't see proof of harm here and hey might be good"--and they come to exactly the opposite conclusion.

In some cases, this mismatch of intuitions may mean that there is no mutually agreeable answer. In most cases, however, it means that to make progress you have to probe the foundational intuitions that are driving the position. Upon recognizing that they are intuitions (which are fast, but not necessarily that reliable), everyone gains a little extra capacity to consider whether the conclusion is actually well-founded.

I do agree that the "OMG age difference!" bigotry is expressed as bigotry most of the time, and it is illiberal, and it is not particularly different in nature from other bigotries. But you cannot assume that people are on board with liberalism: it must be constantly defended (or adjusted, if an adequate defense cannot be mounted).