If you never interact with someone again, their intent is irrelevant to you.

Once there may be another interaction, intent is a highly pertinent signal regarding what you can expect future interactions to be like.

When you build a society of people it is full of actions over and over and over again, based on past actions, if not with the same people, at least potentially with equivalent people.

Anyway, I don't even think you believe your own argument.

For example, suppose you have a peanut allergy and go to a party, and the host hands you a sandwich. You take a bite, good! And then another...wait...is...that peanut dressing? MURDERER! THE HOST IS MURDERING YOU! You hurl the sandwich to the ground, snatch up a chair, bash them over the head, kick them in the stomach as they fall to be extra-sure they can't finish the deed, and flee, hoping you can get to your EpiPen in the car in time. Totally sensible reaction, right?

Sometimes intent doesn't matter. But mostly, saying so is just an excuse for acting like an asshole.