It goes both ways, though! Throughout recent history, girls have been discouraged from doing most things that are considered "important" or "impressive".
So while there is some dismissal of traditionally feminine things as unimportant because they are feminine, it's also the case that unimportant things have been more available as accepted interests for women than for men.
If you look at classical music, it used to be that women weren't allowed to perform, and the very best men gained a lot of praise for their skill. Nowadays, the very best everyone gains a lot of praise for their skill. Hilary Hahn is a spectacular violinist, and is widely recognized and celebrated as such.
Almost nobody says, "oh, don't play violin, that's for men".
But if you look at things like sports vs fashion, the comparison isn't really fair because in the former case the entire point is to develop skill to an extreme degree and show off the best of the best in a way where you can tell that they're the best of the best. Even if you have fashion shows, that level of "wow, this is the best humanity has to offer" is much more debatable. (The peak intensity is also a lot lower, which on average grabs attention less. Team sports also hit the tribal-existential-relevance button, which also evokes a lot of attention.)
Yes, sports don't accomplish much. But they pull on human attention in a different way than, say, fashion, and they have a much more deeply supportable quality metric.
Women are allowed to like fashion that applies to themselves. Men are allowed to like fashion that applies to cars or the lawn. Both are allowed to like fashion that involves making themselves look better via fitness. It's all the same type of thing, though: for the most part it doesn't amount to very much beyond associating oneself with a more pleasing aesthetic. (Fitness does have health benefits, but our society spends almost no time acknowledging this beyond lip-service.)
Yes, fashion doesn't accomplish much. But because it also is a more everyday thing--no peak human, no tribal unity--it isn't reasonable to expect the intensity of focus for it that sports get.
I don't think you can point to a disparity like this and say: well, fashion doesn't get enough respect because people associate it with women. When men are allowed to care about fashion it doesn't get as much respect either. This seems more a case to me of historical attitudes being "oh, men, you have better things to think about," but "oh, women, don't bother your pretty little heads about anything but being pretty little heads".
Men get made fun of for caring too much about their lawns. It's not because it's "girly". It's because it's self-interested, pedestrian, and ultimately relatively inconsequential. Women get a lot less grief for caring about their hair--and rightly so, since how we present ourselves to each other is a lot more important for social interactions than how our lawns present to each other, and we are a very social species. But the lack of credibility comes from the nature of the endeavor.
That isn't to say that there aren't cases where there's an unfair disparity. There are: for instance baby showers tend to be a girl thing, and superbowl parties tend to be a guy thing. What's more important, supporting a new baby, or watching highly-paid guys throwing an oblong ball around and crashing into each other? And yet we don't have an appropriate balance of reverence for the former and mirthful dismissal of the latter save as entertainment. How many guys are all, "Yeah, I'll skip the superbowl and take the kids to their activities so you can attend the baby shower"? But society's attitude should be towards a guy who won't do that should be, far more than it is, "What! You're so superficial!"
When it comes to fine dining, even though everyone is doing it because good food tastes nice, there does seem to be a whiff of "he deserves it" but "she's just indulging herself".
And of course people should be able to like whatever colors they like. (I'm not sure guys are even allowed to like colors?)
Anyway, the point is, if historical culture has pushed women towards interests considered "less important", and women have picked them up, you can't conclude that nowadays people consider them "less important" because they're associated now with women; it might be a more fundamental difference than that, and women are fitting in to a culture built on top of a lot of "oh, leave the important stuff to men" that hasn't had time to work itself out yet.