When theoretical physics is tested by experiment, it can be incredibly sound science.

When it isn't or can't be, it produces pretty mathematics, but of dubious connection to reality. Theoretical physicist in some areas (e.g. string theory) are not sciences--science adjacent, yes; almost-science, yes; but fundamentally missing the empirical grounding that makes the process work.

Career and status concerns impinge upon the quality of thought and speed of progress sometimes (c.f. replication crisis), but fundamentally do not allow experimentally tractable sciences to become not-sciences in any significant way. If somehow they do, it becomes incredibly obvious to any serious observer because answers are always "everyone knows that X" rather than "This is so cool! Look at this data!" (You do have to dig enough to get through an overlay of politics or capitalism or whatever--if you listen to the CDC, it's not always easy to tell which parts of what they say are well-grounded in science and which are grounded in political expediency or whatever else.)

For instance, if you ask whether nutrition is a science, the answer pretty quickly comes back not really in a lot of cases. Fundamental questions like "is this good for long-term health" are backed up by very poor data in many cases.

If the question is, "Can I only look for the label 'science' and then have a good idea about the reliability?" the answer is, well, not really...because...what gets called a "science" isn't constrained by experiment. We don't test fields to see if they're sciences and publish the results. If we did that, labeling them as sciences would be a science, and be highly reliable. But it's not. Oops. So the application of the label is a usual human affair and therefore kinda sloppy.

In particular, the social sciences aren't really sciences in practice a lot of the time, so as a caveat, the caveat is mostly that the term is something of a misnomer.