Rex Kerr
4 min readMar 16, 2023

--

No, no, it's because the false positives (and negatives) keep getting found out, and the real ones keep getting built on, at the level of detail of individual experiments. The overall progress of technology is also very impressive, but it's the experiment-by-experiment level of detail that shows that it's working and why it's working.

As Hermes says, scientists try to build on each others' work in their own labs. This generally only happens when it's both non-obvious and makes a big difference. Otherwise, why even bother copying someone else?

But this means that it's hard to get confused about what's happening. For instance, if someone publishes a new high-temperature superconducting material, and you get or make some, it's not that hard to distinguish superconductivity from regular conductivity in most circumstances--and it's almost impossible to get accidental superconductivity from a material you don't even think you're testing. And of course this is one of the first things you'll do, because you either need a room-temperature semiconductor (what you're doing won't work otherwise), or you want to try to make a better high-temperature semiconductor by tweaking the first recipe.

Or, if someone publishes a set of four genes that de-differentiate cells (which is, normally, not supposed to happen: terminal differentiation is called "terminal" because cells can't go back from it) and allows them to divide or differentiate into a different type of cell, people go express those genes in their cells. And, again, it's not that hard to tell, say, a fibroblast and a pluripotent stem cell. There are all sorts of reasons why people might want to completely or partially dedifferentiate cells which rely completely on the method actually working, so if it's wrong (or has limitations), the wrongness or limits are found out.

The point is that experimental scientists want to use things. They want to use theories, they want to use processes, they want to use properties, they want to use objects--in every case, to exploit the critical novel feature of the whatever-it-is in order to gain more new insights.

In a regime where almost everything "works" (theory of childhood development, let's say), this is a very weak constraint. In a regime where almost nothing works (frequency mixing of light), this is an incredibly strong constraint. The big ideas are constantly being deployed in these almost-nothing-works regimes and end up implicitly tested along with the new novel finding, over and over and over.

In theory, it sounds like this process should take place. If you work in the hard sciences, you typically see it happen right in front of you--or you are doing it yourself!--over and over and over and over.

It's not entirely clear how one ought to demonstrate the effectiveness because to be definitive, one would need an even more reliable method that you could use to check the scientific method. But we don't have one. If we use internal checks--do we observe mistakes being corrected--then the answer is obviously yes, it works. One can use more sophisticated methods to analyze whether particular types of problems are likely to contribute to an overall mistake by the field (such as this one evaluating whether p-hacking leads to overall mistakes as might be revealed by a meta-analysis (which is typically what happens if something is important enough to warrant multiple papers, but the papers don't all agree): https://pubmed.ncbi.nlm.nih.gov/25768323/).

So overall, I think the evidence is about as much in agreement with my assertion as it could be, with the proviso that there haven't been that many tests of it.

Now, there are some important caveats. Sometimes people try to make important policy decisions on the basis of a handful of studies--in this case the crisis is a crisis in proportion to how important the policy decision is (and how much irreproducibility can be controlled by good experimental design, pre-registration, and the like). And sometimes people just find something, write it up, and nobody follows it up for a long time. It just sits there, untested, waiting to enlighten someone...or...maybe...to mislead them. That's not good. But it is only a waste of time and money once someone starts assuming the wrong result is right and tries to build upon it until it becomes clear that it really is wrong. It doesn't threaten the whole endeavor.

The social "sciences" are in much worse shape, however. The interpretation of results tends to be deeply theory-laden to begin with, and experiments often end up being not very definitive. Furthermore, a lot of people have the postmodernist / critical theorist mindset instead of a scientific one, which leads them even further away from a reliable mode of inquiry if they get a sniff of a power imbalance somewhere (for instance).

I don't think it's fair to just discard the social "sciences" as rubbish, but it is fair to worry that the replication crisis has a disproportionately large impact on them--and if you add that to a culture of methodology that is woefully ill-equipped to actually answer the questions being asked (for good reason: the problems are legitimately extraordinarily hard), a big dose of skepticism is warranted until and unless you actually go into the literature and examine the methods and the reproducibility.

(Aside: an excellent discussion of the critical importance of replication with bearings on the replication crisis is found in Siorksi and Andrioletti, 2003: https://link.springer.com/article/10.1007/s10699-023-09901-4. This seems pretty much on target to me.)

--

--

Rex Kerr
Rex Kerr

Written by Rex Kerr

One who rejoices when everything is made as simple as possible, but no simpler. Sayer of things that may be wrong, but not so bad that they're not even wrong.

Responses (1)