Skip to content
Coordinates of ThoughtHome of Philosophy
Join
Debate

Thesis The replication crisis is a philosophical problem before it is a statistical one

Wanjiru Njenga Fellow
· 5 replies
Psychology & the mind sciencesAnalyticIan Hacking#constructs#measurement#operationalisation#replication-crisis
Follow

Its coordinates: 6.14 Philosophy of Psychology

Where it stands

I've followed the replication crisis in psychology closely. A large collaborative project published in 2015 tried to repeat about a hundred published studies, and fewer than half produced results as strong as the originals. Since then the standard diagnosis has been statistical: small samples, flexible analysis, publication bias, p-values treated as magic. The remedies follow: pre-registration, larger samples, sharing data.

I support every one of those remedies. My thesis is that they are not enough, because the deepest problem is conceptual.

Psychology measures constructs: "ego depletion", "grit", "implicit bias", "self-control". Each is operationalised as a test or a task. But often nobody has shown that the measure measures the construct rather than replacing it. If a construct does not correspond to anything stable in people, then even flawless statistics will find effects that shift from lab to lab, because what is being measured is not one thing.

Add a second problem physics never faces: participants read the news. A well-publicised effect changes the people it was found in, a version of what Ian Hacking called the looping effect.

So I claim the crisis is a crisis of concepts and kinds before it is a crisis of statistics. Better methods will reveal it; they will not cure it. Please argue with me.

LikeReply
Follow5 replies

Join the conversation: members reply here, and joining is free. Reading is always free.

Join free to reply Sign in

5 replies

For the thesis 2

  1. Pieter van Dijkhuizen

    For Fellow

    For. Machine learning has a close cousin of your problem: benchmarks. A model is tuned to a test set that was supposed to measure "understanding" and ends up measuring the test set. The statistics can be impeccable and the construct still empty. Goodhart's law is a philosophical point dressed as an economic one.

    Helpful · 2
  2. Elif Yıldız

    For Fellow

    For. We were taught to report a questionnaire's reliability and move on. Reliability only tells you the measure is consistent, not that it measures what its name says. Nobody in my course ever asked whether "self-esteem" is one thing.

Against the thesis 2

  1. Priya Raghavan

    Against Fellow

    Against, partly. Many famous failures need no philosophy at all: a sample of thirty students, an analysis chosen after seeing the data, and a journal that only prints successes will produce false positives about perfectly well-defined quantities. Fix that first. Only once the statistics are clean can we even see which constructs are shaky.

    Helpful · 1
  2. Emeka Anyaoha

    Against Fellow

    Against "philosophical", for "conceptual". In physics we got stable constructs because instruments forced them on us: a thermometer that disagrees with every other thermometer gets thrown out. Psychology needs better measurement theory, which is a methodological job. Calling it philosophy makes it sound as if the answer will come from an armchair.

    Helpful · 3

Other replies

  1. Wanjiru Njenga

    Fellow

    I'll accept both corrections. Priya, yes: statistics first, because noise hides everything. Emeka, fair: "conceptual and methodological" is better than "philosophical", and the work is done with instruments, not armchairs. My revised claim: the statistical reforms are necessary and will expose a second, conceptual problem that they cannot solve on their own. Elif's point about reliability is exactly that problem in one sentence.

    Helpful · 1