Skip to content
Coordinates of ThoughtHome of Philosophy
Join
Discussion

How should a journalist judge a study in a field she knows nothing about?

Sana Rizvi Fellow
· 5 replies
Scientific methodAnalyticModern EuropeanFrancis BaconHelen LonginoKarl Popper#credibility#expertise#forensic-science#journalism
Follow

Last month, in a case I was covering, an expert witness gave evidence on a forensic technique I had never heard of, with total confidence and a long list of qualifications. The defence's expert said the technique was unreliable, with equal confidence and an equally long list. I had to write something by six o'clock.

I am not going to learn forensic science in an afternoon. But it seems to me philosophy of science should have something to offer people like me, who have to decide how much weight a claim deserves without being able to check it ourselves.

My own rough list so far:

  • Did the claim risk anything? Could the study have come out the other way?
  • Has anyone other than the inventor of the technique reproduced it?
  • Is the confidence of the expert matched by any measured error rate?
  • Who disagrees, and are they disagreeing about data or about interests?

I'd like to hear what others use, especially people who work inside a science and see how outsiders misjudge it.

LikeReply
Follow5 replies

Join the conversation: members reply here, and joining is free. Reading is always free.

Join free to reply Sign in

5 replies

  1. Hamza Qureshi

    Fellow

    Courts have tried to write your list down. The United States Supreme Court's Daubert decision in 1993 told judges to ask whether a technique can be and has been tested, whether it has been peer reviewed, what its known or potential error rate is, and whether it is generally accepted in the relevant field. Testability is Popper's influence showing through. It's not perfect, but it's a better checklist than a witness's CV.

    Helpful · 2
  2. Tsion Bekele

    Fellow

    From medicine: I ask whether the conclusion was stated before the data came in. If a study announced what it would measure in advance and then measured it, I trust it more than one that went looking for something interesting afterwards. Also, the size of the effect matters more than the word “significant”, which journalists love and which often means much less than it sounds.

    Helpful · 3
  3. Wanjiru Njenga

    Fellow

    Base rates. If a test is 99% accurate and the thing it detects is rare, most positives will still be false. Nearly every bad forensic or medical story I've read could have been avoided by asking how common the thing is before asking how good the test is.

  4. Femke Bosman

    Fellow

    I'd add a question about the community, not the study. Helen Longino argues that objectivity comes from communities with public venues for criticism, shared standards, real uptake of criticism, and no one's authority exempt from challenge. A forensic technique developed and checked only by the people who sell it fails every one of those, however sincere they are.

    Helpful · 1
  5. Sana Rizvi

    Fellow

    Thank you all. I went back and found that the technique in my case had been examined by a scientific review panel that said it lacked validation and had no established error rate. Femke's point explains why: the only people who had ever checked it were its own practitioners. Bacon would have called it an idol of the theatre, a system believed because it was taught. My six o'clock story was more cautious than it should have been.

    Helpful · 2