Some notes on Truth and transparency in expertise research

I recently read Rachel A. Searston and colleagues’ Truth and transparency in expertise research.

Searston and colleagues warn that there may be many false positives in the expertise literature (as in the rest of the psychological literature). In particular, they note that expertise research often uses small samples and lacks random assignment. This makes any statistical inference challenging (especially in light of other known problems like QRPs and publication bias). Searston and colleagues go on to say that expertise researchers ought to address such problems by emulating other fields engaged in a credibility revolution. This usually includes improvements aimed at transparency and openness.

As to the credibility revolution, the authors explain how they have sought to embody it in their own work. For instance, they use tools like the Open Science Framework to collaborate closely with the experts they study. They also preregister their studies and use open workflows to allow for greater scrutiny of their work.

Overall, this paper provides important guidance for expertise researchers who want to make their work more credible and more in line with the expectations of modern (open) science.