Some notes on Preregistering qualitative research

Structured observation

I just read Tamarinde Haven and Dr. Leonie Van Grootel’s article, Preregistering qualitative research. We’ve been thinking a lot lately about research credibility in legal scholarship (see the last blog post) and lots of legal scholarship happens to use qualitative methods. So, Haven and Van Grootel’s article naturally seemed quite useful. In this short post, I will talk about how their article indeed helped inform my thinking about how legal researchers could use preregistration, but it also raised a lot of questions and did not quite support its conclusions in some places.

The distinctive qualitative landscape

The desirability of preregistration has been thoroughly discussed when it comes to quantitative research (and especially with respect to null hypothesis significance testing; see Nosek et al, 2018). And both quantitative and qualitative research rely on data collection. Haven and Van Grootel write:

Qualitative research aims to answer the “how,” “why,” and “what” questions of a phenomenon (Green and Thorogood 2014). Qualitative research often uses language as its data, be it written or oral, although it may use photos, videos, or other types of behavioral recordings. The qualitative data are often collected via an interview, a focus group (structured group discussion), or via observation. Qualitative research tries to reveal the perspectives of the subjects or patients that the research question regards. It uses an “emergent design,” referring to the iterative process of combining data analysis, preliminary data inspection, and data collection.

One difference, however, is that postdiction is a feature rather than a bug in qualitative research. In Grounded Theory (where observation leads to theory and further observation), researchers are meant to rely on data to develop hypotheses.

The logic of Grounded Theory of and other qualitative methods requires a a second difference. Those methods require greater flexibility that could be abused in null hypotheses significance testing:

The researcher has the freedom to engage in a cyclic process of data collection and data analysis. The number of participants in the sample is not fixed beforehand: if necessary, the researcher can choose to sample new participants and go back into the field when saturation has not been reached yet. In addition, the researcher needs room to adjust her data collection instruments during the process if the diversity in the sample requires this. All in all, to achieve the full potential of postdiction and qualitative research, the qualitative research design requires large yet careful flexibility on the part of the researcher.

A third difference is that there is typically greater subjectivity in qualitative research practices. In fact, researchers are meant to bring a certain perspective to the interpretation of their data.

Resistance to preregistration of qualitative research

Haven and Van Grootel then suggest that the three above features of qualitative research (postdiction, flexibility, and subjectivity) are also three sources of resistance to preregistration. The postdictive nature of qualitative research makes it difficult to find a priori hypotheses to preregister. Preregistration may be seen to inhibit flexibility. And, finally, preregistration often seems to involve objective standards that are irrelevant to qualitative research.

Rebutting the resistance

The authors then address those resistances: (1) study designs and plans can be registered (i.e., it doesn’t have to be a hypothesis); (2) flexibility is not inhibited because it is completely acceptable to change approaches and methods as you go - preregistrations just make that more clear; and (3) there is no hard and fast rule that a preregistration must contain objective measures.

Does preregistration enhance the credibility of qualitative research?

This is where I had some reservations with the Haven and Van Grootel’s argument. They say:

Likewise, by preregistering qualitative research, it enables other researchers to assess whether the researcher used the right collection methods, the right data analysis methods, as well as whether the interpretation based on the data is convincing. If that is the case (right methods/ convincing interpretation), the qualitative study is more credible. Credibility is not an undebated term in qualitative research and here we follow Eisner’s (1991) interpretation of credibility when he states, “We seek a confluence of evidence that breeds credibility, that allows us to feel confident about our observations, interpretations and conclusions” (110). Ideally, this would lead to “an agreement among competent others that the description, evaluation and thematic … are right” (112).

All of that is fine, but I wonder why all of that can’t be explained in the report itself. In other words, “ the right collection methods, the right data analysis methods, as well as whether the interpretation based on the data is convincing“ need not be in the preregistration.

I think the practical benefits the authors list are more reasonable. They suggest that preregistration will help combat publication bias, making research projects more findable despite the results. And they say that preregistration creates incentive to be more careful in reporting everything that was done and in reporting all changes made to protocols.

Conclusion (and application to law)

Legal scholars regularly analyse cases other archival material in flexible and subjective ways. While, as Haven and Van Grootel say, this may seem anathema to preregistration, it is not. Predicted steps, approaches, and expectations can be preregistered in a living document. This will make the work easier to find and for others to better evaluate and perhaps emulate.

Stray thoughts

The authors say early on: “Preregistration is a measure recently introduced to reduce research misbehavior and improve reproducibility in quantitative research.“ I wonder if they are talking about social science because preregistration has been used since the late 1980s in clinical medical research.

Without getting into postdiction violating the principles of null hypothesis significance testing (NHST), the authors say that when presenting unanticipated findings as predicted, “postdiction is abused and a shady form of postdiction is presented as predictions“. I get that this is shady, but if we aren’t living in a NHST world (where its assumptions would be violated), does this make the results more likely to be wrong? Or is it just misleading?