Attending a psychology conference this month, I was struck by an unsettling trend in social-science research. The abstract book revealed that nearly all of the 1,500 posters and talks related to some kind of confirmatory work — using statistical analyses to test an existing hypothesis. Hardly any reported the sort of open-ended exploratory research needed to come up with those hypotheses.
In my view, it’s crucial for the social sciences to embrace a research culture that balances hypothesis testing with hypothesis generation, embracing exploratory studies alongside confirmatory ones.
Replication games: how to make reproducibility research more systematic
To understand the difference between exploratory and confirmatory studies, consider a research group that is interested in links between children who watch violent films and aggressive behaviour. In an exploratory study, the group might ask parents about the children’s aggressive behaviour and collect information about household habits, individual characteristics and media content that could influence the children. The authors would inspect the data in many ways, looking for any patterns. The patterns that emerge, alongside theoretical background knowledge, could then be used to build specific predictions — for instance, that parents’ explanation of violent content might decrease media-induced violence in adolescents, but not younger children.
Confirmatory studies can seek to support or question these hypotheses using a fresh data set. To conduct a confirmatory study properly, best practice dictates that researchers outline their analysis plan before beginning their work, often through preregistration — recording it in a public repository. Preregistration helps to prevent researchers from constructing their hypotheses and analyses after peeking at their data. In the example above, for instance, if a group’s analyses do not confirm the general hypothesis, the researchers might be tempted to tweak the type of analysis they perform until it shows a statistically significant effect for, say, early adolescent boys, after controlling for family income. This broadly condemned practice opens the door to false-positive findings.
Like many in the social sciences, I was trained to think of hypothesis testing as the main job of a researcher, with exploration hardly worthy of publication. But I changed my mind as I began to lose faith in my own ability to preregister my studies properly. I wholeheartedly believe that preregistration is crucial to transparent science. But often, I would find that I hadn’t done enough exploration to plan my hypothesis-testing analyses properly.