In the socioeconomic assessment arena we may do slightly better than the medical research sphere. We typically conduct studies that have a bit larger samples (typical medical studies have 50 or less individuals) and we may not face the number of potential hypotheses that may be posed in the medical field. Yet we do not have proper treatment and control experiments as in the agronomic sciences which allows the researcher to isolate better the relationship between cause and effect. Thus, socioeconomic assessments are subject to many of the same forces, factors, research dynamics and outcomes as those faced by medical research.
In essence, Ioannidis paper is really a wake up call and a demand for ensuring more high quality research and standards to conduct such research. We cannot rely on perceptions, subjectivities and ill-defined anecdotes to guide policy and decision making. Granted, the later are important in helping frame the hypotheses and with thoroughly and carefully conducted qualitative research, can be quite important resources and knowledge to define policy and decision making. We have to be careful about an indiscriminate call for more research as a way to weed out poor quality research.
As Alex Tabarrok reminds us in the blog Marginal Revolution quality checks such as conducting meta analysis of the literature can help address several issues, yet it can also open new ones. In Tabarrok’s words:
Sadly, things get really bad when lots of researchers are chasing the same set of hypotheses. Indeed, the larger the number of researchers the more likely the average result is to be false! The easiest way to see this is to note that when we have lots of researchers every true hypothesis will be found to be true but eventually so will every false hypothesis. Thus, as the number of researchers increases, the probability that a given result is true goes to the probability in the population, in my example 200/1000 or 20 percent. A meta analysis will go some way to fixing the last problem so the point is not that knowledge declines with the number of researchers but rather that with lots of researchers every crackpot theory will have at least one scientific study that it can cite in it’s support.