After the long winded title, we have to start somewhere with the discussion on the role of evidence.
I think that a good starting point is really understanding that we have to be quite vigilant about evidence and its robustness as the starting point in understanding the role it can (mis)play in supporting regulatory and decision making process.
Evidence can be used or abused depending in many cases on the intent of the proponent. Even when the researcher in good faith examines an issue, there is always the possibility that a specific data set or analysis can be wrong. See for example the often cited paper by John P. A. Ioannidis in PLoS Medicine “Why Most Published Research Findings Are False”.
The main message in this paper is not, by the way, that we should distrust science, but really that one specific study can be wrong and we should not drive too many conclusions from just one study, rather look at the body of evidence out there in support of a specific innovation or medicine for that matter.
For those of us who are involved in policy and socioeconomic research there are many traps and issues that we have to deal with as we do not usually have the luxury of a controlled experiment such as medical researchers do in practice. In some instances we will never have such studies (although there have been some attempts to conduct experimental approaches in the economics and social sciences). We can, however try to address some of these issues methodologically and through careful examination of issues.
The fact that we are dealing with human subjects opens a lot of discussions about the data leading us astray in many ways. See for example the blog post in Science-Based medicine for examples of ways that anecdotes can be impacted by human nature “The Role of Anecdotes in Science-Based Medicine”
More to come….