Beware the Outliers That Please

Kevin Drum finds an interesting study of studies:
The Washington Monthly

So far, this is unsurprising. Publication bias is a well-known and widely studied effect, and it would be surprising if G&M hadn't found evidence of it. But take a closer look at the graph. In particular, take a look at the two bars directly adjacent to the magic number of 1.96. That's kind of funny, isn't it? They should be roughly the same height, but they aren't even close. There are a lot of studies that just barely show significant results, and there are hardly any that fall just barely short of significance. There's a pretty obvious conclusion here, and it has nothing to do with publication bias: data is being massaged on wide scale. A lot of researchers who almost find significant results are fiddling with the data to get themselves just over the line into significance.

In any system where continued occupation depends on particular results, does it surprise anyone that those results appear to be found more often? It is a corollary of Sinclair Lewis' observation that it is very hard to convince a man of a truth when his livelihood depends on his believing in a lie. (Paraphrase)

This effect is less obvious, and yet stronger than malice in explaining the vast amount of contradictory scientific (so called) facts believed by the general populace.

--ml

Technorati Tags: science skepticism