Studies Show
Most Major Questions Have Evidence on Multiple Sides (Or They Wouldn’t Be Questions)
In 2022, seventy-six research teams were separately sent the same data on immigration and local support for social safety nets (to ascertain whether increased immigration changed attitudes on social welfare). They each developed their own models. About half of them found no result (no statistical effect). A quarter found a statistically significant positive result and a quarter found a statistically significant negative result.
In other words, given the same data, different researchers with different worldviews came to completely different conclusions despite using similar statistical tools. Said one political scientist, “this shakes my faith in social science profoundly, because… normally, when social research is done, one team does this and they pick their own data.”1 But when seventy-six teams were given “the same data, there was still an almost even split between positive and negative results.”
Were this the only sign of trouble for the social sciences, or for statistical analysis, or for the sciences more broadly, we might not need to worry. But the storied replication crisis (two-thirds of published studies in psychology were shown not to replicate) combined with the fact that on almost every important question in economics, psychology, sociology, nutrition, health,2 and even sports science,3 etc. studies can be found to support multiples sides really should shake our faith in “studies showing” the definitive truth.
In my day job, I try to help people improve their health and physical performance, so I try to stay up as best I can on the science of exercise and nutrition. For anyone who doesn’t pay attention to these, there are fierce debates about the best exercise protocols, and especially about nutrition. People who don’t pay much attention can easily be fooled, because advocates on different sides of these hotly-debated questions will cite peer-reviewed scientific research.