Lies, Damned Epistemological Crisis, and Statistics

I began this post on January 28, 2011. It joined my queue of other begun and never completed posts which is at this point quite lengthy. I penned the title in a fit of inspiration which may have been more fit than inspiration, but there it is, and I’m not going to change it. It is taken, many will note, from the quote often attributed to Mark Twain but which, it seems, really originated from the British prime minister Benjamin Disraeli. He is reported to have remarked that there are three kinds of lies: Lies, damned lies, and statistics. I’m not sure what role statistics played as a shaper of human opinion and decision making in the 19th century, but he could not have imagined how influential his third category of lies would become in our own. And since this subject has been bouncing around my head for years, this post will be, apologetically, abnormally long.

What spawned the post was originally this article in the Atlantic Monthly with the curiously familiar title “Lies, Damned Lies, and Medical Science”. The article is a fascinating profile of a Greek medical researcher whose labor is aimed at debunking the claims of other medical researchers. He is not bitter nor one motivated by some high level rejection. Rather, his concern for the work of medicine drives him to hold researchers to a high standard of accuracy.

That he does NOT find that high level of accuracy in medical research disturbed me greatly. Researchers, like many of us, are measured by their results. Funding flows to promise. There exists an immense pressure upon researchers to demonstrate positive results in order to keep their positions and their funding. Such pressure can skew findings, can tilt the table so that we find what we are looking for. Hence, my epistemological crisis: whom do I believe? The one who says that eating eggs is bad for me or the one who says that it has no discernible effect in shortening my life? Do I believe the research that says at my age I should get a PSA test, or the one that says that this test has led to much unnecessary treatment?

So goes medicine, and so, sadly, goes religion. Churches have drunk deeply of the statistical Kool-aid in recent years. Recently, I’ve been approached by several people with stats in hand proving that the church is failing young people who are, supposedly, abandoning the church in droves. Some statistics become so dispersed that they attain something of an unquestionable canonical status. Is it not absolutely true that there is no discernible difference in the divorce rates between secular and Christian people? Common thinking, fed by certain popularized studies, says so. But is it true? No.

Years ago, wanting to not be left at the station as a pastor, I began to pay attention to the epicenter of contemporary evangelical statistical research: The Barna Group headed by George Barna. Barna’s name in evangelical culture is synonymous with polling data and his surveys are quoted widely with great authority. “Barna says…” is a powerful rhetorical weapon.

As I received my periodic reports from The Barna Group, I began to notice the disturbing trend that every report ended with something like this, “You can read more about this important study in George Barna’s new book….” Everything led to a book. (He has 28 of them on sale on his web site.)

And what sells books? Controversy and panic. Nearly everything he published had the air of alarm about it. The church was failing here; young people were being lost there; beliefs were eroding, people departing. I grow tired of the doomsayers.

I sense great similarities between the alarmists among us and the medical researchers desperate attempt to achieve publishable results.

Planned Parenthood needs to elevate the pro-life threat into a frightening frenzy to generate its support (I know – I was once on their mailing list, though I don’t know how). The same approach is adopted by Evangelical alarmist groups – be it Focus on the Family or the American Family Association or any number of other groups dependent upon fundraising. The greater the alarm, the greater the threat, the better the flow of money. And that disturbs me. So, I shut down and mistrust all alarmist rhetoric.

But that flows from my bias. I could never back up my resistance to Barna and other alarmists. Recently, though, some well placed Christian scholars have publicly taken issue with Barna and his methodology and results. Reflecting on one study in 2010, Calvin College philosophy professor Jamie Smith was quoted by Justin Taylor with this criticism of Barna:

This is not social scientific data that would ever pass muster in the scholarly field of sociology of religion (as represented, for instance, by work done in the Society for the Scientific Study of Religion). Indeed, I find it hard not to find this almost laughable in its methodological naivete and anecdotal nature.

Recently Taylor pointed out another public rebuff of Barna by Baylor University sociologists Rodney Stark and Byron Johnson in the Wall Street Journal.

As for media-hyped studies about religion, one should always beware of bad news bearers.

Stark loves the role of myth buster, and one could write this off as a bitter feud among those who get attention and those who don’t. But my experience tells me that it is quite easy to make statistics do what we want them to do. And the result is that the church becomes an alarmist place, with God’s people, serving the one in whom is all authority in heaven and earth, in a body against which the gates of hell will not prevail, cowering in fear and apprehension.

I agree with Stark and Johnson: Beware the bad news bearers. Check and double check all statistical claims. Use a source other than one whose work is used to sell books. And let us become more known for the good news we proclaim than the bad news we fear.