Tuesday, 27 October 2009

Bad science, lazy reporting, and a side order of heteronormativity

Some articles, you can read the first paragraph, or maybe the headline, and get an instant sense that it bears no relation to the real story.

So, the BBC is reporting on a recent paper in the European Journal of Operational Research. First mistake: the paper is described as being by a Bath University team. Of the five authors, only one (Fragnière, the second author), is from Bath, so this seems to be stretching the definition a bit. The others are from various Swiss institutions. So, no prizes for guessing the source of the press release.

Second mistake: the BBC says:

The researchers studied interviews of more than 1,500 couples who were married or in a serious relationship.

Five years later, they followed up 1,000 of the couples to see which had lasted.

The paper itself says:

We use data from the 1999 Swiss contemporary family survey by Widmer et al., 2003. This is a representative random sample of 1534 couples, married or unmarried, ages 18–75, residing in the three linguistic areas of Switzerland. In 2004, the authors did a follow-up study on 1074 of those couples to determine how many had separated (Widmer et al., 2006 E.D. Widmer, J. Kellerhals and R. Levy, Types of conjugal interactions and conjugal conflict: A Longitudinal Assessment, European Sociological Review 20 (2006), pp. 63–77.Widmer et al., 2006).

Someone else did that research. They used that research as a basis for their own work. There's a huge difference there.

Both the paper and the BBC, unnaturally, say "couples" when they mean "male-female couples".

The BBC then says:

They found that if the wife was five or more years older than her husband, they were more than three times as likely to divorce than if they were the same age.

If the age gap is reversed, and the man is older than the woman, the odds of marital bliss are higher.

The latter "finding" isn't statistically significant, however.

Add in a better education for the woman - Beyonce has her high school diploma, unlike husband Jay-Z - and the chances of lasting happiness improve further.

The paper really doesn't imply this at all. It finds statistically a significant difference where marriages between people at different educational levels are more likely to last the full five years than marriages between people at the same educational level. The paper doesn't give the 95% significance range, only the value and whether this is significant at the 95% level, so while the five-year stability is increased more for marriages where the female partner is more educated than for those where the male partner is more educated, both are increased above the baseline.

Back to the BBC

Those who have never divorced fare better too. But couples in which one member has been through a divorce in the past are less stable than those in which both members are divorcees.

The only case that is statistically significantly different to the baseline case they chose (neither divorced) is where the male partner is formerly divorced but the female partner is not.

That's three of the four factors discussed. Here's the fourth, which is larger than the age one and doesn't get a mention. From the paper:

Wife and husband are both Swiss (REF)    -

Wife and husband are both from Western countries    3.451*

So, yes, if you want a stable marriage, the best approach is to be Swiss and marry someone else who is Swiss. British readers of BBC news are somewhat out of luck. (Note: this result may not generalise outside Switzerland, but then, the others might not either)

This, incidentally, has all been drawn from the first three chapters of the paper. The press release, and the BBC, seem to have stayed away from the fourth chapter onwards, probably because it then starts to get very strange. There's a slight hint of it in Fragnière's quote in the BBC article.

The researchers constructed a model where they'd take the demographics from the study, and their found weightings for the data, and then swap people around between couples to produce the set of couples with the lowest predicted divorce rate. That didn't make it into the news article, did it. I bet it didn't get as far as the press release before someone in Bath Uni's marketing department thought "this sounds a bit creepy".

To create our optimization model, we assumed a central matrimonial agency with knowledge of all the characteristics of all potential mates, whose mission is to optimize romantic matches.

Okay, maybe more than a bit.

The researchers acknowledge that this swap (99.81% of the couples were found to be "incorrect") is based on a whole bunch of limitations, and might not actually give the same result in reality (they don't suggest trying it, but they do suggest using these factors to give hints in matchmaking services)

There are several practical issues related to our model that we would like to address at this point. This study refers to married, or formerly married couples, and this does not take into account the odds of initially liking each other. In a real case of “marriage agency” the quality of the allocation must also include high probability of liking each other from the beginning. Moreover, in the real world the size of the men and women sets is different, which obviously is not represented in the model since our sample solely contained information about couples. These certainly represent limitations of our model

(emphasis mine)

A major limitation that they've not addressed is the sexuality of these couples. There are probably enough bisexual people within these couples that a straightforward swap of one man for another won't always be "optimal. Of course, that would complicate the mathematical model significantly, so it's not surprising that they didn't try it, but it is disappointing (and also not surprising) that they didn't notice and/or acknowledge this as a limitation.

It's not the first painfully heteronormative study around relationships, and it won't be the last.

So yes, a study of already dubious value, combined with some even worse reporting. Specifically, I think, "copying the press release without verifying it", since the New Zealand Herald, the Daily Mail, and the Telegraph all make similar mistakes (and all use one or both of the same two celebrity couples as examples). I wonder how many of the journalists bothered to use their news organisations subscriptions to check out the original paper (since none of them mentioned the last three chapters, I guess none)