Tuesday, July 13, 2010

Jumping to Conclusions: Disease Burden and "Intelligence"


Last week, The Economist publicized some research by Christopher Eppig, a graduate student at the University of New Mexico. Eppig's research posits that the relationship between national disease burden and national IQ scores is causal, and that this decreased IQ level is thus a symptom, and not a cause, of underdevelopment in countries that lack health and hygiene infrastructures. The Economist summarizes how
Eppig and company reached their conclusion of causation (emphasis mine):
The correlation is about 67%, and the chance that it might have come about at random is less than one in 10,000. But correlation is not causation, so Mr Eppig and his colleagues tried to eliminate other possible explanations. Previous work has offered income, education, low levels of agricultural labour (which is replaced by more mentally stimulating jobs), climate (the challenge of surviving cold weather might provoke the evolution of intelligence) and even distance from humanity’s African homeland (novel environments could encourage greater intelligence) as explanations for national differences in IQ. However, all of these, except perhaps the last, are also likely to be linked to disease and, by careful statistical analysis, Mr Eppig and his colleagues show that all of them either disappear or are reduced to a small effect when the consequences of disease are taken into account.
Wow! They accounted for a lot of different stuff there! Do you think they accounted for every possible confounding factor? (Not possible!) It's also certain that the previous work they rely on included major ambiguities and assumptions in itself. And did you notice that the authors included distance from Africa as a cause for increased intelligence? It's like magic! Or really crappy analysis. I expected them to throw in craniometry as a potential confounding factor for good measure.

Now, a longstanding complaint of mine is that The Economist doesn't rigorously source articles, making it difficult for readers to go back to the studies and decide for themselves. Sometimes (oftentimes?) journalists don't understand a correct / responsible way to interpret data. So here's the full text of Eppig's article as published in Proceedings of the Royal Society.

First of all, I take issue with the scientific measurement of intelligence on such a broad, multi-cultural scale. Like love, joy, or pain, intelligence is something that is really difficult to define. And like, IQ tests, how bullshitty are those? Perhaps it is true that intelligence would vary on a national level - but I suspect that you would find even greater variance in definitions of intelligence across cultures. Even within our own culture, how would you describe a star mathematician, brilliant actor, and shrewd politician in a way that classified them all as equally intelligent.

Secondly, to develop these conclusions, Eppig combined the results of several different studies. This presents huge methodological issues which I don't even fully understand, but think of it like this: You have one set of data showing the price of apples in grocery stores in 30 of the 50 US states. Another set of data shows the average amount families spend on groceries in 40 of the states, with some degree of overlap in coverage. You combine the two data sets to show that apple prices are driving more than half of the variation in family grocery budgets! In my firm, we have to combine tons of data from various sources of differing levels of credibility and detail. Any of the analysts will tell you, that by the time you get to a final model, it's a pretty tenuous representation of reality. When assumptions build on assumptions to reach tentative conclusions, there's not much there there.

Thirdly, and not to be too much of a nitpicker, countries don't catch infectious diseases, people do (so use protection kids!) Wouldn't it be more meaningful to look at the effect of disease on individual's psychological abilities? Although it would still be difficult to ensure that those people who have diseases are comparable in all other ways to those who don't, this would get us closer to a reasonable conclusion.

...

This graphic from NewScientist perfectly encapsulates the study: "People born in a country with a high rate of infectious disease are likely to have a lower IQ." Even if they move? What about people who never get sick? Does treatment make a difference? Does the disease in question matter? (This graphic mentions tetanus and malaria - I was under the impression that these diseases had different effects on the human body, but maybe I'm crazy.) What about sub-geographies of low disease burden countries where there is a high rate of infectious disease? Would the results hold on a county level in the US? Like, seriously, WTF!

It was in looking for Eppig's current university post that I discovered he was a grad student currently managed by Dr Randy Thornhill, a zoologist with a research focus in evolutionary psychology (a field that itself gets pretty hocus-pocus, but let's not get too off-topic). Eppig's research interests are described as "sex steroids and dominance in men, concealment of estrus in women, estrus detection by men." As the great Dave Barry would say, I am not making this up. Boy would I like to hear what this guy has to say on gender interactions. I would bet the blog that it is complete bullshit. Does it count as "estrus detection" if my brother sees a tampon wrapper in the bathroom wastebasket? Why exactly is the nature of this researcher's fascination with menstruation, and why does it revolve around concealment and detection? It's like James Bond meets Are you there God? It's me Margaret. Also, if this is what Eppig studies, isn't he pretty far afield in publishing on infectious diseases and IQ distribution?

In conclusion, be a critical reader! And Economist, please, check your sources.

3 comments:

  1. it is annoying that epidemiology has like its own special ed. standards, where "large correlations" is more than enough to pass. I think that hinders the whole field.

    anyways, just wanted to add a note on Thornhill: He wrote the book where he explains rape using (as it seems only) his own research in insects: Rape is driven by a non-zero probability of immediate reproductive success and as such is a short term strategy and part of a deterministic human (male) nature. Interestingly, that would make human sexual behavior more related to that of instects than that of most other primates.
    Revisited (and crushed) by Emery Thompson in ch. 14 in Muller & Wrangham: Sexual coercion in primates and humans

    ReplyDelete
  2. I totally agree that this is "science" (see: how to detect bad science), but will offer a few slight crumbs of defense on his behalf:
    1) At least this argument isn't racist. The other people who used this intelligence data used it to say that the differences in IQ were the CAUSE of differences in development. I certainly find the reverse story more plausible.
    2) This research is meaningless, because there are worldwide correlations between LOTS of things, including the price of gas and the distance between Europe and the US. BUT, there is some actual causally identified research showing that poor nutrition and other problems of poverty can have cognitive consequences, such as this very well identified study by Erica Field regarding the relationship between iodine deficiency and school performance, or Douglas Almond's work on the flu pandemic and educational attainment.

    Now for the parts I don't like:
    1) It's a frickin correlation. Really, folks, are we still doing this?
    2) IQ tests were designed by people in developed countries (specifically white, male people) to test for things those same privileged individuals could do. There is no chance of comparability across different cultures, school systems, and languages. For example, East Asian languages requires the memorization of thousands of characters. This requires a different skill than putting a small number of characters together in different combinations to make words. IQ tests that test for memorization may score East Asians higher, while those that test for pattern recognition (or something, I don't know, I'm making this up) may score Westerners better. Surprisingly (sarcasm), the more developed countries publish more science and have more scientists working in developing countries, and so they get to determine the terms of the evaluation which they then deem the less developed (higher disease-burden) countries to score poorly on.

    In conclusion, there's lots of actual good science on the connection between health and cognitive functioning that The Economist could have given ink to, but instead they chose to look at a correlation based on demonstrably false assumptions that have been used to advance much more damaging theories.

    ReplyDelete
  3. I agree with what you're saying Coca Colo - there is probably plenty of good work to do on the neurological effects of specific diseases or kinds of diseases, especially with children. But that's not really what The Economist accomplished here. They primarily served as a platform for an unqualified researcher way out of his field of expertise, as well as promoted some simplistic and just plain inaccurate ways of building models and coming to policy decisions.

    Also, the weird interest in menstruation concealment further undermines the researcher's credibility to me. I guess we'll just have to wait and see what he publishes next :-S

    ReplyDelete

Commenting is now open, but we'd love it if you chose one username so other commenters can get to know you. To do this, select "Name/URL" in the "Comment as" drop down. Put the name you'd like others to see; the URL is optional.

Any profanity, bigotry, or synonyms for "[ ] sucks!" will be deleted. We welcome criticism as long as you're making a point!