Nicholas Kristof and the Ecological Fallacy

On several recent Sundays,  the New York Times Sunday Review section has carried Op-Ed columns by Nicholas Kristof that assert, in his own words, that "unconscious bias remains widespread in ways that systematically benefit both whites and men. So white men get a double dividend, a payoff from both racial and gender biases."  Titles for these columns include strings of monosyllabic words like: "When Whites Just Don't Get It" and "Straight Talk for White Men".  These columns elicit more opposition from Times Nation than the typical editorial in the Times, which Kristof curiously and patronizingly responds to by continuing to tell his readers that they "Just Don't Get It". 

The purpose of this post is to explain that it is Kristof who doesn't get "it", that he doesn't understand how to read the studies he quotes nor does he understand basic logic. Specifically, his argument suffers from the "ecological fallacy", a fallacy common among journalists who have a educational background which, while it may have resulted in high marks that lead them to think they are highly intelligent, has usually omitted a rigorous investigation of statistics and logic, which leads them to consistently over-state the significance of studies in social science. 

The ecological fallacy is simply this: attributing to individual members of a sample population a characteristic demonstrated by a statistical analysis of the population as a whole.   It is well recognized in social science (yet sadly often committed in discourse  grouped under the rubric of "social justice").  When committed in the context of race, gender, etc., it is often just an intellectual version of stereotyping.  I have no difficulty agreeing with Kristof that unconscious bias exists and affects some proportion of the decisions made in the population.  But it doesn't mean that everyone shares it or that every white male's station in life can be attributed to it. Those are fallacious statements, and they are why his white male readers react angrily to his generalizations about them.

Simple, even crude, examples will serve to illustrate the ecological fallacy:  blacks tend to make less than whites, so it must be true that any given black member of a professional sports team makes less than any given white player on that team.   Or, women tend to make less than men, so any given female CEO must make less than any given male CEO.  Both those inferences are fallacious - the differences  within the populations studied are erased in the statistical analysis of the population, even though the statistical analysis is itself true about the whole population. The flaw lies in the generalization that the statistic describes every member of the group equally  -- if the reasoning were true, one would not need statistical analysis, ironically.  The reasoning is simply fallacious. 

I will use the studies Kristof cites in his last column to show his unwitting embrace of this fallacy.

He first cites a study that he describes as " a huge interactive exploration of 14 million reviews on RateMyProfessors.com that recently suggested that male professors are disproportionately likely to be described as a “star” or “genius.” Female professors are disproportionately described as “nasty,” “ugly,” “bossy” or “disorganized.” 

The interactive site does not lend itself to being quoted because it is principally visual; you plug in a word and get a graph of how often it was used in describing male or female professors.   I tried a couple of words Kristof quoted and they showed a 2:1 or 7:3 ratio in favor of males.  As Kristof noted, one possible inference might be that the descriptions are in fact accurate, although he sort of laughs that possibility aside (which I think is cavalier given that females represent a slight majority of college students and the sample size is remarkably large).  We also don't know the total number of male and female professors that were being rated, their experience, etc.  But for this purpose, I will assume the exercise demonstrates the unconscious bias he suggests.  Still, there were 30-33% of the population that did not demonstrate pro-male bias.  That is at least 4.2 million out of 14 million who don't fit his claim.  It's the ecological fallacy to suggest that everyone who rated a professor was biased.  And it's the ecological fallacy to suggest that every male professor was unfairly overrated or that every female professor was unfairly underrated. 

Another point about the 30-35% figure bears noting as well.  If we assume that in an unbiased world the breakdown would be 50/50, i.e., each word would be used as often to describe a male as well as a female professor, then how much bias does a 70/30 split show? I suggest that the right answer is 35% (50% total female population - 32.5% females receiving equal or better treatment) / 50% total female population.  That is, you would expect females and males to receive 50% each by random allocation, so only you need to quantify the deviation from 50% (there is an argument that the denominator should be 100%, because the range of possible distributions ranges from 100/0 male to 0/100 female in which case the % of bias would be only 17.5%, but I don't need to take that view given I make my point with the more conservative denominator).  

Significantly, this means a solid majority -- 65% roughly -- of the 14 million student ratings exhibit no bias at all, thus supporting my point that it is fallacious to claim everyone has it or that every white male benefits from it. In this case, 35% of the population had a different result due to bias, that's material but not a majority, let alone something universal. To even call it pervasive is a stretch, with all due respect.

The second study he cites says: "researchers at North Carolina State conducted an experiment in which they asked students to rate teachers of an online course (the students never saw the teachers). To some of the students, a male teacher claimed to be female and vice versa. When students were taking the class from someone they believed to be male, they rated the teacher more highly. The very same teacher, when believed to be female, was rated significantly lower."

I could not access the study itself, only an article about it that Kristof links to.  So it is difficult to say anything conclusive about it (that did not stop Kristof, of course).  The article indicates the sample size is small (43 students, gender distribution unspecified) and it is very unclear how "same"  the instruction was in the several classes, thus raising doubts as to how comparable the cross-class results are. Finally, this is a single-blind study (the teachers knew what the test was testing) and there is no way of knowing if the teachers themselves acted in a way that biased the ratings.  The only specific data point that the article mentions is that, although homework was returned at the same time in each class, the male teachers were rated as having done so faster.  However, the respective ratings they quote are clearly averages (4.35 & 3.55) meaning that there could well have been individual raters who did not display a pro-male bias at all, which is my point again. It's fallacious to imply that all students gave a biased rating of their teacher.

The third study he cites (which by the way is about a dozen years old) "sent out fictitious résumés in response to help-wanted ads. Each résumé was given a name that either sounded stereotypically African-American or one that sounded white, but the résumés were otherwise basically the same. The study found that a résumé with a name like Emily or Greg received 50 percent more callbacks than the same résumé with a name like Lakisha or Jamal. Having a white-sounding name was as beneficial as eight years’ work experience."

All  of which is true, but two things should be noted, as the study itself does: first, that callbacks only represented 1/8  of the responses; 7/8 of the responses did not involve callbacks and there the resumes received equal treatment.  That is, the complete distribution of responses was:

87.5%: No discrimination

9%: Favored White Applicant

3.7%: Favored Black Applicant

See page 11 of the study.  So the extent of bias was much lower than implied.  Kristof gives no credit to the existence of procedures that generate an unbiased result and counteract bias thereby.

Secondly, and more to my point, even if the 9:3.7 ratio were applied to all outcomes, the amount of biased decisions would not constitute a majority of the decisions: the study sent 1 "white" resume for every "black" resume, so to quantify the amount of bias - even among just the callback pool and ignoring the much larger "no discrimination" pool  - I convert the percentages to raw numbers - say 900 and 370 -- determine how many callbacks each race would have received in an unbiased world (635), and then calculate how much deviation occurred from that number: (635-370) / 635, or roughly 42%.  Again a sizable number, but still less than half of all decisions.  So again, more than half of the sampled population showed no bias at all.  (Again, I am not contending the population exhibited no bias, or that it was insignificant; I am saying no inference may reasonably be drawn about either the population as a whole, whites as a whole, or any individual white person, either as applicant or resume recipient, from the proportion of bias detected in the sample). 

The last study he cites involves asking "professors to evaluate the summary of a supposed applicant for a post as laboratory manager, but, in some cases, the applicant was named John and in others Jennifer. Everything else was the same. “John” was rated an average of 4.0 on a 7-point scale for competence, “Jennifer” a 3.3. When asked to propose an annual starting salary for the applicant, the professors suggested on average a salary for “John” almost $4,000 higher than for “Jennifer.”

Following the link he provided led me to the study but I could not find the underlying data, just the averages Kristof reports.  So unfortunately there was no way to determine the number or percentage of respondents who did not display a pro-male bias.  (Otherwise that study is a randomized double-blind study and thus the best one in his arsenal for his argument; also an interesting fact about the study is that, even carried out to two decimal places, there was not the slightest shred of difference between the average responses of female and male respondents -- each applicant was ranked virtually the same by each gender).

But from what he cites, we can see that, properly understood, a majority of the sampled populations show no bias at all in favor of whites or males.  We can also see from the ratemyprofessors exercise and the last study that there is no evidence of males favoring males or whites favoring whites in his examples.  For both of these reasons, and another one I will give at the end, there is no logic when he goes on to universalize these results as follows:

"It’s not that we white men are intentionally doing anything wrong, but we do have a penchant for obliviousness about the way we are beneficiaries of systematic unfairness. Maybe that’s because in a race, it’s easy not to notice a tailwind, and white men often go through life with a tailwind, while women and people of color must push against a headwind."

Those claims about "white men" just reiterate the ecological fallacy; they are pure stereotyping. There are individual white men who experience greater headwinds and lesser tailwinds than some women and people of color, not all of whom must push against an equally large headwind.  Clearly if you believe the studies he cites, a majority of whites and males receive no tailwind at all from being white or male.  Looking at averages or other statistical measures obscures this fact - and that's where the ecological fallacy does its harm: if the average ranking of males or whites is n more favorable than that of females or blacks, it doesn't mean that each white or male got the same n  boost from being white or male. To infer that is to commit the ecological fallacy.  

If we put these insights all together, a legitimate inference might be that the majority of whites or males receive no privilege whatsoever, while a minority receive a considerable boost, and the average is just what it is.  

A final reason Kristof's generalizations about "we white men" are fallacious is fairly obvious but rarely recognized -- that every exceptionally successful white male has had to out-compete the rest of the white male population too.  If you recognize this, as well as the size of the white male population, it becomes clear that the most successful white males are the least likely to have benefited from being white or male, given that they out-competed the many other holders of these supposed privileges.  Conversely, the benefits of the supposed privileges probably are greater the closer the white male is to the median or average, because his unique accomplishments are smaller in proportion to the benefits he supposedly received.   Not to personalize this to much, but to illustrate the point,  I attended all-male institutions in both high school and college and finished at the top of the class in both.  The population that took the high school entrance test was all male, by definition and almost entirely white, and I had the highest score on it.  I took the allegedly white-male-biased SAT with hundreds of thousands of other white males and outscored 99% of them.  Then I attended a predominantly male elite law school and finished somewhere in the top third, entered a predominantly male class of associates at an elite law firm and made partner, which less than a quarter of all my white male competitors did.  I respectfully submit that. given the number of white males I out-competed, and the decades  over which I did so, the most reasonable inference is that, even in a beautifully unbiased world, I would have had a fairly similar level of success. 

These are the points that Kristof doesn't get, when he encounters pushback from readers in Times Nation.  It is not that he is wrong about the presence of unconscious bias in the world, but that he is wrong to portray it as universal, when it's not even a majority view (and may be a very small minority among his readers when you consider how left-leaning Times Nation is. relative to the population as a whole), and wrong to claim that every white male has benefited from it as much as he thinks they all have.  That is why his readers are mad, not because they're in denial about their bias, but because they're more likely than not right that they are in fact not biased.  The world, even the white male subset of it, is more diverse, and yet more fair, than the ecological fallacies on the Times OpEd page lead one to perceive.