about a new study indicating that racial bias plays a role in how the home plate umpire calls a baseball game. I only had aTime
magazine story to rely on in writing this report. But thanks to my intrepid editor, I now have access to
I raised a number of questions. Today, I have some answers.
My primary problem was I didn’t know how the pitches were evaluated. Were the authors looking at video from the centerfield camera? Or from behind home plate? It turns out that the authors weren’t looking at video. They went to ESPN.com and pulled up the play-by-play of each game from 2004-2006. This is the box score from the August 14, 2004 game between the Detroit Tigers and the Anaheim Angels, played in Anaheim. Pitching for the Tigers is Nate Robertson, a white pitcher. He throws 89 pitches in five innings, 52 for strikes. He surrenders nine hits, six earned runs, and two walks while striking out one batter. The Angels pitcher is Ramon Ortiz, a player of Hispanic origin. He pitches 4.2 innings while surrendering eight hits, six earned runs, and three walks while striking out two batters on 95 pitches, 55 of which are strikes. The Angels win the game 11-8. The home plate umpire is Jim Wolf, a white man.
This is the pitch-by-pitch play-by-play of that game. Note what it says for the top of the first inning: the first batter is Jason Smith. Ortiz throws a ball, a strike (looking), then a 5-3 groundout. The next batter is Carlos Guillen. Ortiz throws a ball, another ball, then Guillen doubles to center. Ivan Rodriguez is next. He sees a ball, another ball, then he fouls off a pitch before flying out to center. Rondell White bats clean-up and he gets a strike (looking), strike (swinging), then a ball, another ball, and the inning ends on a 6-3 groundout.
Notice what that pitch-by-pitch doesn’t say. It doesn’t say if that first pitch to Smith, a ball, was a curve that bounced in front of the plate, or whether it was a fastball that just missed nicking the plate. It doesn’t say if that strike which Smith looked at was a slider that caught the edge of the plate or a fast ball that caught the edge.
This play-by-play is missing a crucial element. How can I know, how can the authors know, if Wolf is treating Robertson and Ortiz the same? There’s no evidence. No detail. Yet, the authors used this ESPN pitch-by-pitch play-by-play for every game from the 2004 to 2006 seasons to tell me there is a racial bias among major league umpires.
There are lots of explanations and footnotes in the study telling me how they accounted for variables like stadium configuration, the distance of the seats from home plate, which team was the home team, the attendance, and whether Ques-Tec was used. I get percentages of white to black umpires, to white to black to Hispanic to Asian pitchers. They try to wipe out any statistical bias that could be generated by the non-random assignment of pitchers to face various opponents and of umpire rotation.
And I’m going to give the authors that. The numbers say what the numbers say. But as Mark Twain says, there are three kinds of lies: lies, damn lies, and statistics. nd I’m saying these numbers are statistics, so they’re not quite actually the truth.
I can’t tell you Jim Wolf is treating Robertson differently from Ortiz because I don’t know how he called the exact same pitch at the exact same spot on the outside corner of the plate from both pitchers. It doesn’t tell me if Wolf gave Robertson the low strike at the knees but didn’t do that for Ortiz. The pitch-by-pitch doesn’t tell me. And if it doesn’t tell me it doesn’t tell the authors of the study.
If the authors had collected video evidence of every pitch of every game of the 2004 to 2006 time period. And if they then watched every pitch of every game from this time period, and if they then charted the location of every pitch of every game from this time period, and if they’d told me that, based on this they could tell that Nate Robertson threw 14 pitches that nipped the outside corner and that 10 were called strikes, while Ramon Ortiz threw 14 pitches that nipped the outside corner, but only six were called for strikes, and further, if they could then tell me that based on this study they were able to determine that Jim Wolf called that that same pitch at that same location a strike for white pitchers more than he did for black, Hispanic and Asian pitchers, they might be on to something. And I might be willing to buy into the study.
But they didn’t. And I can’t.
I’ve got another problem with this study. It’s an academic paper, but it has yet to be published in academic journal. Which means that it hasn’t undergone the rigorous peer review process that would validate its conclusions. When it’s undergone that testing process, and other masters of the economics field have signed off on these conclusions, I might also be willing to buy into this study.
If you like this story, consider signing up for our email newsletters.
SHOW ME HOW
You have successfully signed up for your selected newsletter(s) - please keep an eye on your mailbox, we're movin' in!
As it is, we have an unpublished academic paper that Time magazine takes at face value. The author of the Time magazine piece did an awful job on this story. The obvious questions aren’t asked: like whether they watched the games and what others in the economics field thought of the study.
If I, a lowly blogger for the Houston Press can identify these problems, you’d think a big-time writer for Time magazine would too. But I guess you’d be wrong for thinking that. -- John Royal
P.S. Here’s a link to the actual study. Maybe I’m missing something. It just seems obvious to me that the location of the pitch would actually be important.
P.P.S. I’ve contacted one of the authors of the study about my concerns, but I’ve yet to hear back from him. I’ll let you know later about how badly I’ve gotten this wrong.