HHP: Hard Data on Ball/Strike Calls - How Good/Bad are the Umpires

[skanar]

OK, Frobby's post on ball/strike calls got me thinking, and when I start thinking and want to procrastinate from actual work I research baseball. So I went and got the Pitch/FX data from every game the Orioles have played this year, and did some work.

I looked at the starters for the Orioles and their opponents. Using the Pitch/FX graphs and their strike zone box, I counted every ball and called strike and whether it was in the zone or not. I did not record how far out of the zone it was or whether it was consistent from pitcher to pitcher.

I totaled these numbers in a few ways. First, I was able to get the number of bad calls per game and a percentage of the ball/strike calls that were bad. Also, I took all of the bad calls that favored the Orioles and subtracted all of the bad calls that favored their opponent to get a differential (so, a positive number means that the umps were favoring the Orioles). Finally, I took that same idea and got the percentage of the bad calls that went the Orioles' way. Here's the data:

Date  Opponent   #Bad  %Bad  Diff %FavorOs Pitchers              Umpire
4/6   Twins       15   13.9   -7    26.7   Arrieta vs Pavano     Tim Tschida
4/7   Twins       12   12.6   +2    58.3   Hunter vs Liriano     Jeff Nelson
4/8   Twins       15   15.3   +3    60.0   Hammel vs Swarzak     Bill Welke
4/9   Yankees     25   25.0   -3    44.0   Matusz vs Nova        Paul Schrieber
4/10  Yankees     25   20.2   -9    32.0   Chen vs Garcia        Tim Welke
4/11  Yankees     16   15.4  -12    12.5   Arrieta vs Sabathia   Laz Diax
4/13  Blue Jays    8    8.7    0    50.0   Hunter vs Morrow      Jim Wolf
4/14  Blue Jays   20   18.9   +6    65.0   Hammel vs Alvarez     Derryl Cousins
4/15  Blue Jays   12   10.6   -8    16.7   Matusz vs Drabek      Bob Davidson
4/16  White Sox   18   14.5  -10    22.2   Arrieta vs Humber     Lance Barrett
4/17  White Sox   13   13.5   -1    46.2   Chen vs Danks         Gary Darling
4/18  White Sox    9    8.6   -1    44.4   Hunter vs Peavy       Cory Blaser
4/19  White Sox   17   14.0   -1    47.1   Hammel vs Floyd       Jerry Meals
4/20  Angels      15   16.5   -5    33.3   Matusz vs Williams    Dale Scott
Totals           220   14.9  -46    39.5

Couple of caveats: Pitch/FX strikezone obviously isn't perfect, and I likely made several counting mistakes, so these statistics can almost certainly be improved.

However, it seems like the O's are getting shafted strongly overall. In particular, Jake Arrieta has gotten 6 generous strike calls and squeezed 20 times (-14 overall) and Brian Matusz has gotten 8 generous strike calls and squeezed 19 times (-11 overall). Our other three pitchers are close to even: Hunter +3, Hammel -3, Chen -1. Opposing pitchers have gotten 65 generous strikes and squeezed 45 times in total.

And, of course, the bias was most egregious in the Yankees series.

However, it doesn't seem to matter THAT much, at least in terms of wins and losses. We beat the Twins with a -7, lost to the Yankees with a -9, beat the White Sox with a -10, lost to the White Sox with a -1. Really, we need more data to see how much the effect of this ball/strike differential matters to winning.

Two games in particular stand out as good umpiring jobs: 4/13 vs Blue Jays and 4/18 vs White Sox. Kudos to Jim Wolf and Cory Blaser. Two stand out as bad: 4/11 vs Yankees and 4/16 vs White Sox. Let's hope we see less of Laz Diaz and Lance Barrett.

Happy to answer additional questions that people may have.

[weams]

OK, Frobby's post on ball/strike calls got me thinking, and when I start thinking and want to procrastinate from actual work I research baseball. So I went and got the Pitch/FX data from every game the Orioles have played this year, and did some work.

I looked at the starters for the Orioles and their opponents. Using the Pitch/FX graphs and their strike zone box, I counted every ball and called strike and whether it was in the zone or not. I did not record how far out of the zone it was or whether it was consistent from pitcher to pitcher.

I totaled these numbers in a few ways. First, I was able to get the number of bad calls per game and a percentage of the ball/strike calls that were bad. Also, I took all of the bad calls that favored the Orioles and subtracted all of the bad calls that favored their opponent to get a differential (so, a positive number means that the umps were favoring the Orioles). Finally, I took that same idea and got the percentage of the bad calls that went the Orioles' way. Here's the data:

Date  Opponent   #Bad  %Bad  Diff %FavorOs Pitchers              Umpire
4/6   Twins       15   13.9   -7    26.7   Arrieta vs Pavano     Tim Tschida
4/7   Twins       12   12.6   +2    58.3   Hunter vs Liriano     Jeff Nelson
4/8   Twins       15   15.3   +3    60.0   Hammel vs Swarzak     Bill Welke
4/9   Yankees     25   25.0   -3    44.0   Matusz vs Nova        Paul Schrieber
4/10  Yankees     25   20.2   -9    32.0   Chen vs Garcia        Tim Welke
4/11  Yankees     16   15.4  -12    12.5   Arrieta vs Sabathia   Laz Diax
4/13  Blue Jays    8    8.7    0    50.0   Hunter vs Morrow      Jim Wolf
4/14  Blue Jays   20   18.9   +6    65.0   Hammel vs Alvarez     Derryl Cousins
4/15  Blue Jays   12   10.6   -8    16.7   Matusz vs Drabek      Bob Davidson
4/16  White Sox   18   14.5  -10    22.2   Arrieta vs Humber     Lance Barrett
4/17  White Sox   13   13.5   -1    46.2   Chen vs Danks         Gary Darling
4/18  White Sox    9    8.6   -1    44.4   Hunter vs Peavy       Cory Blaser
4/19  White Sox   17   14.0   -1    47.1   Hammel vs Floyd       Jerry Meals
4/20  Angels      15   16.5   -5    33.3   Matusz vs Williams    Dale Scott
Totals           220   14.9  -46    39.5

Couple of caveats: Pitch/FX strikezone obviously isn't perfect, and I likely made several counting mistakes, so these statistics can almost certainly be improved.

However, it seems like the O's are getting shafted strongly overall. In particular, Jake Arrieta has gotten 6 generous strike calls and squeezed 20 times (-14 overall) and Brian Matusz has gotten 8 generous strike calls and squeezed 19 times (-11 overall). Our other three pitchers are close to even: Hunter +3, Hammel -3, Chen -1. Opposing pitchers have gotten 65 generous strikes and squeezed 45 times in total.

And, of course, the bias was most egregious in the Yankees series.

However, it doesn't seem to matter THAT much, at least in terms of wins and losses. We beat the Twins with a -7, lost to the Yankees with a -9, beat the White Sox with a -10, lost to the White Sox with a -1. Really, we need more data to see how much the effect of this ball/strike differential matters to winning.

Two games in particular stand out as good umpiring jobs: 4/13 vs Blue Jays and 4/18 vs White Sox. Kudos to Jim Wolf and Cory Blaser. Two stand out as bad: 4/11 vs Yankees and 4/16 vs White Sox. Let's hope we see less of Laz Diaz and Lance Barrett.

Happy to answer additional questions that people may have.

Of course this year is still a very small sample size. Any idea what body type the poorer umpires are? Are they big guys?

[CA-ORIOLE]

My only comment would be is how the distribution of those calls is with respect to the area of the strike zone. It's corner game and I dont think we play it well enough. Also, my general observation from the Yankee series was that we let balls go for strikes 2 inches off the corner and be called out while the Yankees went down swinging/hitting on those same pitches. Also, the fact that guys like Arrieta and Matusz (who have/had questionable fastball command) are having more issues isn't actually all that surprising.

[Lucky Jim]

OK, Frobby's post on ball/strike calls got me thinking, and when I start thinking and want to procrastinate from actual work I research baseball. So I went and got the Pitch/FX data from every game the Orioles have played this year, and did some work.

I looked at the starters for the Orioles and their opponents. Using the Pitch/FX graphs and their strike zone box, I counted every ball and called strike and whether it was in the zone or not. I did not record how far out of the zone it was or whether it was consistent from pitcher to pitcher.

I totaled these numbers in a few ways. First, I was able to get the number of bad calls per game and a percentage of the ball/strike calls that were bad. Also, I took all of the bad calls that favored the Orioles and subtracted all of the bad calls that favored their opponent to get a differential (so, a positive number means that the umps were favoring the Orioles). Finally, I took that same idea and got the percentage of the bad calls that went the Orioles' way. Here's the data:

Date  Opponent   #Bad  %Bad  Diff %FavorOs Pitchers              Umpire
4/6   Twins       15   13.9   -7    26.7   Arrieta vs Pavano     Tim Tschida
4/7   Twins       12   12.6   +2    58.3   Hunter vs Liriano     Jeff Nelson
4/8   Twins       15   15.3   +3    60.0   Hammel vs Swarzak     Bill Welke
4/9   Yankees     25   25.0   -3    44.0   Matusz vs Nova        Paul Schrieber
4/10  Yankees     25   20.2   -9    32.0   Chen vs Garcia        Tim Welke
4/11  Yankees     16   15.4  -12    12.5   Arrieta vs Sabathia   Laz Diax
4/13  Blue Jays    8    8.7    0    50.0   Hunter vs Morrow      Jim Wolf
4/14  Blue Jays   20   18.9   +6    65.0   Hammel vs Alvarez     Derryl Cousins
4/15  Blue Jays   12   10.6   -8    16.7   Matusz vs Drabek      Bob Davidson
4/16  White Sox   18   14.5  -10    22.2   Arrieta vs Humber     Lance Barrett
4/17  White Sox   13   13.5   -1    46.2   Chen vs Danks         Gary Darling
4/18  White Sox    9    8.6   -1    44.4   Hunter vs Peavy       Cory Blaser
4/19  White Sox   17   14.0   -1    47.1   Hammel vs Floyd       Jerry Meals
4/20  Angels      15   16.5   -5    33.3   Matusz vs Williams    Dale Scott
Totals           220   14.9  -46    39.5

Couple of caveats: Pitch/FX strikezone obviously isn't perfect, and I likely made several counting mistakes, so these statistics can almost certainly be improved.

However, it seems like the O's are getting shafted strongly overall. In particular, Jake Arrieta has gotten 6 generous strike calls and squeezed 20 times (-14 overall) and Brian Matusz has gotten 8 generous strike calls and squeezed 19 times (-11 overall). Our other three pitchers are close to even: Hunter +3, Hammel -3, Chen -1. Opposing pitchers have gotten 65 generous strikes and squeezed 45 times in total.

And, of course, the bias was most egregious in the Yankees series.

However, it doesn't seem to matter THAT much, at least in terms of wins and losses. We beat the Twins with a -7, lost to the Yankees with a -9, beat the White Sox with a -10, lost to the White Sox with a -1. Really, we need more data to see how much the effect of this ball/strike differential matters to winning.

Two games in particular stand out as good umpiring jobs: 4/13 vs Blue Jays and 4/18 vs White Sox. Kudos to Jim Wolf and Cory Blaser. Two stand out as bad: 4/11 vs Yankees and 4/16 vs White Sox. Let's hope we see less of Laz Diaz and Lance Barrett.

Happy to answer additional questions that people may have.

Great, rep-worthy stuff. The flip side, of course, and just as important is how the other team's pitches are called. When the other team is getting more generous calls, and shafted less often, then it adds to the distortion (given that an ump establishes a zone anew, each game). Amazing.

[skanar]

My only comment would be is how the distribution of those calls is with respect to the area of the strike zone. It's corner game and I dont think we play it well enough. Also, my general observation from the Yankee series was that we let balls go for strikes 2 inches off the corner and be called out while the Yankees went down swinging/hitting on those same pitches. Also, the fact that guys like Arrieta and Matusz (who have/had questionable fastball command) are having more issues isn't actually all that surprising.

This is certainly true and a weakness of my method. In some plots, there were several "generous" strikes clustered together just off the plate for one or both pitchers. I did not attempt to correct for this. Still, I was surprised at the size of the disparity - I was expecting the values to be closer to even.

Also, for weams: small sample size is of course a concern. Ideally, I'd do this over a whole season, but it's actually pretty time consuming and work intensive to do the count. Maybe if I have a really big project to put off....

The flip side, of course, and just as important is how the other team's pitches are called.

This is included in the differential score for each game as well as the %Bad, which is the percentage of "bad" calls that favors the O's. If it's under 50%, the O's were hurt by the umpiring, if it's over 50%, the O's were helped. Of course, anything in the 40-60 range is pretty even.

[Lucky Jim]

This is certainly true and a weakness of my method. In some plots, there were several "generous" strikes clustered together just off the plate for one or both pitchers. I did not attempt to correct for this. Still, I was surprised at the size of the disparity - I was expecting the values to be closer to even.

Also, for weams: small sample size is of course a concern. Ideally, I'd do this over a whole season, but it's actually pretty time consuming and work intensive to do the count. Maybe if I have a really big project to put off....



This is included in the differential score for each game as well as the %Bad, which is the percentage of "bad" calls that favors the O's. If it's under 50%, the O's were hurt by the umpiring, if it's over 50%, the O's were helped. Of course, anything in the 40-60 range is pretty even.

I got it. I wasn't clear at first, and edited. But my point was only that it adds to the issue, percentages aside. (I.e., we tend to think of our pitchers as being "squeezed," when in reality they're being subjected to a different strike zone.)

[skanar]

I got it. I wasn't clear at first, and edited. But my point was only that it adds to the issue, percentages aside. (I.e., we tend to think of our pitchers as being "squeezed," when in reality they're being subjected to a different strike zone.)

Very true. That's why I looked specifically for a way to see whether there was a difference between the two teams and came up with the differential. Interestingly, there didn't seem to be relationship between how "badly" the game was called (%Bad) and how favored (or not) the Orioles were by the calls. Some umpires called a game that agreed with PitchFX and hurt one team (ie, few bad calls but they all went the same way), some called an even game that agreed with PitchFX, some called a game that disagreed with PitchFX and hurt one team, and some called a game that disagreed and favored one team.

[Hang10]

This is great research. Thank you very much.

Confirmed what I already knew about games against the Yankees.

[CA-ORIOLE]

This is certainly true and a weakness of my method. In some plots, there were several "generous" strikes clustered together just off the plate for one or both pitchers. I did not attempt to correct for this. Still, I was surprised at the size of the disparity - I was expecting the values to be closer to even..

Fair enough, I just think you need a percentage of calls per quadrant/area instead of up/down to get a better feel for this. The other factor being individual pitchers command and framing issues.

[Lucky Jim]

Today, Haren had 12 balls called strikes and 1 strike called a ball.

Chen had 3 balls called strikes and 9 strikes called balls.

[JohnnyK27]

Today, Haren had 12 balls called strikes and 1 strike called a ball.

Chen had 3 balls called strikes and 9 strikes called balls.

This is really sad ... I was asking Frobby for the chart on Chen. Its sad that the strike zone would be that different for one pitcher over another. One could argue that Chen might have gotten to the next inning if not for the difference in the zone.

[nevadaO]

I wish there was someone we could show this to. I'm used to our pitchers getting shafted when it's against the Yankees and Red Sox, but it literally has been every game save for maybe 2 this season so far. It's ludicrous.

[skanar]

Today, Haren had 12 balls called strikes and 1 strike called a ball.

Chen had 3 balls called strikes and 9 strikes called balls.

I agree with this count. Let's keep this up all season!

By my +/- counting stat, this was the most lopsided game the Orioles have played this season (-17). It ties two other games for most bad calls (25) and comes in 2nd in terms of percentage of bad calls (to 4/9 vs Yankees).

Yesterday, Jake had 4 strikes called balls and 2 balls called strikes.
Jered Weaver had 4 strikes called balls and 6 balls called strikes.
Game score: -4

[brianod]

OK, Frobby's post on ball/strike calls got me thinking, and when I start thinking and want to procrastinate from actual work I research baseball. So I went and got the Pitch/FX data from every game the Orioles have played this year, and did some work.

I looked at the starters for the Orioles and their opponents. Using the Pitch/FX graphs and their strike zone box, I counted every ball and called strike and whether it was in the zone or not. I did not record how far out of the zone it was or whether it was consistent from pitcher to pitcher.

I totaled these numbers in a few ways. First, I was able to get the number of bad calls per game and a percentage of the ball/strike calls that were bad. Also, I took all of the bad calls that favored the Orioles and subtracted all of the bad calls that favored their opponent to get a differential (so, a positive number means that the umps were favoring the Orioles). Finally, I took that same idea and got the percentage of the bad calls that went the Orioles' way. Here's the data:

Date  Opponent   #Bad  %Bad  Diff %FavorOs Pitchers              Umpire
4/6   Twins       15   13.9   -7    26.7   Arrieta vs Pavano     Tim Tschida
4/7   Twins       12   12.6   +2    58.3   Hunter vs Liriano     Jeff Nelson
4/8   Twins       15   15.3   +3    60.0   Hammel vs Swarzak     Bill Welke
4/9   Yankees     25   25.0   -3    44.0   Matusz vs Nova        Paul Schrieber
4/10  Yankees     25   20.2   -9    32.0   Chen vs Garcia        Tim Welke
4/11  Yankees     16   15.4  -12    12.5   Arrieta vs Sabathia   Laz Diax
4/13  Blue Jays    8    8.7    0    50.0   Hunter vs Morrow      Jim Wolf
4/14  Blue Jays   20   18.9   +6    65.0   Hammel vs Alvarez     Derryl Cousins
4/15  Blue Jays   12   10.6   -8    16.7   Matusz vs Drabek      Bob Davidson
4/16  White Sox   18   14.5  -10    22.2   Arrieta vs Humber     Lance Barrett
4/17  White Sox   13   13.5   -1    46.2   Chen vs Danks         Gary Darling
4/18  White Sox    9    8.6   -1    44.4   Hunter vs Peavy       Cory Blaser
4/19  White Sox   17   14.0   -1    47.1   Hammel vs Floyd       Jerry Meals
4/20  Angels      15   16.5   -5    33.3   Matusz vs Williams    Dale Scott
Totals           220   14.9  -46    39.5

Couple of caveats: Pitch/FX strikezone obviously isn't perfect, and I likely made several counting mistakes, so these statistics can almost certainly be improved.

However, it seems like the O's are getting shafted strongly overall. In particular, Jake Arrieta has gotten 6 generous strike calls and squeezed 20 times (-14 overall) and Brian Matusz has gotten 8 generous strike calls and squeezed 19 times (-11 overall). Our other three pitchers are close to even: Hunter +3, Hammel -3, Chen -1. Opposing pitchers have gotten 65 generous strikes and squeezed 45 times in total.

And, of course, the bias was most egregious in the Yankees series.

However, it doesn't seem to matter THAT much, at least in terms of wins and losses. We beat the Twins with a -7, lost to the Yankees with a -9, beat the White Sox with a -10, lost to the White Sox with a -1. Really, we need more data to see how much the effect of this ball/strike differential matters to winning.

Two games in particular stand out as good umpiring jobs: 4/13 vs Blue Jays and 4/18 vs White Sox. Kudos to Jim Wolf and Cory Blaser. Two stand out as bad: 4/11 vs Yankees and 4/16 vs White Sox. Let's hope we see less of Laz Diaz and Lance Barrett.

Happy to answer additional questions that people may have.

Thanks, this is great info! That being said, I can't disagree more that it doesn't matter that much. The O's get screwed everytime they play the Yankees. We have two extra inning nail biters that could have gone either way against them. To say that it doesn't matter that they got 18 bad calls going against them is crazy. One bad call followed by an rbi hit makes all the difference in the world. Buck getting thrown out today was absolutely necessary. How can anyone compete on an unlevel playing field? It's Fing bad enough the Yankees outspend them 2-1.

[vatech1994]

It makes me feel better that what I thought were the "worst" games were actually the worst. Today's game and the Sabathia game stuck out to me before I saw this thread. We did an amazing job to get that win today. Really excellent effort all around. We lose that game 19 of 20 times with the slanted zone, on the road, last game of a 10 day/10 game trip.

Awesome stuff and great idea to do this daily. If you guys get tired, I can do some games. Rep given.