HHP: Shutdowns, Meltdowns, and the Orioles Bullpen: a Statistical Digression

[skanar]

I was reading this article on Fangraphs, and it reminded me of a pair of statistics I had really likes when they came up with them, but never paid that much attention to after. Since I had some free time after my tutoring appointment this morning, I thought I would do some Orioles-related analysis with Shutdowns and Meltdowns.

For those unfamiliar with the idea, I'll give a quick explanation. Fangraphs has a concept called Win Probability Added (WPA), which measures the change in how likely each team is to win the game after each play, then credits that change to the pitcher and hitter who caused it (in equal amounts). I don't want to go into a long explanation of WPA here; the general idea is that you get more credit for making a difference when the game is close or especially when it is close in the late innings. Here's a link to an article that explains WPA in detail; here's a link to a calculator that lets you find the WPA for any game situation. Note that all defense (including errors) is credited/blamed to the pitcher.

Shutdowns and Meltdowns are based on WPA. If a reliever improves his team's chance of winning the game by 6% or more (from 52% to 59%, for example), he gets one Shutdown. A closer who finished off a winning game for his team will often get about that much: entering the 9th with a 2-run lead, the home team wins 92.6% of the time; a closer who pitches that inning successfully gets (1.00-.926)=.074 WPA. Shutdown numbers are comparable to save numbers, but without the clunky 3-run limit, and they can be acquired by a middle-inning reliever who pitches well while the game is close, or who strands multiple runners. One can think of a Shutdown as a fixed Save.

Meltdowns are just the opposite. A reliever who decreases his team's chance of winning by 6% or more gets one Meltdown. There's no great analogy to a current baseball stat, but a reliever who enters when his team has the lead and allows the other team to tie it (or worse) will essentially always get a Meltdown.

Since they depend on the current leverage of the game, it's hard for a reliever to pick up a SD or MD in a blowout (whether a winning or losing blowout). So a reliever who usually pitches in garbage time is likely to have few of both. A reliever who is constantly used in high-leverage situations will have a lot of both; obviously, the quality of his pitching is measured by how many more SDs than MDs he has (almost all pitchers have more SDs than MDs).

Now that I've caused everyone without a statistics PhD to glaze over, let's use Shutdowns and Meltdowns to take a look at the current Orioles bullpen. I've taken the SD and MD numbers from Fangraphs, and calculated two additional numbers: the SD to MD ratio and the percentage of relief appearances (games minus games started) that resulted in a SD or MD. This last one is largely outside the pitchers' control, depending more on when the manager decides to put them in, but I thought it might be interesting anyway.

And now, the stats. Here's some historical context, with names that will be familiar to many of us (career totals):

                 SD   MD   RA    SD/MD  (SD+MD)/RA
Mariano Rivera   507  105  972   4.83   63.0
Trevor Hoffman   518  138  1035  3.75   63.4
Goose Gossage    419  185  965   2.26   62.6
Armando Benitez  324  115  762   2.82   57.6
Arthur Rhodes    301  116  792   2.59   52.7
Jorge Julio      133   82  453   1.62   47.5
BJ Ryan          193   80  560   2.41   48.8
Chad Bradford    145  104  561   1.39   44.4
Jamie Walker     118   99  543   1.19   40.0
Danys Baez       164   93  481   1.76   53.4
Jesse Orosco     356  220  1248  1.62   46.2
John Smoltz      131   20  242   6.55   62.4
Francisco Rodriguez 266 63 533   4.22   61.7

Jesse Orosco is #2 all time in MDs (behind Kent Tekulve). Rivera and Hoffman are 1-2 in SDs. Francisco Rodriguez set the single-season record for SDs in 2008, with 53. Mike Marshall set the single-season record for MDs in 1974, with 29. Since 2000, Aaron Heilman had the most MDs in a season, with 21 for the 2008 Mets.

Here's the current Orioles bullpen:

                SD   MD  RA   SD/MD  (SD+MD)/RA
Koji Uehara      23   4   45  5.75   60.0
Jim Johnson      60  30  147  2.00   61.2
Jeremy Accardo   66  34  205  1.94   48.8
Josh Rupe        23  11   82  2.09   41.5
Kevin Gregg     127  63  400  2.02   47.5
Jason Berken     11   8   43  1.38   44.2
Mike Gonzalez   118  50  333  2.36   50.5
-------------------------------------------
Totals          428 200  1255 2.14   50.0

And as another point of reference (selected arbitrarily), here's the same stats for the White Sox bullpen:

                SD    MD   RA   SD/MD  (SD+MD)/RA
Matt Thornton   133   68   412  1.96   48.8
Chris Sale        9    3    25  3.00   48.0
Jesse Crain     100   71   378  1.41   45.2
Will Ohman       58   57   395  1.02   29.1
Sergio Santos    16    8    58  2.00   41.4
Tony Pena        90   49   295  1.84   47.1
Phil Humber       4    4    26  1.00   30.8
-------------------------------------------
Totals          410  260  1589  1.58   42.2

League average SD/MD was around 1.75 in 2010. Just by eyeballing the data, it appears that having an SD/MD around 2.00 makes you a decent Major League reliever. Up around 2.50 is All-Star caliber, while anything at or about 3.00 is a potential Hall-of-Famer. Around 1.50 is ok, but anything approaching 1.00 is pretty bad. Relievers around 1.00 are probably not long for the Majors.

It jumps out at me that the better relievers, as measured by SD/MD, also tend to have a high (SD+MD)/RA - suggesting that managers are pretty good at putting their good relievers into the game at important times.

So, what does this mean for the O's? Well, Koji, Rupe, and Berken just don't have a long enough track record to get a good handle on their potential. Still, in their respective partial seasons, Koji was absurdly good and Berken was no better than mediocre. Johnson, Gregg, Accardo, and Gonzalez all have similar stats, and all are above-average, good but not great. So, that's one potentially great reliever, four above-average ones, and two with little data and some upside. Overall, looking at career totals and ignoring injury history and recent trends, the O's should have an above-average but unspectacular bullpen this year: it compares quite favorably to the White Sox, for sure.

Also clear from this data is that Koji is by far the guy we should trust in the 'pen, while JJ, Gregg, Accardo, and Gonzalez will save about twice as many games as they blow.

If people want (and if I have another chunk of time free at some point), I'll try to replicate this analysis to compare the O's bullpen to the other 'pens in the AL East.

[malkusm]

Thanks for this. Very interesting!

[now]

Awesome post! We need more of this stuff.

[srock]

This is really good stuff.

[Tony-OH]

Outstanding post..Looks like a Hangout Highlight Post (HHP) to me!

[Frobby]

What did these numbers look like for us in 2010? "Above average but unspectacular" is something I'd take in a heartbeat, compared to the bullpen performance last year and each of the last 5 years.

[skanar]

What did these numbers look like for us in 2010? "Above average but unspectacular" is something I'd take in a heartbeat, compared to the bullpen performance last year and each of the last 5 years.

I'm working on getting some context from the SD/MD numbers over the last few years - I'll try to put together a long post tonight or tomorrow. The O's SD/MD ratio was 1.56 last year (120 SD, 77 MD) and major league average was exactly 1.70 (3498/2056). The O's have been below average every year from 2005-2010. The Rays (2.33), Padres (2.22), and Yankees (2.15) had the best bullpens by this stat last year; the Diamondbacks (1.21), Mariners (1.28), and Pirates (1.30) had the worst.

[Slappy]

Awesome stuff man! I like this leverage method of analyzing relievers, because frankly there is no good alternative way to analyze relievers. It certainly confirms what I think most of us intuitively know - Koji is by far our best option at closer.

[Can_of_corn]

Awesome stuff man! I like this leverage method of analyzing relievers, because frankly there is no good alternative way to analyze relievers. It certainly confirms what I think most of us intuitively know - Koji is by far our best option at closer.

No, it shows that Koji is by far our best option in high leverage situations. Unfortunately the "closer" in today's game is the guy that gets to come in to start the 9th with his team ahead by 1-3 runs, let Gregg do that, I want Koji being called in the 8th with one out and two on.

I am of the view that the teams best relief pitcher is wasted if the manager follows the closer group mindthink.

[BaltimoreTerp]

No, it shows that Koji is by far our best option in high leverage situations. Unfortunately the "closer" in today's game is the guy that gets to come in to start the 9th with his team ahead by 1-3 runs, let Gregg do that, I want Koji being called in the 8th with one out and two on.

I am of the view that the teams best relief pitcher is wasted if the manager follows the closer group mindthink.

Haven't you seen the commercial? Showalter claims to not be one of those managers

[DrungoHazewood]

Yea, this is great stuff. I saw the other day when Tom Tango linked to the Fangraphs piece introducing the shutdown/meltdown concept. I'd like to see some historical comps. I have a feeling that earlier eras saw more meltdowns and fewer shutdowns just because guys like Gossage or Marshall would often come into a game early enough that all kinds of things could change during their appearance. But that's just a feeling.

[DrungoHazewood]

No, it shows that Koji is by far our best option in high leverage situations. Unfortunately the "closer" in today's game is the guy that gets to come in to start the 9th with his team ahead by 1-3 runs, let Gregg do that, I want Koji being called in the 8th with one out and two on.

I am of the view that the teams best relief pitcher is wasted if the manager follows the closer group mindthink.

I used to think that the best use of a dominant reliever was always to just get him warmed up and in the game whenever you had a close game and the other team was rallying after the 6th inning or so.

But I've kind of changed my thinking. That might work if you did it just right, and you had a rubber-armed reliever who could get warmed up really quick. But there is value in letting a guy warm up right, and knowing that he's always coming in at a specific time in the game. Especially when your best reliever has his arm attached with chewing gum and bailing wire.

A save situation is often the highest leverage part of a game. Or if it's not, it's often close. Look at Fangraphs, and almost all of the leaders in average leverage index were closers, used in the typical modern closer way. And if you look at the all time leaders in LI I don't see any advantage from the old 1970s style firemen. On average Jonathan Papelbon is brought into higher leverage situations than Rollie Fingers or Goose Gossage.

[blid]

Does this work well for hitters? Could I compare Miguel Tejada's WPA with Nick's to see who is more clutch?

[Tony-OH]

Awesome stuff man! I like this leverage method of analyzing relievers, because frankly there is no good alternative way to analyze relievers. It certainly confirms what I think most of us intuitively know - Koji is by far our best option at closer.

It certainly explains why I feel so much more confident when he's in the game.

[Can_of_corn]

It certainly explains why I feel so much more confident when he's in the game.

Now that I agree with.