Two Myths I'm Ready to Debunk

[square634]

Another answer provided has been this “65%" figure. Looking at that article linked, the 65% that some people appear now to be taking as gospel refers to “the success rate for a typical weighted three-year projection system like Marcel [is] at about 65 percent.” So, based on one projection system that someone analyzed and found about a 65% success rate, we close the book on any other projection formulas?!? Marcel success ≈ about 65% -- so throw out PECOTA, forget any other projection sites/formulas. To me, that 65% is far from gospel. How many projection formulas are there in the world of baseball statistical analysis? What are their success rates?

I don't know PECOTAs from Marcels, but this is fast becoming a very interesting topic to me. Let's all just keep an open mind, not shoot the messenger, and see if we can figure out some new way to do business.

FWIW, I'm pretty sure PECOTA has the best correlation for position players.

The answer to that question isn’t “thousands of players have been studied and AAA players promoted to ML will produce about 90% (or whatever the percentage) of their AAA stats.” The question is how accurate is that 90% figure. Do all AAA players hit that 90% translation? Do 80% of them? Less?

My guess is that the "translation percentage" would be the median performance of players called up. Therefore, if the translation percentage is calculated to be 90%, then 50% of players perform better than that translation, and 50% of players perform worse. By using other factors, such as age, ballparks, PECOTA comparisons, etc. a more complicated projection than a simple percentage (i.e. minor league stats x 90%) can be computed. Some people will not be satisfied with the probability results on an individual basis, but what I think is important is the simple idea that someone who performs better in the minor leagues is more likely to succeed in the major leagues. Even if Player A with a .900 OPS in AAA doesn't reach his projected statistics, he will still almost definitely perform better than Player B with a .600 OPS in AAA, even if Player B performs better than projections would expect. In cases of close statistical differences, more weight should be given to scouts and coaches.

I think most people are upset with the Orioles because statistical differences seem to be largely ignored. Maybe Jack Cust, JR House, and Jon Knott only have a 50% (or some arbitrary percentage) chance to perform well in the majors, but you'll never find out if they are never in the lineup, and it sure beats plugging a .560 AA OPS with a .1% chance of long term ML success into the lineup every day. And in the case of Jack Cust as an example, the Orioles were totally wrong.

[rolliefingers]

One school of thought says, always prefer Player 1 to Player 2. In the long run, you will make mistakes, but you will be right more often than you will be wrong. You will get more hits than misses.

The other school of thought says, look at other stuff and try to sort the 65% of Type A who will be successful from the 35 % who won't be. And try to pick out the 35% of Type B who will be successful and give them a shot.

I would say the Orioles are definitely in the second school of thought. A lot of the critics on this board are in the first school. Which is better? Well, let's just say that whatever the Orioles have been doing, it hasn't been working.

I agree, the Orioles do subscribe to the second school. And they are a walking advertisement for the first school. They have attempted to pinpoint specific differences between players, and they've failed nearly every time.

The Orioles take a player like Jack Cust or J.R. House and look for an excuse not to give him a shot, instead of cutting out the BS - the scouting reports, the rumors of defensive ineptitude - and looking at the numbers.

They also take players like Jay Payton and Jay Gibbons and create rosy scenarios in which they might do well, and overpay them, while releasing the first guys.

I am convinced that they'd be better off with a robot for a GM, which would make decisions based solely on stats. Sure, they'd miss out some players who don't fit that statistical model - and might be a flat-out awful defensive team - but they also wouldn't currently be stuck with Jay Payton or Jay Gibbons.

Again, MiL numbers ain't perfect, but at least they're actual, real results. I'd rather be bullheadedly worshipful of stats than risk overthinking things and making them worse.

[square634]

Well, there is an interesting philosophical question here that underlies many, many discussions we have on this board: are you better off just playing the statistical odds as you see them, or are you better off trying to distinguish the exceptions.

Let me explain. Assume there are two players. One falls into a series of statistical categorties that, without any other input, would lead you to believe that he has a 65% chance of being a successful major leaguer. The other fits into a series of categories that, without more, would lead you to believe that he has only a 35% chance of success.

One school of thought says, always prefer Player 1 to Player 2. In the long run, you will make mistakes, but you will be right more often than you will be wrong. You will get more hits than misses.

The other school of thought says, look at other stuff and try to sort the 65% of Type A who will be successful from the 35 % who won't be. And try to pick out the 35% of Type B who will be successful and give them a shot.

I would say the Orioles are definitely in the second school of thought. A lot of the critics on this board are in the first school. Which is better? Well, let's just say that whatever the Orioles have been doing, it hasn't been working.

Good post. I would certainly be open to a system that could differentiate between who will be the successful and unsuccessful players in each group, but if one exists, it is clear that the Orioles don't know it.

[osfan83]

I think most people are upset with the Orioles because statistical differences seem to be largely ignored. Maybe Jack Cust, JR House, and Jon Knott only have a 50% (or some arbitrary percentage) chance to perform well in the majors, but you'll never find out if they are never in the lineup, and it sure beats plugging a .560 AA OPS with a .1% chance of long term ML success into the lineup every day. And in the case of Jack Cust as an example, the Orioles were totally wrong.

Bingo! It would be an incredable story if Luis Hernandez turned into a solid Major Leaguer. Possible yes, probable, not at all. Yet he takes a roster spot, while someone like Knott, who may only have a 50/50 shot at best, doesn't even get a chance.

The O's are at the Black Jack table, and they insist on hiiting at 17 because sooner or later they will draw a 4. Meanwhile they keep losing money.

[Sports Guy]

Well, there is an interesting philosophical question here that underlies many, many discussions we have on this board: are you better off just playing the statistical odds as you see them, or are you better off trying to distinguish the exceptions.

Let me explain. Assume there are two players. One falls into a series of statistical categorties that, without any other input, would lead you to believe that he has a 65% chance of being a successful major leaguer. The other fits into a series of categories that, without more, would lead you to believe that he has only a 35% chance of success.

One school of thought says, always prefer Player 1 to Player 2. In the long run, you will make mistakes, but you will be right more often than you will be wrong. You will get more hits than misses.

The other school of thought says, look at other stuff and try to sort the 65% of Type A who will be successful from the 35 % who won't be. And try to pick out the 35% of Type B who will be successful and give them a shot.

I would say the Orioles are definitely in the second school of thought. A lot of the critics on this board are in the first school. Which is better? Well, let's just say that whatever the Orioles have been doing, it hasn't been working.

There is no system that can tell you this IMO.

Things happen...Guys get hurt, they develop differently and things like that.

You just have to play the odds.

You just have to look at everything and just go with the best chances.

Let's take Knott for example:

1) Has always hit in the minors but his best stats were in the PCL...So, you have to adjust for that.

2) Always been better against lefties than righties.

3) For a big guy, he moves well but is hardly a defensive whiz but he doesn't kill you out there either.

4) Is old for a prospect....His numbers, while they have been impressive, are being done at older ages now.

So, what does all of this tell you? To me it says that this is a guy who should have value in a platoon type role for 2-3 years and when he starts getting into arbitration, his value isn't there as much anymore because you can find another one just like him for 300K, as opposed to 1-2 million.

I don't need someone like BP to tell me that there is a 65% chance that he can fill this role...That is meaningless to me.

All i know is i would rather see if he works out then spending 4-5 million on a guy like Payton and really, that is all i need to know.

[rolliefingers]

The O's are at the Black Jack table, and they insist on hiiting at 17 because sooner or later they will draw a 4. Meanwhile they keep losing money.

That's a really good analogy.

[Lucky Jim]

There is no system that can tell you this IMO.

Things happen...Guys get hurt, they develop differently and things like that.

You just have to play the odds.

You just have to look at everything and just go with the best chances.

Let's take Knott for example:

1) Has always hit in the minors but his best stats were in the PCL...So, you have to adjust for that.

2) Always been better against lefties than righties.

3) For a big guy, he moves well but is hardly a defensive whiz but he doesn't kill you out there either.

4) Is old for a prospect....His numbers, while they have been impressive, are being done at older ages now.

So, what does all of this tell you? To me it says that this is a guy who should have value in a platoon type role for 2-3 years and when he starts getting into arbitration, his value isn't there as much anymore because you can find another one just like him for 300K, as opposed to 1-2 million.

I don't need someone like BP to tell me that there is a 65% chance that he can fill this role...That is meaningless to me.

All i know is i would rather see if he works out then spending 4-5 million on a guy like Payton and really, that is all i need to know.

I think you're missing the point that Mackus was trying to make (and that I added onto) which is that the percentage (if it's accurate) helps a team efficiently build a farm system by letting it know where its risk lies. It might not help with Knott, in particular (in that it's not giving any certainty about performance) but rather that the risk of Knott failing (which is likely 50-50) could be mitigated by a replacements.

This was sort-of what I thought the O's were doing this off-season when they signed Knott, House AND Sing. Instead, they relied on Jay Payton.

It's really a kind-of Moneyball approach - if you know that Knott is 50-50, you can find cheap (perhaps flawed) replacements to hedge against that failure.

Knott's 65% chance of success (I think it's more likely to be around 40%) DOES tell you something: it tells you that there's a 35% chance that he fails. The amount of money needed in contingencies can be measured accurately if you have a good sense of where the risk lies and at what level.

Maybe the Blackjack analogy is a good one. And maybe rude estimations (i.e, Knott over Payton) are fine when you don't have the capacity to measure the odds more precisely. That said, if you can count cards, you do it. You do it, and you win.

[square634]

Bingo! It would be an incredable story if Luis Hernandez turned into a solid Major Leaguer. Possible yes, probable, not at all. Yet he takes a roster spot, while someone like Knott, who may only have a 50/50 shot at best, doesn't even get a chance.

The O's are at the Black Jack table, and they insist on hiiting at 17 because sooner or later they will draw a 4. Meanwhile they keep losing money.

I'd say it's more like hitting on 20 and hoping you draw an Ace, but a wonderful analogy nonetheless!

[Tony-OH]

Well, there is an interesting philosophical question here that underlies many, many discussions we have on this board: are you better off just playing the statistical odds as you see them, or are you better off trying to distinguish the exceptions.

Let me explain. Assume there are two players. One falls into a series of statistical categorties that, without any other input, would lead you to believe that he has a 65% chance of being a successful major leaguer. The other fits into a series of categories that, without more, would lead you to believe that he has only a 35% chance of success.

One school of thought says, always prefer Player 1 to Player 2. In the long run, you will make mistakes, but you will be right more often than you will be wrong. You will get more hits than misses.

The other school of thought says, look at other stuff and try to sort the 65% of Type A who will be successful from the 35 % who won't be. And try to pick out the 35% of Type B who will be successful and give them a shot.

I would say the Orioles are definitely in the second school of thought. A lot of the critics on this board are in the first school. Which is better? Well, let's just say that whatever the Orioles have been doing, it hasn't been working.

I don't think it's an either or situation. I think you use Type A and Typ2 B and that equals 100%

Seriously though, I don't see any reason why one school of thought should be used. Smart teams use every tool to their advantage and that means using statistical analysis and tradtional scouting.

[Sports Guy]

I think you're missing the point that Mackus was trying to make (and that I added onto) which is that the percentage (if it's accurate) helps a team efficiently build a farm system by letting it know where its risk lies. It might not help with Knott, in particular (in that it's not giving any certainty about performance) but rather that the risk of Knott failing (which is likely 50-50) could be mitigated by a replacements.

This was sort-of what I thought the O's were doing this off-season when they signed Knott, House AND Sing. Instead, they relied on Jay Payton.

It's really a kind-of Moneyball approach - if you know that Knott is 50-50, you can find cheap (perhaps flawed) replacements to hedge against that failure.

Knott's 65% chance of success (I think it's more likely to be around 40%) DOES tell you something: it tells you that there's a 35% chance that he fails. The amount of money needed in contingencies can be measured accurately if you have a good sense of where the risk lies and at what level.

Maybe the Blackjack analogy is a good one. And maybe rude estimations (i.e, Knott over Payton) are fine when you don't have the capacity to measure the odds more precisely. That said, if you can count cards, you do it. You do it, and you win.

But again, an arbitrary % made up, based off stats that we don't know what they are using, is really meaningless.

End of the day, if someone tells me that so and so has a 65% of being a productive MLer and another player has a 35% of being a productive MLer, i will go with the 65% most of the time.

Yes, there will be times when you are wrong but there will be a lot more times when you are right.

But i can still come up with the same estimate even without knowing those percentages.

[Sports Guy]

I don't think it's an either or situation. I think you use Type A and Typ2 B and that equals 100%

Seriously though, I don't see any reason why one school of thought should be used. Smart teams use every tool to their advantage and that means using statistical analysis and tradtional scouting.

Correct.....You use one off of the other...If your scouts say this guy is going to be a star and the stats back that up, then you go for him.

Maybe he doesn't always end up panning out but it is still better odds.

[Lucky Jim]

But again, an arbitrary % made up, based off stats that we don't know what they are using, is really meaningless.

End of the day, if someone tells me that so and so has a 65% of being a productive MLer and another player has a 35% of being a productive MLer, i will go with the 65% most of the time.

Yes, there will be times when you are wrong but there will be a lot more times when you are right.

But i can still come up with the same estimate even without knowing those percentages.

Well OF COURSE an arbitrary number based on unknown stats is worthless. But what about an accurate number? If that was your only point (that we shouldn't rely on meaningless probabilities) then there's no disagreement.

But that's not what a lot of us have been talking about...

The further problem is that you keep reducing the argument to choosing one player over another. But that's not really what the potential of a probabilistic analysis is about. Rather than thinking "I'll choose X over Y" you need to think of it as a way of filling systemic needs. It's more about I'll choose X plus Y over Z because I think that hedges my risk, equals production, and comes more cheaply. Further, it allows a team to take risks on toolsy players who they think they can correct while still bringing along guys whose numbers predict greater success.

Having Luis Hernandez as a prospect wouldn't be the end of the world, if there was someone at an equal level who was more likely to produce. The gamble is having Hernandez as your only shortstop.

Now, the rebuttal to this would simply be: bring in more prospects. The more prospects the better. My only point was that an accurate probabilistic measure would allow that investment to be conducted more cheaply and efficiently. Which is always a good. Always.

[Sports Guy]

Well OF COURSE an arbitrary number based on unknown stats is worthless. But what about an accurate number? If that was your only point (that we shouldn't rely on meaningless probabilities) then there's no disagreement.

But that's not what a lot of us have been talking about...

But how do they come up with that number?

What stats are they using?

[Fan4Life]

But again, an arbitrary % made up, based off stats that we don't know what they are using, is really meaningless.

End of the day, if someone tells me that so and so has a 65% of being a productive MLer and another player has a 35% of being a productive MLer, i will go with the 65% most of the time.

Yes, there will be times when you are wrong but there will be a lot more times when you are right.

But i can still come up with the same estimate even without knowing those percentages.

How long do you give a guy to know whether or not he can reach the level of success you expect from him? Bedard was stuck with quite awhile, DCab as well, others, Knott, Cust weren't given much time. From what we saw of our minor leaguers this season, we were quite disappointed in just about all of them. That's not to say that next season or the season after that some of them or all of them couldn't fulfill our expectations of them, but how long do you wait? Is it strictly an age factor?

[Lucky Jim]

I'm not a statistician. Age-comps would be a good start, of course.