Two Myths I'm Ready to Debunk

[hoosiers]

This core point is what I've been saying for a long time. However, now we have a new bogus claim: everybody's all-the-sudden latching on to 65% to mean something it doesn't mean. That was for using the last 3-years of ML numbers to predict upcoming ML performance. It has nothing to do with the reliability of MiL numbers as a predictor of ML numbers. I expect that the MiL number would be lower than 65%, but nobody seems to know. Seems odd that nobody would know, but that appears to be the case. Until somebody turns up something, let's not pretend it's 65%. We don't know what it is. But, whatever it is, it ain't gonna be better than 65% and it's likely to be noticeably worse than that.

Who has made a "new bogus claim" that minor league numbers can be translated into major league numbers with 65% reliability? Where did anyone post as much?

Further, you have three active participants in this thread above you acknowledging that 65% is a good "indicator"! Who said "reliable"? And you have a pretty thorough espn article saying 65% is about as good as it gets among the very brightest sabermaticians - perhaps a tad better. Are you aware of anything with a track record BETTER than 65%?

[RShack]

Who has made a "new bogus claim" that minor league numbers can be translated into major league numbers with 65% reliability? Where did anyone post as much?

Have you even been reading this thread. I kept asking for some stat-based justification for the by-now-standard claim that MiL numbers are a reliable-albeit-imperfect predictor of ML performance. After a flurry of people saying they didn't need to justify it, and that it was somehow up to me to disprove sweeping claims that were completely unsubstantiated, there was another brief flurry of people posting links. Most of those links had nothing to do with the topic. One of those links cited the 65% number for something else, and multiple people then said to me, "There's your number... it's 65%" If you wanna know who those people are, just go back and look.

[RShack]

Further, you have three active participants in this thread above you acknowledging that 65% is a good "indicator"! Who said "reliable"? And you have a pretty thorough espn article saying 65% is about as good as it gets among the very brightest sabermaticians - perhaps a tad better. Are you aware of anything with a track record BETTER than 65%?

It is routine around here for people to claim that stats are a reliable predictor, albeit imperfect. What we have learned is that this claim is not true. The fact that stat-guys can't do better than 65% does not mean that 65% is good. It's not. As predictors go, it's weak. It's better than a coin flip, but not by all that much. This demonstrates that FO guys should not trust stats as the overriding basis for decision, which runs contrary to the basic premise of many threads around here.

I am well aware that many posters here readily accept that idea that FO decisions are properly made based on a combination of data points that include stats but that also weigh the informed opinion of high-quality evaluators of talent. However, some of the more strident posters look at stats and insist that they alone are a sound basis for determining the moves that the FO should make. I would like to think that the facts available to us will burst that bubble, but I don't really expect this to be the case.

[Sports Guy]

Rshack, your argument is beyond tiring....You know, go do the research yourself.

You want everyone to define a word or do the research for you or whatever...And when they do, you find another reason to whine about it.

So do the research yourself...See what you come up with and then let us know.

If you don't want to do it, that's fine but then drop your bogus, silly, ridiculous argument.

[hoosiers]

Have you even been reading this thread. I kept asking for some stat-based justification for the by-now-standard claim that MiL numbers are a reliable-albeit-imperfect predictor of ML performance. After a flurry of people saying they didn't need to justify it, and that it was somehow up to me to disprove sweeping claims that were completely unsubstantiated, there was another brief flurry of people posting links. Most of those links had nothing to do with the topic. One of those links cited the 65% number for something else, and multiple people then said to me, "There's your number... it's 65%" If you wanna know who those people are, just go back and look.

We all read the ESPN article where the 65% came from. The article spoke clearly about the ability of sabermaticians to predict major league performance. The article, as I recall, said boo regarding forecasting minor league numbers into major league performance. And I do not believe anyone has made "a bogus claim" that the 65% represents forecasting predictability of minor league stats into major league ones.

As far as I can recall, people on this thread claimed that those with better minor league numbers tend to put up better major league numbers.

I would be a little less cavalier when accusing fellow posters of making "bogus claims" when, in fact, none exists.

[RShack]

We all read the ESPN article where the 65% came from. The article spoke clearly about the ability of sabermaticians to predict major league performance. The article, as I recall, said boo regarding forecasting minor league numbers into major league performance. And I do not believe anyone has made "a bogus claim" that the 65% represents forecasting predictability of minor league stats into major league ones.

Dude, what are you arguing about? As I already explained, I was simply referring to things that were posted in this very thread. I had no intention of dragging them up, but since you don't trust me to tell you the truth about that, here's a couple of them...

There are many cases where it is common sense. If you have 2 otherwise comparable MiL players, and if one guy's OPS is .600 and the other guy's is .800, then it's a no brainer which one you take more seriously. That is indeed common sense. That is also a case where stats tell you little that you don't know anyway by having competent coaches (or fans) simply watch them play.

That's not the question. The question is how likely is the .800 guy to perform at a successful level in the big leagues. To put it another way, of all the guys who put up .800 in the MiL, what percentage of them had worthwhile (by any standard you choose) ML careers? For example, if you're playing GM, wouldn't you want to know what percentage of them stuck in the bigs for 5 years or more and put up decent numbers during that time. Or is that somehow irrelevant just a guy's MiL numbers were good?

I wish we could get beyond silly arguing and at least agree that there is indeed something important to learn here, something that we could all profit from.

Well, you have seen the number ay 65%...Can you stop whining about this now?

Did you not see the 65 percent success rate? Your question has been answered.

O.K. Let's call the percentage of all the guys in the MiL who put up .800 who stick in the big leagues for 5 years or more "percentage X." And Let's call the percentage of all the guys in the MiL who put up .600 and went on to stick in the bigs "percentage Y." Can we all agree that percentage X - whatever number it is - is bigger than percentage Y?

So knowing that it's 65% ...

That's 3 right there...

I would be a little less cavalier when accusing fellow posters of making "bogus claims" when, in fact, none exists.

You might try being a little less cavalier when accusing fellow posters of making stuff up when it's right there in black and white in the very same thread. I'm not saying anybody's a bad guy, I'm just saying that some folks are referring to 65% as meaning something it doesn't. If we're gonna be referring to 65%, let's at least get on the same page about what it refers to. What about this is so hard to understand?

(My apologies to others who don't wanna read all this junk...)

[Mackus]

I'm just saying that some folks are referring to 65% as meaning something it doesn't. If we're gonna be referring to 65%, let's at least get on the same page about what it refers to. What about this is so hard to understand?

This isn't exactly what you are looking for, but there are some very consistent statistical analysis that compare the differences between MLB, AAA, Japan, AA, etc. They even break things down into each league of each level, accounting for differences in say the PCL from the IL. I don't know exactly what they are, but for argument sake lets say that AAA is roughly 90% as difficult as MLB, AA is 80%, A is 70%, you could do a rough estimate of saying how consistently a guy's minor league stats are gonna translate to the majors based on his current level.

If past history is accurate 65% of the time, and there is a 90% correlation between AAA and MLB, then a good estimate for how reliable AAA numbers are would be about 90% of 65%, or about 59%. Again thats just a rough estimate/conversion of the numbers that apparently we do know (65% reliability in using past data to predict future data), but its better than pure conjecture.

I didn't actually see whatever post started that 65% number, but is that the liklihood that a guy will continue to have a similar amount of success that they've had previously? If so, then I think an even more valuable rate would be the liklihood that a guy will be within 10% or so of his past performance. That gives you a range of expectations for somebody. My thoughts would be that if you consider "success" being that the player performs within a reasonable range of what you'd expect, that you'd see past stats predict future performance at a higher than 65% clip.

[hoosiers]

rshack, you are correct. Some people misinterpreted the article.

I still think it is irresponsible to accuse these people of making "bogus" claims.

[RShack]

This isn't exactly what you are looking for, but there are some very consistent statistical analysis that compare the differences between MLB, AAA, Japan, AA, etc. They even break things down into each league of each level, accounting for differences in say the PCL from the IL. I don't know exactly what they are, but for argument sake lets say that AAA is roughly 90% as difficult as MLB, AA is 80%, A is 70%, you could do a rough estimate of saying how consistently a guy's minor league stats are gonna translate to the majors based on his current level.

If past history is accurate 65% of the time, and there is a 90% correlation between AAA and MLB, then a good estimate for how reliable AAA numbers are would be about 90% of 65%, or about 59%. Again thats just a rough estimate/conversion of the numbers that apparently we do know (65% reliability in using past data to predict future data), but its better than pure conjecture.

I didn't actually see whatever post started that 65% number, but is that the liklihood that a guy will continue to have a similar amount of success that they've had previously? If so, then I think an even more valuable rate would be the liklihood that a guy will be within 10% or so of his past performance. That gives you a range of expectations for somebody. My thoughts would be that if you consider "success" being that the player performs within a reasonable range of what you'd expect, that you'd see past stats predict future performance at a higher than 65% clip.

Well, first off, thank you for having a good inquisitive spirit about this. I’m tired of arguing about it. It’s the kind of thing that should be interesting, not a matter for pugnacious battles, and it’s certainly good to hear constructive ideas.

The 65% number was a general statement made about the reliability of the best predictive systems for doing a specific task: using the last 3-years of ML performance as a predictor for the subsequent year's performance. We don’t know where the ESPN guy got it from, or anything else about it. We're not even sure which predictive methods he's including in that, but I imagine that he's including the proprietary methods of fee-based subscription stat-services. All of which means the 65% number is a bit of a phantom number… but it’s the only one we’ve got! ;-)

I think your idea of multiplying 65% (or whatever the magic number is) times some appropriate league-factor is logical. I think there is probably something to that. I imagine that it would not turn out to be a simple multiplication of 2 factors; it would probably also have some weird factors and constants and such. But I agree that you’d certainly wish to use such info if you had it, it’s just not clear exactly how to do that.

I think another issue is that the 65% refers to the reliability of the best predictive methods. AFAIK, for many of them, we don’t even know how they work, i.e., some are based on secret proprietary formulas from subscription stat-services that charge money. I don’t see people here using expensive or complex predictive methods for MiL guys. Instead, I see people simply yanking out some MiL guy’s basic simple stats, such as OPS, and using those simple stats *as if* it they are the best predictive tool. Something is very wrong with this, but what is it?

• Either OPS belongs on the list of “best predictive methods”, right there with complex and proprietary stat formulas… in which case why are people doing complex calculations and/or subscribing to proprietary stat-services?
• Or some of the complex predictive methods are indeed better than using a simple-stat like OPS… in which case OPS (and similar simple stats that are used around here as predictors) have reliability that is notably worse than 65% reliability.

My guess is that the latter is true. While the best predictive methods may be 65% reliable, that’s not what people here typically use when discussing MiL guys. Instead, they use superficial stats whose predictive reliability is likely to be way worse than the more complex predictive methods. So if the best complex methods are only 65% reliable, and we’re using simple stats that are worse than that, how much worse than 65% can you go without getting to the reliability of coin-flips?

If we want stats to be useful about this for predicting how specific guys will do, then I don’t think that we’re really looking for a single “magic number” for how reliable stat-based predictions of MiL’ers are. It’s good to have a single number for how predictions do over-all. But if we use that single number to tell us how much to trust the MiL stats of a specific guy, then we’re acting is if all MiL guys present the same predictive challenge. But I don’t think that’s how things really are. We know that some guy’s MiL stats provide us with more confidence than others… not just confidence in the player, but also confidence that the level of performance that is reflected in his MiL stats will carry forward to the ML level. Predictions will be more reliable for players at either extreme of performance. If the MiL player is either Mickey Mantle or Elmer Fudd, then the MiL stats will be hugely accurate at predicting ML success or failure. It’s the guys in the middle where we have the greatest need for, and the greatest problems with, trustworthy predictions based on MiL stats. If we use any one number, we’d be making the mistake of pretending that there is one reliability factor that applies to everybody. Yet, in reality, reliability will be more-trustworthy for guys at the extremes of very good and very bad, and the reliability of predictions with be worse for guys in the middle.

What we need is not a single number but rather a curve for predictive reliability. If we had such a curve, then we could index the player into that curve, based on whatever stats are useful, and then see how confident we can be in whatever prediction is based on those stats. If some stats-folk wanted to go to the trouble of solving this problem, they certainly could do so. And maybe some of them already have, and we just don’t know about it. But, since some people around here have been studying baseball stats forever, I would trust that some of them would’ve heard about it, if it had been done. It would be an importantly useful chunk of info, so I don’t see how it would/could stay a secret… unless it’s proprietary info that’s owned by the subscription services, and they may not want us to know what their reliability is. Since some of the best predictive methods are secret proprietary info, I would expect that info on their reliability prolly is too.

My completely-unsubstantiated hunch is that if we knew what the “Reliability Curve” looked like, it would be a lot like an upside-down bell curve. Where a normal bell curve’s highest point is in the middle and its lowest points are out at the far ends, I expect that the Reliability Curve would be the opposite, with its lowest point in the middle and highest points at the far ends. The more radically-good a player’s MiL numbers are, the more confident we can be that his MiL stats are reliable predictors (of success). The more stinkily-bad they are, the more confident we can be that they are reliable predictors (of failure). The more of a dilemma a guy is, the least trustworthy the predictions will be. So at the very middle of the Reliability Curve, right where we find the guys who are the biggest dilemmas, the curve reaches its bottom at a point where it approaches the reliability of a coin-flip, which means that it tells us exactly what we already know anyway: that the guy is truly a dilemma, and he may or may not succeed. But I’m just guessing….

[BRobinsonfan]

I would like to think that the facts available to us will burst that bubble, but I don't really expect this to be the case.

I don't even know what this means???

[BRobinsonfan]

Rshak,

Maybe this will help us understand where you're coming from. Since you don't think MiL stats are a predictor of future success maybe you could give us a list of five or ten AAA or AA players that have poor to mediocre minor league stats that you see as becoming Major League players within the next couple of years. And, without using stats, tell us why you think they will be.

[Flosman]

Rshack, your argument is beyond tiring....You know, go do the research yourself.

You want everyone to define a word or do the research for you or whatever...And when they do, you find another reason to whine about it.

So do the research yourself...See what you come up with and then let us know.

If you don't want to do it, that's fine but then drop your bogus, silly, ridiculous argument.

You know it is really not a bogus, ridiculous argument. If a team were able to find a system that works better (which I suspect some have, based on thier consistant success) the advantage gained is worth more than we can imagine. This is the type of thing I would assign personel to explore if I were runnning a ML team. Get real smart stats guys to try to find the corrolations in all the data available. This is the crux of shacks position. If you quantify, somehow, scouting reports, makeup assesments, health assessments, onfield proformance it is likely that a better system can be developed. Shack is pointing out that the realitive wieghts of the factors are not known at this time, particularly by any of us. He is doing what people in science do, that being stating that our understanding of a process is not good enough to predict outcomes reliably, and more improtantly why is that and what can we do to change it. I don't think he is asking people to do research for him. I am pretty certain that he believes that the task would be near impossible of any of us because of our limitations relating to the availability to important data and the fact that it would be a full time job for a long time just for the possibilty to develop meaningful skill.

I for one find his view interesting and thought provoking. Does it solve anything no. But it is not inaccurate and if it bothers you so much why do you particapate in the threads so much?

[Flosman]

Rshak,

Maybe this will help us understand where you're coming from. Since you don't think MiL stats are a predictor of future success maybe you could give us a list of five or ten AAA or AA players that have poor to mediocre minor league stats that you see as becoming Major League players within the next couple of years. And, without using stats, tell us why you think they will be.

He is not saying they are not a predictor. He says they are not a reliable predictor. There is a big difference in those two positions. You are also asking him to do something that he has stated that currently can not be reliably done. I think his hypothiesis about and inverse relationship of reliability and on the field proformance realitive to the mean is very interesting and has merit. i think if proven out it says someting about how a ML team should look at developing players. In other words don't hang on to as many players that put up a little better that average minor league stats and expect to be able to consistantly find impact players out of that group.

[BRobinsonfan]

He is not saying they are not a predictor. He says they are not a reliable predictor. There is a big difference in those two positions.

In the spirit of RShak... can you clarify just how big a difference there is in those two positions. Is it 65% different? 21.8% different. I mean... just how big a difference are we talking about here? Is it a statistically significant?

[BRobinsonfan]

Dude, what are you arguing about? As I already explained, I was simply referring to things that were posted in this very thread. I had no intention of dragging them up, but since you don't trust me to tell you the truth about that, here's a couple of them...

That's 3 right there...

You might try being a little less cavalier when accusing fellow posters of making stuff up when it's right there in black and white in the very same thread. I'm not saying anybody's a bad guy, I'm just saying that some folks are referring to 65% as meaning something it doesn't. If we're gonna be referring to 65%, let's at least get on the same page about what it refers to. What about this is so hard to understand?

(My apologies to others who don't wanna read all this junk...)

You are misquoting me in a rather disengenous way through the use of ellipses... I'd appreciate it if you'd rectify that.

Originally Posted by BRobinsonfan

O.K. Let's call the percentage of all the guys in the MiL who put up .800 who stick in the big leagues for 5 years or more "percentage X." And Let's call the percentage of all the guys in the MiL who put up .600 and went on to stick in the bigs "percentage Y." Can we all agree that percentage X - whatever number it is - is bigger than percentage Y?

So knowing that it's 65% ...

WHY DID YOU LEAVE OUT WHAT I SAID AFTER SAYING "SO KNOWING THAT IT'S 65%..." ???