Two Myths I'm Ready to Debunk

[Flosman]

In the spirit of RShak... can you clarify just how big a difference there is in those two positions. Is it 65% different? 21.8% different. I mean... just how big a difference are we talking about here? Is it a statistically significant?

I ran the numbers and it is 87.35475% with a margin of error of +-100%.

Joking aside don't you think the questions he is asking have merit and we would have a much better club if we were the team to develop a better mousetrap? At 65% apparently nobody is that great at it and there is a vast amount of advantage that it would be possible to obtain. I think there is alot of evidence that the O's have actually been worse than other teams at this analysis over recent history. We seem to hang on to the wrong guys and let the right guys go at an alarming rate. A team that could reliably predict these things may have traded Bigbie for something after his initial success, maybe they still have Maine, Julio may have turned into a guy who is a star now.

[Sports Guy]

I ran the numbers and it is 87.35475% with a margin of error of +-100%.

Joking aside don't you think the questions he is asking have merit and we would have a much better club if we were the team to develop a better mousetrap? At 65% apparently nobody is that great at it and there is a vast amount of advantage that it would be possible to obtain. I think there is alot of evidence that the O's have actually been worse than other teams at this analysis over recent history. We seem to hang on to the wrong guys and let the right guys go at an alarming rate. A team that could reliably predict these things may have traded Bigbie for something after his initial success, maybe they still have Maine, Julio may have turned into a guy who is a star now.

No, not at all...it still comes down to common sense....If a guy is awful against AA pitching and is old enough for the league, he isn't likely to be good.

If a guy is in the minors and tears it up, he is likely to be at least a decent player in this league.

Who cares what the % is? That is just ridiculous...What are you basing success off of? For how many years? What stats?

It is just common sense that better performance is likely to carry over than worse performance.

[square634]

My guess is that the latter is true. While the best predictive methods may be 65% reliable, that’s not what people here typically use when discussing MiL guys. Instead, they use superficial stats whose predictive reliability is likely to be way worse than the more complex predictive methods. So if the best complex methods are only 65% reliable, and we’re using simple stats that are worse than that, how much worse than 65% can you go without getting to the reliability of coin-flips?

I think the 65% number is being used out of context in this thread. A .65 coefficient of correlation (which I think was the statistic in the article) has nothing to do with a probability or coin flips. It just means that there is a fairly strong, positive correlation between past minor league statistics and future major league statistics.

My completely-unsubstantiated hunch is that if we knew what the “Reliability Curve” looked like, it would be a lot like an upside-down bell curve. Where a normal bell curve’s highest point is in the middle and its lowest points are out at the far ends, I expect that the Reliability Curve would be the opposite, with its lowest point in the middle and highest points at the far ends. The more radically-good a player’s MiL numbers are, the more confident we can be that his MiL stats are reliable predictors (of success). The more stinkily-bad they are, the more confident we can be that they are reliable predictors (of failure). The more of a dilemma a guy is, the least trustworthy the predictions will be. So at the very middle of the Reliability Curve, right where we find the guys who are the biggest dilemmas, the curve reaches its bottom at a point where it approaches the reliability of a coin-flip, which means that it tells us exactly what we already know anyway: that the guy is truly a dilemma, and he may or may not succeed. But I’m just guessing….

I think there may be some merit to this. If you have Alex Rodriguez's statistics in the minor leagues, it's clear that you will be able to adjust to ML pitching. If you have Luis Hernandez's minor league statistics, it's clear that if you are overmatched at AA, you will be even more overmatched in the major leagues. However, even if reliability decreases in the case of more average players, I think what is important is that their probability of success in the majors is still higher than someone who is terrible in the minors, even if we are less sure what their range of performance in the majors will be.

Now, if the Orioles had shown more aptitude over the last 10 years at choosing players from the minors based on scouting, etc. instead of what the statistical projections said, I think you would see much less argument about the O's roster management. However, since what they have done has clearly not worked, AND the probabilities based on minor league numbers seem to be ignored by the Orioles, I think it is natural that people would question the Orioles' thinking.

I make this analogy a lot, and it isn't perfect since football is less suitable than baseball for statistical analysis, but if the Ravens make a questionable roster decision or draft choice based on the opinions of so-called "experts" like Mel Kiper, I would not be apt to criticize them. Based on past success, Ozzie Newsome and the personnel department are clearly some of the best in the business, so I trust that they have evaluation tools more reliable than anything I or the media have at our disposal. At this point, the Orioles do not deserve that benefit of the doubt.

[Mackus]

Who cares what the % is? That is just ridiculous...What are you basing success off of? For how many years? What stats?

If that percentage could be accurately quanitified, it would be incredibly useful.

If a good prospect has a 65% chance of becoming a good mlb player, you then have a better idea of exactly how deep your system is. If you have 6 offensive prospects that are good prospects, you know that within a couple years you will probably have 4 pretty good MLB hitters.

[rolliefingers]

I think the 65% number is being used out of context in this thread. A .65 coefficient of correlation (which I think was the statistic in the article) has nothing to do with a probability or coin flips. It just means that there is a fairly strong, positive correlation between past minor league statistics and future major league statistics.

Yeah, this needs to be acknowleged. 0.65 is the STANDARD DEVIATION for the numbers listed, not the PROBABILITY between two discrete outcomes (i.e. good MiL numbers succeeding or failing in ML).

Still, though, .65 means the numbers are more similar than they are different, correct?

As Drungo has said many times, MiL numbers is the worst method of predicting ML success...except for all the other ones.

[Sports Guy]

If that percentage could be accurately quanitified, it would be incredibly useful.

If a good prospect has a 65% chance of becoming a good mlb player, you then have a better idea of exactly how deep your system is. If you have 6 offensive prospects that are good prospects, you know that within a couple years you will probably have 4 pretty good MLB hitters.

But again, how are you defining good offensive prospects?

And how did the people who set that 65% number define?

Many people on here seem to think all we do is look at OPS and say this guy will or will not be good.

There is a lot more that goes into it than that.

I don't need someone to tell me a % to know how many of my guys may make it.

Properly analyzing stats, ballpark and league factors, age and scouting reports can give me a better idea than somearbitrary % number that someone came up with.

[Flosman]

No, not at all...it still comes down to common sense....If a guy is awful against AA pitching and is old enough for the league, he isn't likely to be good.

If a guy is in the minors and tears it up, he is likely to be at least a decent player in this league.

Who cares what the % is? That is just ridiculous...What are you basing success off of? For how many years? What stats?

It is just common sense that better performance is likely to carry over than worse performance.

Well in a black and white world you are correct. But in the world a major league team has to operate in there is a heck of alot of grey. Improving the percentage is what would make a team be correct more often than other teams on player/personel decisions. That is what is common sense.

BTW the point you are missing is your "common sense" approach is wrong alot, Shack is saying that there is a need for a better system. Improvements in modeling NEVER happen until someone takes the position that the current Model is not good enough and looks for a better one. You like to preach about the team becoming creative and innovative in order to compete with the AL juggernuts, however you want to ridicule someone who is pointing out something creative and innovative the the team could do to make a difference. The process that Shack is proposing is no differnet than the one that the innovators of the current sabremetric measures used to show that the old triple crown method of looking at stats was seriously flawed in evaluating players. People proposing views like Shack has are the ones that actually lead organizations to break throughs, not the ones that say what we have now is the best that we have and to indentify and measure its flaws goes against "common Sense".

[Lucky Jim]

My completely-unsubstantiated hunch is that if we knew what the “Reliability Curve” looked like, it would be a lot like an upside-down bell curve. Where a normal bell curve’s highest point is in the middle and its lowest points are out at the far ends, I expect that the Reliability Curve would be the opposite, with its lowest point in the middle and highest points at the far ends. The more radically-good a player’s MiL numbers are, the more confident we can be that his MiL stats are reliable predictors (of success). The more stinkily-bad they are, the more confident we can be that they are reliable predictors (of failure). The more of a dilemma a guy is, the least trustworthy the predictions will be. So at the very middle of the Reliability Curve, right where we find the guys who are the biggest dilemmas, the curve reaches its bottom at a point where it approaches the reliability of a coin-flip, which means that it tells us exactly what we already know anyway: that the guy is truly a dilemma, and he may or may not succeed. But I’m just guessing….

This seems to me intuitively correct. And elegantly stated (except for the word 'stinkily').

[Lucky Jim]

If that percentage could be accurately quanitified, it would be incredibly useful.

If a good prospect has a 65% chance of becoming a good mlb player, you then have a better idea of exactly how deep your system is. If you have 6 offensive prospects that are good prospects, you know that within a couple years you will probably have 4 pretty good MLB hitters.

Exactly. It allows you to understand your risk better. Further, if you have four players with a higher likelihood of success than 65% you can take a few long-shots as system filler (i.e, an older player with legit power who's repeating at AA.)

By accurately calibrating the likelihood of success of an individual player, you can invest more efficiently.

[Tony-OH]

But again, how are you defining good offensive prospects?

And how did the people who set that 65% number define?

Many people on here seem to think all we do is look at OPS and say this guy will or will not be good.

There is a lot more that goes into it than that.

I don't need someone to tell me a % to know how many of my guys may make it.

Properly analyzing stats, ballpark and league factors, age and scouting reports can give me a better idea than somearbitrary % number that someone came up with.

This may be the best thing said in this long thread. The reality is that they are all tools to be used to evaluate a prospect in the hopes that he will perform in the big leagues to as close as possible to his expectations.

[Fan4Life]

It is just common sense that better performance is likely to carry over than worse performance.

Actually, what you meant to say is they are both as likely to carry over which is why you chose the guy with better performance.

[Lucky Jim]

This may be the best thing said in this long thread. The reality is that they are all tools to be used to evaluate a prospect in the hopes that he will perform in the big leagues to as close as possible to his expectations.

Isn't that just intuitive, though?

Obviously, analysis should be macro- AND micro-. Like any market (and baseball talent is a market) where one's holdings are comprised of individual investments, it's important to have an idea of what to expect on a big-picture level AND an idea of what to look at individually.

But I'm not sure that anyone's claimed these are mutually exclusive. A probabilistic curve that tells you how a player's performance in minors should translate to the majors doesn't absolve a team of looking at individual factors. It just allows them to more accurately analyze what they are holding, in terms of value, at any given time.

[Mackus]

Isn't that just intuitive, though?

Obviously, analysis should be macro- AND micro-. Like any market (and baseball talent is a market) where one's holdings are comprised of individual investments, it's important to have an idea of what to expect on a big-picture level AND an idea of what to look at individually.

But I'm not sure that anyone's claimed these are mutually exclusive. A probabilistic curve that tells you how a player's performance in minors should translate to the majors doesn't absolve a team of looking at individual factors. It just allows them to more accurately analyze what they are holding, in terms of value, at any given time.

Exactly, knowing the numbers behind how often and how well a player's minor league stats mirror their major league stats after adjustments would be a great tool for analyzing the overall depth of your system. Its not going to help much for individual players, there are still massive fluctuations, but it would be a great tool for gauging what the strengths and weaknesses of your system are.

[Witchy_Chick]

I see some things getting lost in the shuffle (again).

To summarize what I believe this entire thread is about:

RShack (and others, including myself) take exception to the certainty with which a number of posters here quote stats this and stats that to praise or bash certain players. (These stats may pertain to MiL players and how they might fare in the majors (i.e. Knott/House/Penn/etc), or they may pertain to ML players and how they might (or should) fare from year-to-year based on peripherals (i.e Trax).)

RShack questioned this be-all-and-end-all use of stats, and pretty much got blasted for it. A number of people have attempted to answer his question, by using the wrong information. If I have this correct, he is looking for an accuracy percentage. Of the players that translate from MiL -> ML, how many of them hit the established translation metric? The answer to that question isn’t “thousands of players have been studied and AAA players promoted to ML will produce about 90% (or whatever the percentage) of their AAA stats.” The question is how accurate is that 90% figure. Do all AAA players hit that 90% translation? Do 80% of them? Less?

Another answer provided has been this “65%" figure. Looking at that article linked, the 65% that some people appear now to be taking as gospel refers to “the success rate for a typical weighted three-year projection system like Marcel [is] at about 65 percent.” So, based on one projection system that someone analyzed and found about a 65% success rate, we close the book on any other projection formulas?!? Marcel success ≈ about 65% -- so throw out PECOTA, forget any other projection sites/formulas. To me, that 65% is far from gospel. How many projection formulas are there in the world of baseball statistical analysis? What are their success rates?

I don't know PECOTAs from Marcels, but this is fast becoming a very interesting topic to me. Let's all just keep an open mind, not shoot the messenger, and see if we can figure out some new way to do business.

Witchy

[Frobby]

This may be the best thing said in this long thread. The reality is that they are all tools to be used to evaluate a prospect in the hopes that he will perform in the big leagues to as close as possible to his expectations.

Well, there is an interesting philosophical question here that underlies many, many discussions we have on this board: are you better off just playing the statistical odds as you see them, or are you better off trying to distinguish the exceptions.

Let me explain. Assume there are two players. One falls into a series of statistical categorties that, without any other input, would lead you to believe that he has a 65% chance of being a successful major leaguer. The other fits into a series of categories that, without more, would lead you to believe that he has only a 35% chance of success.

One school of thought says, always prefer Player 1 to Player 2. In the long run, you will make mistakes, but you will be right more often than you will be wrong. You will get more hits than misses.

The other school of thought says, look at other stuff and try to sort the 65% of Type A who will be successful from the 35 % who won't be. And try to pick out the 35% of Type B who will be successful and give them a shot.

I would say the Orioles are definitely in the second school of thought. A lot of the critics on this board are in the first school. Which is better? Well, let's just say that whatever the Orioles have been doing, it hasn't been working.