Two Myths I'm Ready to Debunk

[Sports Guy]

How long do you give a guy to know whether or not he can reach the level of success you expect from him?

This depends on many factors.

Bedard was stuck with quite awhile, DCab as well

DCab has shown the signs, before this year, that he was ready take the next step...he fell back this year. Bedard had an injury and in his first year back(1 year removed from TJ surgery), he was already a league average guy and just got better from there.

others, Knott, Cust weren't given much time.

Because of stupidity.

From what we saw of our minor leaguers this season, we were quite disappointed in just about all of them. That's not to say that next season or the season after that some of them or all of them couldn't fulfill our expectations of them, but how long do you wait? Is it strictly an age factor?

I am not sure there is really any MiLer to be really disappointed with except Erbe and Beato. Other than that, i am satisfied this year.

[Flosman]

SG you are missing the point Shack and a couple of others here are trying to make. There is no logical reason that a predictive system can not have a corrolation coefiecent higher than .65. It is a very difficult system to Model but .65 is not a high enough number to claim much skill in forcasting. I know it is the best to this point in time, but really your position on this subject is no different than the old school scouting gaurd defending itself against the sabremetric group. Because people are asking these questions better systems will be developed over time and being on the forefront of this will help an organization. Say Drungo, in his spare time, develops a system that raises that number to .85 and only one team has access to this information. That team will consistantly out perform all of the other teams and over time it will be by a large margin. The team will make the right call on players earlier and more often correctly than the other teams. This is the reason that it is not meaningless

[nadecir]

Seriously though, I don't see any reason why one school of thought should be used. Smart teams use every tool to their advantage and that means using statistical analysis and tradtional scouting.

Smart teams do things like hire Bill James in their scouting department. And smart teams follow Bill James' advice when he recommends signings like David Ortiz.

http://en.wikipedia.org/wiki/Bill_James

Smart teams take a look at their farm system, and if the farm system has been bad for a number of years, they replace the team's scouts, and invest by expanding the number of people in the scouting department.

http://sonsofsamhorn.net/index.php?showtopic=16978

Smart teams invest in international player development, and find gems like Hideki Okajima in international markets.

It is unfortunate that we have smart teams in our division.

[Sports Guy]

SG you are missing the point Shack and a couple of others here are trying to make. There is no logical reason that a predictive system can not have a corrolation coefiecent higher than .65. It is a very difficult system to Model but .65 is not a high enough number to claim much skill in forcasting. I know it is the best to this point in time, but really your position on this subject is no different than the old school scouting gaurd defending itself against the sabremetric group. Because people are asking these questions better systems will be developed over time and being on the forefront of this will help an organization. Say Drungo, in his spare time, develops a system that raises that number to .85 and only one team has access to this information. That team will consistantly out perform all of the other teams and over time it will be by a large margin. The team will make the right call on players earlier and more often correctly than the other teams. This is the reason that it is not meaningless

But that number is going to come from things like that....:

Age

BB rate

K rates

HR rates

OPS

league factors

park factors

scouting reports

age comps

Those are the types of things that the % number should come from.

So, if you are using that formula and using those things for some kind of formula, why on earth do you need someone to tell you that 70% of the time the player will be productive with those factors being on his side?

Just seems kind of obvious to me.

[RShack]

Who cares what the % is? That is just ridiculous...What are you basing success off of? For how many years? What stats?

It is just common sense that better performance is likely to carry over than worse performance.

It's also "just common sense" that a Cat-5 hurricane is more dangerous than is a Cat-2 hurricane. If all other things are equal, we know that a Cat-5 hurricane will do more damage than a Cat-2 hurricane. But that does not mean that efforts to develop good models of hurricane prediction are "just ridiculous". Nor is it ridiculous to acknowledge that current predictions methods are seriously inadequate. Nor is it ridiculous for people to *know* the degree to which they should trust existing models to be flat-out wrong. Hurricane guys will *emphasize* all these things. Most hurricane guys are *happy* to tell you how *wrong* they are. In contrast, baseball guys seem to want to conceal and/or ignore these things.

I think the hurricane analogy is useful here, because several things about predictive hurricane models are very similar to predictive baseball models:

• In both cases, the problem is extremely hard.
• In both cases, one big part of the problem is that we don't have enough of the necessary data.
• In both cases, we're not even sure what the set of "necessary data" includes.
• In both cases, we're not sure how to best use what data is available (ref: the Masters Thesis by the K-State stats guy).
• In both cases, better models have valuable implications.
• In both cases, people routinely misuse what the best existing models tell us. (With hurricanes, it's the TV-guys getting everybody terrified about a projected hurricane path 4-days out when we *know* that the average error of existing models is about 100-miles per day.)

There are two BIG differences between baseball models and hurricane models:

• The guys who do hurricane models are quite happy to tell you what their flaws are. They are quite happy to give you rules of thumb about how *wrong* they are. In contrast, the guys who do baseball models evidently avoid talking about how wrong their models are, to the point where we can't even readily find any reliability info about them whatsover. Maybe part of that is because the hurricane guys are not charging subscription prices for their models and therefore have no economic reason to hide their poor reliability. Maybe it's like Ford having economic reasons to not tell people how badly the Pinto sucked. Or maybe there are other reasons. Maybe there are good non-selfish reasons. But I can't think of what they could be. When used responsibly, one of the things that stats are best at is telling you how much you should-or-shouldn't trust them. It's flat-out bizarre that people who do baseball stats don't include this kind of information.
• When people talk about the inadequacy of hurricane models, nobody takes it as a big insult and starts hammering you about it. People who are weather freaks readily accept the facts about how much to NOT trust hurricane models. Too bad it's not true about baseball models.

My impression is that most of us here are not "students" of baseball stats, in the sense that students really study a field. I think it's more correct to say that most of us are "consumers" of baseball stats. An "informed consumer" uses products wisely. Stats are a tool, and an informed consumer of tools does not use a hammer when what he needs is tweezers. The underlying issue here is that many of us routinely use baseball stats as a hammer, i.e, they use what can be the right tool inappropriately, i.e., they use stats for the wrong job. In large part, I think this is because people don't realize the degree to which the best predictive baseball models can be expected to be wrong. This appears to be largely due to the fact that the people who do baseball models don't tell us what we need to know about how wrong they typically are. Instead, they seem to want us to think that people who use baseball stats as predictors will be greeted as liberators ;-)

[RShack]

I think the 65% number is being used out of context in this thread. A .65 coefficient of correlation (which I think was the statistic in the article) has nothing to do with a probability or coin flips. It just means that there is a fairly strong, positive correlation between past minor league statistics and future major league statistics.

I agree that the 65% number is probably a correlation coefficient (even though we really don't know what it is). I agree this is not the same thing as a simple binary probability. However, I disagree that using coin-flips as a reference point is not useful. It is useful. For any given binary choice (like success/failure), a coin-flip will give you the right answer half-the-time. A correlation of .65 is similarly weak. It is *not* true to say that .65 is a "fairly strong" value for prediction. It is not. It is well below any standard threshold for trustworthy predictions. The fact that it is strong when comparing the predictive utility of baseball stats is simply a reflection of how weak baseball stats are as predictors. [This is NOT diss'ing people who do good work on baseball stats. It is simply pointing out how hard the problem is.]

even if reliability decreases in the case of more average players, I think what is important is that their probability of success in the majors is still higher than someone who is terrible in the minors, even if we are less sure what their range of performance in the majors will be.

But that's not what we use them for. We use them for guys who are neither terrible nor great. We use them for the dilemma-cases. I believe that it is precisely for the dilemma-cases where they tell us the least, and are probably about as useful as coin-flips.

I think it is natural that people would question the Orioles' thinking.

Oh, I agree with this. But that doesn't mean that we should make other blatant mistakes... especially when we know that we are making fundamental mistakes.

[RShack]

Yeah, this needs to be acknowleged. 0.65 is the STANDARD DEVIATION for the numbers listed...

We're not sure exactly what .65 is, since the guy never said. But it is most likely a correlation coefficient I can't imagine that it would be standard deviation.

Still, though, .65 means the numbers are more similar than they are different, correct?

This is exactly where people get into trouble using stats. This does not say they have much predictive utility.

[Flosman]

But that number is going to come from things like that....:

Age

BB rate

K rates

HR rates

OPS

league factors

park factors

scouting reports

age comps

Those are the types of things that the % number should come from.

So, if you are using that formula and using those things for some kind of formula, why on earth do you need someone to tell you that 70% of the time the player will be productive with those factors being on his side?

Just seems kind of obvious to me.

Well just how obvious is it really. Almost any dolt can identify the Arods of the baseball world where all of these factors are overwelmingly positive. The same can be said for the other end of the spectrum. Now just how to you handle the guy that has mixed evidence? Go by a hunch? I doubt you would advocate that. You would wiegh what you deem more and less important and make what you think is the best decision. However wouldn't it be smarter to find what ties to actual future results and measure those things and then use that data to assist in decision making. That is all the other side of this debate is suggesting.

Also I would like to point out that the pupose of developing a better modeling system is not to have it tell you that a player will be good 70% of the time with this set of data. It is to be able to predict what any player in your system will likely do with a greater degree of certainty.

[square634]

But that's not what we use them for. We use them for guys who are neither terrible nor great. We use them for the dilemma-cases. I believe that it is precisely for the dilemma-cases where they tell us the least, and are probably about as useful as coin-flips.

I think it is much more reliable than coin flips, but ignoring that for a second, the point isn't that everyone knows a player from the Jack Cust/JR House/Jon Knott mold will put up an X OPS in the major leagues. The point is that Jack Cust/JR House/Jon Knott have never been given a chance and will/have almost certainly put up a higher OPS than the alternatives.

[Sports Guy]

Well just how obvious is it really. Almost any dolt can identify the Arods of the baseball world where all of these factors are overwelmingly positive. The same can be said for the other end of the spectrum. Now just how to you handle the guy that has mixed evidence? Go by a hunch? I doubt you would advocate that. You would wiegh what you deem more and less important and make what you think is the best decision. However wouldn't it be smarter to find what ties to actual future results and measure those things and then use that data to assist in decision making. That is all the other side of this debate is suggesting.

Also I would like to point out that the pupose of developing a better modeling system is not to have it tell you that a player will be good 70% of the time with this set of data. It is to be able to predict what any player in your system will likely do with a greater degree of certainty.

But you are going to come up with that system by using all the stuff i have said.

Look, if you need a % to tell you these things, that's fine. If Shack needs that, that is fine as well.

I am ok not knowing some arbitrary % that i have no idea how they come up with it and instead i will use all the tools i have and evaluate that way.

[RShack]

SG you are missing the point Shack and a couple of others here are trying to make. There is no logical reason that a predictive system can not have a corrolation coefiecent higher than .65. It is a very difficult system to Model but .65 is not a high enough number to claim much skill in forcasting. I know it is the best to this point in time, but really your position on this subject is no different than the old school scouting gaurd defending itself against the sabremetric group. Because people are asking these questions better systems will be developed over time and being on the forefront of this will help an organization. Say Drungo, in his spare time, develops a system that raises that number to .85 and only one team has access to this information. That team will consistantly out perform all of the other teams and over time it will be by a large margin. The team will make the right call on players earlier and more often correctly than the other teams. This is the reason that it is not meaningless

Thank you ;-)

Here's something else from that ESPN link that we might want to consider:

With all these tools at their disposal, you might expect the experts to achieve huge success rates, routinely nailing the vast majority of their projections. But various studies, done by industry leaders and outsiders alike, peg the success rate for a typical weighted three-year projection system like Marcel at about 65 percent. The goal for primo projectionists is to eke out a bit more accuracy, for a year-to-year success rate approaching 70 percent.

This says that the *goal* is to improve to somethng "approaching 70%", i.e., high-60's. This not only indicates that the people involved realize how hard the problem is, it also indicates that they seem to think the problem is intractable given available data. I say this because even if they reach high-60's, they've still got a predictor that is not strong. Again, one of the rough rules of thumb is that you need to be in the 80's to reach the very minimum threshold of a reliable predictor, and that you need to be in the mid-to-high 90's in the engineering world of building bridges and stuff. I think the point that people should take from this is that it is folly to think that you can truly take an "engineering" approach to building a baseball org. As many people have said, stats are one tool in the toolbox.

If somebody was gonna pay me to work on this, I'd pursue three thngs:

• I'd have student-assistants pursue kind of thing the K-State guy was doing but on a larger scale.
• I'd put a big focus on using neural nets to tackle this, just to see what they found.
• I'd borrow heavily from the specific kind of AI work that one of my former-colleagues is doing with DNA.

Before somebody says, "Why don't you?", I want to point out that doing these things is a more-than-full-time job. It's a 60-hours-per-week kind of job, not a hobby. In other words, it'd be great fun to do if the O's would pay for it ;-)

[RShack]

Look, if you need a % to tell you these things, that's fine. If Shack needs that, that is fine as well.

I am ok not knowing some arbitrary % that i have no idea how they come up with it and instead i will use all the tools i have and evaluate that way.

That makes sense. But the tools you have do include some knowledge about the lack of reliability of baseball stats as predictive tools. You may (or may not) choose to use that knowledge as well. It appears to me that you choose not to. It appears to me that you choose not to largely because, if you did, it would decrease the illusion of certainty that you wish to have. (I could be wrong, I'm just going by your usual stance of great certainty that your opinions are correct and that those who disagree are stupid.)

[Sports Guy]

That makes sense. But the tools you have do include some knowledge about the lack of reliability of baseball stats as predictive tools. You may (or may not) choose to use that knowledge as well. It appears to me that you choose not to. It appears to me that you choose not to largely because, if you did, it would decrease the illusion of certainty that you wish to have. (I could be wrong, I'm just going by your usual stance of great certainty that your opinions are correct and that those who disagree are stupid.)

I fully acknowledge that there is no certainty and some kind of % in the 65-70% seems reasonable to me.

Again, no matter the %...Whether it be 55% or 80%, i am still going to get the players and go with the players that do well in the categories i mentioned.

[RShack]

I think it is much more reliable than coin flips...

Why?

I realize that this is a popular opinion. I don't see why people have it. I think most of it is due to the fact that people really do want something to guide them, and that stats are all that people have available on the web. So, I think it's mainly a case of "if all you have is a hammer, then you treat the world like it's a nail". This is understandable. However, I don't know of any reason whatsoever to think that it's better than a coin flip for the "dilemma guys". For Babe Ruth, sure... but that's not who we wonder about...

...but ignoring that for a second, the point isn't that everyone knows a player from the Jack Cust/JR House/Jon Knott mold will put up an X OPS in the major leagues. The point is that Jack Cust/JR House/Jon Knott have never been given a chance and will/have almost certainly put up a higher OPS than the alternatives.

I think that many posters do treat MiL stats as a reliable predictor for dilemma guys. I think they do this because they "feel sure" that MiL stats are reliable predictors even though there is no reason (that I know of) to think that. The very idea that there is "almost certainty" about the ML success of House/Knott is evidence that people are using MiL stats in a way that AFAIK is not justified.

PS: I don't know if you noticed, but in the Masters Thesis by the K-State guy, he ranked Cust as the #1 prospect for ML success from the entire group of 2002 MiL hitters. This was based on his analysis of 2002 MiL data vs. what had actually happened with those guys by 2005. I thought it was very interesting (although it was a pain in butt to read).

[BRobinsonfan]

One way to test RShack's theory is to simply look back at established major league players and see what their minor league stats are like. If the "stats guy" theory holds up - one would expect them to have good minor league numbers. If RShack's theory holds up - one might expect them to have a variety of performance records in the minor league's.

While the following research is by no means as comprehensive and as detailed as what Drungo and others would do - it's at least a quick snap shot on the question.

I did this in about 20 minutes using baseball reference minor league stats. I simply took three established players from the Orioles, Yankees and Red Sox and looked back at their minor league performance. The hypothesis is that minor league stats are a "reliable" predictor of major league performance. Without getting into the definition of reliable - I'd say the following quick example supports that contention.

Before people jump on me I'll mention some glaringly obvious problems with the examples below. 1) It is, of course, an extrememly small sample size; and 2) I didn't bother to take out minor league stats that were accumulated after someone made the majors as part of a rehab assignment or for some other reason. (In most cases this was extremely small anyway) And finally, I just grabbed batting average and OBP under the theory that since power usually develops later that it wasn't necessarily as important as other stats for minor league purposes. I also avoided it because I didn't want to get into park/league factors and stuff like that. Leaving it out may have been a mistake... However, most of the following had a +800 OPS in the minors if I remember correctly.

At any rate - here they are:

BA OBP

Miguel Tejada .272 .347

Brian Roberts .281 .376

Nick Markakis .301 .381

David Ortiz .310 .381

Manny Ramirez .307 .404

Mike Lowell .295 .359

Derek Jeter .308 .380

A. Rodriguez .327 .386

Jorge Posada .261 .364

The above, of course, is not meant to be definitive - but merely to add to the discussion. I'll play around some more with the minor league stats on baseball reference as I have time.