PECOTA projections

[DrungoHazewood]

3) To my knowledge no system has ever been able to beat PECOTA over a decent sample size, but I could be remembering incorrectly. I certainly know that no system with human input has ever been able to beat even fairly simple model projections. It just turns out a lot of what we view as common sense is incorrect.

I think PECOTA does no better than many other systems and rarely if ever takes the top spot. It's only marginally better than Marcels, which just weights the last 3-4 years adds a touch of aging and assumes all rookies are average. But no system dramatically out-performs Marcels.

From 2007-2010 Marcels did about as well as any system.

[Phantom]

I think PECOTA does no better than many other systems and rarely if ever takes the top spot. It's only marginally better than Marcels, which just weights the last 3-4 years adds a touch of aging and assumes all rookies are average. But no system dramatically out-performs Marcels.

Yeah, I think you are right. While the human systems can't keep up, I was surprised to see that Vegas projections actually did pretty decent.

[DrungoHazewood]

Yeah, I think you are right. While the human systems can't keep up, I was surprised to see that Vegas projections actually did pretty decent.

I used to be a big champion of PECOTA. You can probably search here and find some embarrassing posts I made years ago. It makes sense - it establishes detailed comps, like using big first basemen with high K rates to project Chris Davis. But the results weren't any better than a very basic system. There's no magic, for all of PECOTA's crunching of comps and data and historicals and body types and whatever, it's really no better than a super-simple 3-2-1 averaging of recent history, regression and aging.

[Frobby]

I wish I had saved all of the different studies I've seen about projection systems.

PECOTA projects playing time based on depth charts, so yes, technically at this point they are projected depth charts but eventually they will be the official ones. 538 seems to think that PECOTA is doing better than ever on individual and playing time predictions:

http://fivethirtyeight.com/features/is-2015-the-year-baseballs-projections-failed/

Of course, it says they're doing worse on team projections. Interesting article.

From my point of view as a fan, the more unpredictable baseball is, the more interesting it is. The years where you could just pencil New York and Boston into two playoff spots are over, and that's a good thing.

[bird watcher]

I wonder if they have looked at which data measures lead to smaller deviations from the prediction? In other words to teams whose wins rely more on homeruns in general have smaller or larger deviations? Perhaps teams with more players similar to Jason Heyward where a ton of their win projection revolves around defense have a smaller or larger deviation?

Sometimes certain factors make a weather forecast inherently less valuable and sometimes the predictive value is much higher.

[DrungoHazewood]

I wonder if they have looked at which data measures lead to smaller deviations from the prediction? In other words to teams whose wins rely more on homeruns in general have smaller or larger deviations? Perhaps teams with more players similar to Jason Heyward where a ton of their win projection revolves around defense have a smaller or larger deviation?

Sometimes certain factors make a weather forecast inherently less valuable and sometimes the predictive value is much higher.

"They" have absolutely run component-level regressions.

[bird watcher]

"They" have absolutely run component-level regressions.

Do they give varying deviation projections with teams? If two teams are both projected at 85 wins it would seem possible that one of those might have a higher chance of significant deviation than the other. I would think the Orioles have a huge range in their projection deviation given the fluctuation in pitching performance over the past two years.

[jsbearr]

I used to be a big champion of PECOTA. You can probably search here and find some embarrassing posts I made years ago. It makes sense - it establishes detailed comps, like using big first basemen with high K rates to project Chris Davis. But the results weren't any better than a very basic system. There's no magic, for all of PECOTA's crunching of comps and data and historicals and body types and whatever, it's really no better than a super-simple 3-2-1 averaging of recent history, regression and aging.

Just to be clear...around 2013 PECOTA was effectively reimagined. A lot has changed, but it still is relatively the same with accuracy and precision.

[Phantom]

Of course, it says they're doing worse on team projections. Interesting article.

From my point of view as a fan, the more unpredictable baseball is, the more interesting it is. The years where you could just pencil New York and Boston into two playoff spots are over, and that's a good thing.

Oh, I completely agree. At the same time, I think it's cool that statistical models can predict such a chaotic and emotional system better than humans. I think we would be fools to ignore the predictions just because we want them to be wrong about our team. For a few years, it seemed possible that we had discovered something the "systems" didn't know. A way to consistently do the inconsistent. Last year should have disabused us of that notion.

[SilentJames]

Hey everyone, it is June 16 and the Orioles are already more than halfway to their PECOTA projection!

Sent from my iPhone using Tapatalk

[OFFNY]

Hey, everyone.

It is June 16th, and the Orioles are already more than halfway to their PECOTA projection!

Not bad.