Evaluating April MLB Predictions (2008)

Discuss all things Cardinals Baseball
Post Reply
User avatar
fanforever
Retired
Posts: 67648
Joined: April 17 06, 5:03 pm
Location: Northern NJ-The Twilight Zone

Evaluating April MLB Predictions (2008)

Post by fanforever »

http://vegaswatch.net/2008/09/evaluatin ... -2008.html
We did this last year, in a couple parts. It quickly became obvious that the computer projections are much more accurate than those of ESPN's analysts; not at all surprising, considering the various methods used. But PECOTA also fared well against the sabermetrically-inclined analysts, like Joe Sheehan, Keith Law, and Rob Neyer.

For now, I'll use 12 sets of predictions: six from ESPN (Stark, Kurkjian, Olney, Law, Phillips, Neyer), two from BPro (PECOTA, Joe Sheehan), three from Yahoo! (Steve Henson, Tim Brown, and Jeff Passan), and the over/unders. After the season is over, I'll have a post incorporating a few more computer projections, and comparing how everyone did both this year and over the last couple. For now, here are some of the best and worst individual predictions, as well as whose overall predictions were the most accurate.

(Note: These lists aren't just based on who was the closest- I also factored in how far off the other predictions were. So predicting at team within two games if the average prediction was eight games off would be higher than predicting a team exactly if the average was just three games off.)

The Best

1. PECOTA, Tampa Bay Rays
Predicted wins: 88
On pace for: 97.4

Well, this was a pretty good call, wasn't it. According to the metric I've come up with to compare these projections, this was the second best one over the last two years, behind only PECOTA's 2007 White Sox projection. Tampa Bay has surpassed even the most optimistic of expectations; even if you just look at their Pythagorean record, of 87-66, they're on pace to beat PECOTA's projection by four games.

Also close: Nobody in this field.

2. PECOTA, Seattle Mariners
Predicted wins: 75
On pace for: 60.0

As we'll see later, it was a very good year for PECOTA. Both the Tampa Bay and Seattle projections seemed somewhat insane six months ago, but as it turns out they actually weren't extreme enough. PECOTA may not have nailed this one exactly, but it saw at least some of this regression coming, and knew they were going to take a large step back from 2007. Others weren't quite as prescient.

Also close: Neyer, 77.

3. Steve Henson, Houston Astros
Predicted wins: 81
On pace for: 85.2

Henson was much more optimistic about the Astros than anyone else, except maybe the great Ed Wade. He ends up looking good here, but this was mostly luck, as Houston's Pythag (73-81) is the inverse of their actual record (81-73). While that's a good thing for Henson, it's probably bad for the Astros in the long run, as it gives them the mistaken impression that they're close to being a playoff-caliber squad.

Also close: Nobody.

4. Jayson Stark, Chicago White Sox
Predicted wins: 85
On pace for: 89.4

We're left to wonder why Stark thought the White Sox would bounce back--I kind of doubt it was because he thought Quentin would have a 148 OPS+ and Danks would suddenly transform into an ace--but this is impressive nonetheless. Chicago got a bit of "revenge" on PECOTA this year, which had them winning just 77 games.

Also close: Phillips (84)

Now for the fun part...

The Worst
Image
1. Steve Phillips, Seattle Mariners
Predicted wins: 92
On pace for: 60.0

Wow. This is about as wrong as you can possibly be. Phillips missed by 32 games, which is just an incredible amount. Think about how many things had to happen for him to be this far off:

* Steve Phillips, despite doing a terrible job as GM of the Mets, gets hired by ESPN. They let him go on TV and give his "analysis", as well as make predictions. In fact, they pay him a large sum of money to do these things. Possibly the most confounding occurrence of this whole process.

* The Mariners outplay their Pythag by nine games in 2007, going 88-74 despite being outscored by 19 runs. Many people get the false impression that they're a contender heading into 2008.

* Seattle trades a large portion of their farm system for Erik Bedard, causing some to believe they're now the favorite in the AL West.

* Even compared to the most pessimistic expectation, everything goes horribly wrong for the Mariners. Silva has a 65 ERA+. Bedard makes just 15 starts, and is only slightly above average when he does pitch. Their DH hits .234/.274/.338. Miguel Cairo starts 36 games at first base. And so on.

* Blogger writes post on meaningless preseason predictions.

It really has been an incredible ride. Cherish this moment; it's possibly nobody will ever be this wrong again.

Also very wrong: Kurkjian (91), Passan (91).

2. Steve Phillips, Texas Rangers
Predicted wins: 64
On pace for: 78.4

I really have no idea how he came up with this one; it didn't make sense prior to the season, and it still doesn't in late September. They won 75 games in 2007, and were outscored by only 28 runs (79-83 Pythag). Phillips wasn't nearly as far off on this one, but it's almost as bad as his Seattle prediction, since it's not like the Rangers shocked the world by playing .480 baseball.

Also very wrong: Nobody.

3. Steve Henson, Tampa Bay Rays
Predicted wins: 72
On pace for: 97.4

So, Mr. Henson, what do you think of Tampa's 2008 outlook?

"The Rays are improving but are still middle-school level to the Red Sox graduate students."

Truly enlightening. At least he wasn't the only one who didn't see this coming.

Also very wrong: Brown (73), Kurkjian (75), Olney (75)

4. Buster Olney, Baltimore Orioles
Predicted wins: 56
On pace for: 70.9

Olney did the same thing with the Nationals in '07, predicting they'd lose 113 games, and claiming the top spot here when they lost just 89. It seems as though he just gets totally caught up in the story--"the Orioles are going to be really bad; how bad? well look, I'm going to predict they lose 106 games!"--without realizing how silly it is to make such extreme predictions.

Also very wrong: Sheehan (57)

Here are the overall standings, using RMSE (lower is better, obviously):
Image

You may notice that all these numbers are a good deal higher than they were last year, when the average was 7.12. It turns out that 2007 was historically easy to predict; the average here (11.37) is a little higher than '05-'06, but not that much.

The top two are the same as last year, with Neyer and PECOTA ahead of the pack. This does not come as a surprise, as both use similar methodologies, which are certainly more advanced than most.

The two names at the bottom also haven't changed, although this year it's Olney who fared unbelievably poorly, while last year it was Phillips. 13.05 is incredibly bad; if you just used 2007 records, without looking at RS/RA or regressing or accounting for any offseason moves or doing anything, you get 12.46. So Olney's input actually detracted from the information given. Impressive.

stldogget05
Veteran Player
Posts: 909
Joined: April 25 06, 10:41 pm

Re: Evaluating April MLB Predictions (2008)

Post by stldogget05 »

I wonder if the various media outlets ever look at things like this. Sense being an analyst is these peoples job they should hold some sort of accountability for what they. I'm not saying it should be a huge amount because obviously crazy things happen in sports, but if someone is consistently wrong like Phillips maybe it is time to reevaluate his position as an analyst.

User avatar
ghostrunner
Hall Of Famer
Posts: 28744
Joined: April 18 06, 9:40 pm

Re: Evaluating April MLB Predictions (2008)

Post by ghostrunner »

To me it's merely interesting. Does anyone outside of Vegas care how right these guys are? I suppose it's useful to have a gauge of how accurate various systems are, but I don't care that much if individuals like Kurkjian and Olney aren't right much of the time. They're good advocates for the game, and I enjoy their input most of the time. Phillips doesn't impress me on any level, though occasionally he has some decent insight into the GM world.

User avatar
mcgee51taguchi99
Perennial All-Star
Posts: 3649
Joined: May 25 06, 2:47 pm
Location: Crying/injoyin watching tha Crew in tha Playoffs!!!

Re: Evaluating April MLB Predictions (2008)

Post by mcgee51taguchi99 »

stldogget05 wrote:I wonder if the various media outlets ever look at things like this. Sense being an analyst is these peoples job they should hold some sort of accountability for what they. I'm not saying it should be a huge amount because obviously crazy things happen in sports, but if someone is consistently wrong like Phillips maybe it is time to reevaluate his position as an analyst.
I want the people held accountable who make guys like Phillips and Kruk go on record with their so-called predictions. It's fun and all I guess, but what these guys think will happen really isn't important whatsoever.

User avatar
Jmodene
Still waiting for someone to use a Weird Al song
Posts: 15535
Joined: April 18 06, 5:54 pm
Location: Las Vegas, NV

Re: Evaluating April MLB Predictions (2008)

Post by Jmodene »

I still remember, in 2004 or 05, John Kruk going on "Baseball Tonight" in July and seriously suggesting the Astros should trade Roger Clemens to the Cardinals so he could "finish with a winner".

User avatar
G. Keenan
Sucking on the Rally Nipple
Posts: 23459
Joined: April 16 06, 6:03 pm
Location: Chicago

Re: Evaluating April MLB Predictions (2008)

Post by G. Keenan »

ghostrunner wrote:To me it's merely interesting. Does anyone outside of Vegas care how right these guys are? I suppose it's useful to have a gauge of how accurate various systems are, but I don't care that much if individuals like Kurkjian and Olney aren't right much of the time. They're good advocates for the game, and I enjoy their input most of the time. Phillips doesn't impress me on any level, though occasionally he has some decent insight into the GM world.
I would think people would care how accurate the various mathematical modeling systems are. A lot of people use them for fantasy baseball, which sometimes costs money. Outlets that compete over saber fans would presumably want to have a more accurate system than the competition.

AWvsCBsteeeerike3
"I could totally eat a pig butt, if smoked correctly!"
Posts: 27273
Joined: August 5 08, 11:24 am
Location: Thinking of the Children

Re: Evaluating April MLB Predictions (2008)

Post by AWvsCBsteeeerike3 »

G. Keenan wrote:
ghostrunner wrote:To me it's merely interesting. Does anyone outside of Vegas care how right these guys are? I suppose it's useful to have a gauge of how accurate various systems are, but I don't care that much if individuals like Kurkjian and Olney aren't right much of the time. They're good advocates for the game, and I enjoy their input most of the time. Phillips doesn't impress me on any level, though occasionally he has some decent insight into the GM world.
I would think people would care how accurate the various mathematical modeling systems are. A lot of people use them for fantasy baseball, which sometimes costs money. Outlets that compete over saber fans would presumably want to have a more accurate system than the competition.
I'd like to punch buster olney in the kidney.

Post Reply