Projecting the performance of baseball players is a very mature science. Projections continue to improve every year, and one factor is the plethora of competition. With some projection systems honing their craft for more than a decade, and new competitors joining every year, the level of competition – and quality of the advice – couldn’t be higher. Our goal here at FantasyPros is to help you win your league, and step 1 is choosing the right projections to inform your decisions.

Methodology

For our accuracy competition, we evaluated 17 different projection systems. There are several ways to evaluate the accuracy of projections. One option is to find a single composite stat like wOBA or WAR that encompasses a player’s entire game and evaluate that. Since we are primarily concerned with fantasy, we’re using a system that isolates the ten standard scoring categories in a 5×5 league (R, HR, RBI, SB and AVG for hitters, W, SV, K, ERA and WHIP for pitchers) as well as playing time (at-bats and innings pitched). For our 5×5 categories, we treated each as a rate stat so as to isolate playing time and talent projections. For example, we evaluate HR/AB rather than just HR. That way a system gets good marks for correctly predicting a player’s talent as a home run hitter, even if their total was off due to a bad playing time projection. Last year, Mike Trout hit 39 HRs on 471 ABs, which is a rate of 8.3%. Suppose a system projected he would hit 48 HRs on 575 ABs. That AB projection is too high, but the projected home run rate is right on target at 8.3%. This system would get a bad accuracy score for at-bats, but would be considered very accurate for home runs because it correctly predicted how effective Trout would be at hitting homers.

To calculate our accuracy metric, we first look at the errors of each projection system for a player, and count the mean and standard deviation. Our 16 systems missed on Trout’s at-bats projection by an average of 60, with a standard deviation of about 18. For each projection system, we divide by the standard deviation and subtract the mean to get a z-score. For example, numberFire projected Trout to have 505 at-bats. Compared to an actual outcome of 471, that’s a miss of just 34, which is 26 better than the average miss of 60. We divide by the standard deviation of 18 to find that numberFire got a z-score of about -1.5 for Trout’s AB projection. A negative number is good here – this means numberFire was about one and a half standard deviations better than the average projection system when it comes to predicting Trout’s at-bats. To get numberFire’s overall score for at-bat projections, we simply average these z-scores across all players.

To get overall accuracy for a projection system, we sum up their average z-scores across each stat category, and then rank the projection systems according to that metric (remember, a lower z-score is better, and a score of 0 is average).

Results

Without further ado, here are the most accurate fantasy baseball projections for 2018:

^{*Full disclosure – the writer of this article is also the creator of the Scutaro projections.}
Here’s the category breakdown for hitters:

And for pitchers:

Analysis

These results are extremely encouraging if you are a fan of the wisdom of the crowd. The top three projection systems are each based on a consensus, though they represent three very different philosophies on how to go about that.

• The FantasyPros Zeile Consensus Projections, named for Todd Zeile, are a straight average of as many freely available projection systems as we can find. In 2018 we used 11 different projection sources: site projections from ESPN, CBS, Razzball, numberFire, RotoChamp and TG Fantasy Baseball, as well as my own Scutaro projections, ZiPS, Steamer, and projections from Clay Davenport and Jason Johnson. No weight was given to the past accuracy of these systems – each contributed equally to the final product.
• The Average Total Cost (ATC) projection system, as described in this article, was created by Ariel Cohen and Reuven Guy and is hosted by FanGraphs. It is a consensus of ZiPS, Steamer, FANS and other projection systems (the full list is not specified). Unlike the Zeile Consensus, ATC doesn’t treat each projection system equally. Rather, each projected stat is a weighted average of projections from each system, with the weight determined by the past accuracy. This means each stat has a different set of weights, based on how accurate each system is for that particular stat. It also includes some manual adjustments by the author for playing time projections.
• The FanGraphs Depth Charts projections, as described in this article, project playing time manually based on the opinions of Fangraphs’ staff of baseball writers. For performance stats, they are a straight average like Zeile, but unlike Zeile they chose to include just two trusted sources: ZiPS and Steamer.

The power of aggregating projections is clear in the results. Both Zeile and ATC ranked in the top 5 for all but 2 of the 12 stat categories: Zeile ranked 6th in WHIP and 13th in batting average, while ATC was 8th in wins and tied for 9th in batting average. Zeile was the most accurate system for both playing time metrics (at-bats and innings pitched) as well as stolen bases and strikeouts. ATC ranked in the top 5 for both playing time metrics, and was the #1 system for both runs scored and WHIP. The only other system to rank in the top 5 for both playing time categories was the RotoChamp site projections, which ranked 2nd in both metrics, beating every other non-consensus source. While Zeile’s rank as the most accurate system is a win for the gather-as-much-data-as-possible strategy, the race was tight for these top 3. The fact that Depth Charts were able outperform any single projection system by aggregating just two (very respected) sources is a great indicator that aggregating multiple sources is a good approach.

Congratulations are also due to Derek Carty at THE BAT, Razzball’s site projections, Steamer and Jason Johnson’s projections. Those were the top four of the non-consensus projection systems, and the only ones to place in the top-10 overall for both hitting and pitching.

Another thing we can glean from the z-scores is how remarkably tight the race is. In every stat category, the best projection system had a score at or above -.280, meaning the winners weren’t huge outliers. Even the best system can only be expected to beat an average system by less than a third of a standard deviation.

This wraps up our analysis of the accuracy of 2018 fantasy baseball preseason projections. For a similar analysis of expert rankings, check out our Most Accurate Fantasy Baseball Experts for 2018.

—

Jacob Herlin is a Senior Data Analyst for FantasyPros. You can find him on twitter @jacoblawherlin.