Baseball is a privileged sport when it comes to projections. The sheer number of games played in an MLB season, as well as the independent nature of all the events that occur in a baseball game, means analysts have a lot to work with when making projections. More than half of the 2020 MLB season was canceled due to the pandemic. Whether you think it was wise to have baseball at all or not, we did get 60 regular season games. That is still downright luxurious compared to some other sports – that’s more games than an NFL team plays in three years. That means we have enough data that we can actually draw meaningful conclusions about which projection systems were the best, despite the shortened season, so that’s exactly what we’re going to do.

Methodology

There is a wide variety of approaches one can take when measuring baseball projection accuracy. The correct one depends entirely on how you use projections. An MLB team negotiating trades and contracts might be interested in WAR (wins above replacement), because it directly measures how much a player contributes to a team’s success. For fantasy players, however, WAR measures some things that are irrelevant to us, like defensive performance. It also places little emphasis on some things we care deeply about, like stolen bases and RBI.

For our analysis, we chose to evaluate projection systems on how well they predict the standard 5×5 categories – R, HR, RBI, SB and AVG for Hitters, and W, SV, K, ERA and WHIP for Pitchers – with one minor complication. Many of these stats reflect not only how well a player plays, but how much he plays. Predicting playing time is extremely valuable in fantasy, but we don’t want to penalize a projection that gets a player’s home run rate exactly right if he misses half the season with injury. To solve this, we separate at-bats and innings pitched as their own categories, and we measure counting stats on a per-playing-time basis. So our scores for home runs are actually based on HR/AB, and K is actually K/IP, for example.

We calculate the actual accuracy metric in a few steps. For a given player and stat, we first calculate the “raw” error for each projection system. Trea Turner recorded a batting average of .335, so a system that projected him to hit .280 would have a raw error of .055. Once we have raw errors for every projection source, we turn them in to z-scores – we calculate the average and standard deviation across the pool of different projections, and calculate how much better or worse than the average each projection was, measured in standard deviations. The average error for Trea Turner’s batting average was .049 with a standard deviation of .005, so that raw error of .055 is 1.2 standard deviations above the average, so that system gets a score of 1.2. (A score lower than the average would give a negative z-score, and in general lower scores are better once we’re in this format.) We then calculate a system’s overall score for AVG by averaging together these z-scores across the entire pool of players.

One of the advantages of using z-scores is that it puts each stat on an even playing field. The raw score for an AVG projection is on a wildly different scale than one for AB, but this allows us to compare accuracy across stats in a way that makes sense. Once we have an accuracy score for each stat, we can declare a winner for hitters by adding together the scores for each hitting category, and likewise for pitchers. Our overall accuracy winner is based on the sum of the hitter and pitcher scores.

2020 Projections Accuracy

Without any more math to read, here are the standings for the most accurate 2020 baseball projections:

A huge congratulations is in order for Fangraphs and Ariel Cohen for the Average Total Cost (ATC) projection system, which generated the most accurate 2020 projections overall, coming in at #1 for hitters and #2 for pitchers. This is ATC’s second year at the top, after ranking #1 for both hitters and pitchers in 2019. ATC was also #2 in 2018. That’s a deeply impressive track record, considering we’ve only been measuring accuracy this way for three years. ATC is a consensus of other projection systems, with unequal weighting between the component systems, designed to give each source more say in the stats where it is most accurate.

It’s encouraging that ATC and FantasyPros’ own Zeile Consensus Projections are the top two, as both are built out of a consensus of several other sources. The consistent success of both systems means that this “wisdom of the crowd” philosophy is a sound one. Of course, they could not exist but for the component projections they are built from. Derek Carty’s THE BAT projections deserve special recognition for being the most accurate non-consensus projection system, including ranking ahead of Zeile and FanGraphs’ Depth Chart Projections (a combination of ZiPS and Steamer with playing time adjustments by FanGraphs’ writers) for hitters.

Category Results

Here are the accuracy scores for individual categories for hitters:

And for pitchers:

—

Jacob Herlin is a Data Scientist for FantasyPros. You can find him on twitter @jacoblawherlin.