If you are like me, you had a lot of respect for FiveThirtyEight. They represented a new breed of political journalists who could explain the world with hard data. FiveThirtyEight aggregates polls, assigns weights to them based on historic accuracy and produces more accurate forecasts than any individual pollster could. Nate Silver shot to fame by forecasting the 2008 Presidential and Senate races with great accuracy.
This Primary season however, FiveThirtyEight has gone from punditry to hackery (not my words). What follows is an analysis of how every one of their forecasts compared to actual outcomes.
Note: Here is Nate Silver explaining how "Polls Plus" averages are calculated. They apparently include "endorsements, state and national fundraising totals, favorability ratings, ideology ratings and national polls".
See the bottom of this post for the raw data for probability distributions and confidence intervals for each democratic primary race. This data was manually transcribed from 538's website by hand. Ive checked them several times for errors, but if you do find one let me know.
| State | Candidate | "Polls Plus" Average | Actual Result | Forecast - Actual Result |
| Alabama | sanders | 23.2 | 19.2 | 4 |
| Alabama | clinton | 73.9 | 77.8 | -3.9 |
| Arkansas | sanders | 32.2 | 29.7 | 2.5 |
| Arkansas | clinton | 64.3 | 66.3 | -2 |
| Connecticut | sanders | 46.5 | 46.4 | 0.1 |
| Connecticut | clinton | 51.1 | 51.8 | -0.7 |
| Wisconsin | sanders | 50.2 | 56.6 | -6.4 |
| Wisconsin | clinton | 47.2 | 43.1 | 4.1 |
| WestVirginia | sanders | 49.6 | 51.4 | -1.8 |
| WestVirginia | clinton | 46.6 | 35.8 | 10.8 |
| Virginia | sanders | 34.6 | 35.2 | -0.6 |
| Virginia | clinton | 62.3 | 64.3 | -2 |
| Vermont | sanders | 86.7 | 86.1 | 0.6 |
| Vermont | clinton | 11 | 13.6 | -2.6 |
| Texas | sanders | 32.4 | 33.2 | -0.8 |
| Texas | clinton | 64.6 | 65.2 | -0.6 |
| Tennessee | sanders | 33.9 | 32.4 | 1.5 |
| Tennessee | clinton | 62.7 | 66.1 | -3.4 |
| SouthCarolina | sanders | 28.7 | 26 | 2.7 |
| SouthCarolina | clinton | 67 | 73.5 | -6.5 |
| RhodeIsland | sanders | 49.3 | 54.6 | -5.3 |
| RhodeIsland | clinton | 47.9 | 43.6 | 4.3 |
| Pennsylvania | sanders | 39.7 | 43.6 | -3.9 |
| Pennsylvania | clinton | 57.8 | 55.6 | 2.2 |
| Oklahoma | sanders | 47.3 | 51.9 | -4.6 |
| Oklahoma | clinton | 47.4 | 41.5 | 5.9 |
| Ohio | sanders | 43.1 | 42.7 | 0.4 |
| Ohio | clinton | 54.1 | 56.5 | -2.4 |
| Nevada | sanders | 46 | 47.3 | -1.3 |
| Nevada | clinton | 52.4 | 52.6 | -0.2 |
| NewYork | sanders | 41.3 | 42 | -0.7 |
| NewYork | clinton | 56.3 | 58 | -1.7 |
| NewHampshire | sanders | 57.2 | 60.4 | -3.2 |
| NewHampshire | clinton | 39.9 | 38 | 1.9 |
| NorthCarolina | sanders | 36.2 | 40.8 | -4.6 |
| NorthCarolina | clinton | 60.8 | 54.6 | 6.2 |
| Missouri | sanders | 48 | 49.4 | -1.4 |
| Missouri | clinton | 48.8 | 49.6 | -0.8 |
| Mississippi | sanders | 14.5 | 16.5 | -2 |
| Mississippi | clinton | 79.2 | 82.6 | -3.4 |
| Michigan | sanders | 37.5 | 49.8 | -12.3 |
| Michigan | clinton | 60 | 48.3 | 11.7 |
| Massachusetts | sanders | 44.8 | 48.7 | -3.9 |
| Massachusetts | clinton | 52.3 | 50.1 | 2.2 |
| Maryland | sanders | 39.8 | 33.2 | 6.6 |
| Maryland | clinton | 57.5 | 63 | -5.5 |
| Louisiana | sanders | 17.9 | 23.2 | -5.3 |
| Louisiana | clinton | 75 | 71.1 | 3.9 |
| Iowa | sanders | 45.1 | 49.6 | -4.5 |
| Iowa | clinton | 48.3 | 49.9 | -1.6 |
| Indiana | sanders | 43.3 | 52.5 | -9.2 |
| Indiana | clinton | 54.2 | 47.5 | 6.7 |
| Illinois | sanders | 44.1 | 48.7 | -4.6 |
| Illinois | clinton | 51.8 | 50.5 | 1.3 |
| Georgia | sanders | 28.9 | 28.2 | 0.7 |
| Georgia | clinton | 68 | 71.3 | -3.3 |
| Florida | sanders | 32.1 | 33.3 | -1.2 |
| Florida | clinton | 64.9 | 64.4 | 0.5 |
| Alabama | trump | 41.4 | 43.42 | -2.02 |
| Alabama | cruz | 16.8 | 21.09 | -4.29 |
| Alabama | rubio | 22.7 | 18.66 | 4.04 |
| Alabama | kasich | 5.6 | 4.43 | 1.17 |
| Arkansas | trump | 32.4 | 32.79 | -0.39 |
| Arkansas | cruz | 27.9 | 30.5 | -2.6 |
| Arkansas | rubio | 24.9 | 24.8 | 0.1 |
| Arkansas | kasich | 4.6 | 3.72 | 0.88 |
| Arizona | trump | 45.2 | 45.95 | -0.75 |
| Arizona | cruz | 33.9 | 27.61 | 6.29 |
| Arizona | rubio | 0 | 0 | 0 |
| Arizona | kasich | 18.4 | 10.57 | 7.83 |
| Connecticut | trump | 56.1 | 57.87 | -1.77 |
| Connecticut | cruz | 12.2 | 11.71 | 0.49 |
| Connecticut | rubio | 0 | 0 | 0 |
| Connecticut | kasich | 29.3 | 28.36 | 0.94 |
| Florida | trump | 42 | 45.72 | -3.72 |
| Florida | cruz | 18.8 | 17.14 | 1.66 |
| Florida | rubio | 27.7 | 27.04 | 0.66 |
| Florida | kasich | 9.7 | 6.77 | 2.93 |
| Georgia | trump | 36.8 | 38.81 | -2.01 |
| Georgia | cruz | 20.9 | 23.6 | -2.7 |
| Georgia | rubio | 23.8 | 24.45 | -0.65 |
| Georgia | kasich | 8 | 5.59 | 2.41 |
| Illinois | trump | 32.7 | 38.8 | -6.1 |
| Illinois | cruz | 28.4 | 30.23 | -1.83 |
| Illinois | rubio | 16.7 | 8.74 | 7.96 |
| Illinois | kasich | 20.2 | 19.74 | 0.46 |
| Indiana | trump | 43.6 | 53.25 | -9.65 |
| Indiana | cruz | 37.6 | 36.64 | 0.96 |
| Indiana | rubio | 0 | 0 | 0 |
| Indiana | kasich | 16.2 | 7.57 | 8.63 |
| Iowa | trump | 25.6 | 24.3 | 1.3 |
| Iowa | cruz | 24.3 | 27.64 | -3.34 |
| Iowa | rubio | 18.1 | 23.12 | -5.02 |
| Iowa | kasich | 2.8 | 1.86 | 0.94 |
| Kansas | trump | 33.1 | 23.35 | 9.75 |
| Kansas | cruz | 31.8 | 47.5 | -15.7 |
| Kansas | rubio | 19.5 | 16.83 | 2.67 |
| Kansas | kasich | 14.4 | 11.07 | 3.33 |
| Louisiana | trump | 44.1 | 41.45 | 2.65 |
| Louisiana | cruz | 30 | 37.83 | -7.83 |
| Louisiana | rubio | 17 | 11.22 | 5.78 |
| Louisiana | kasich | 6.3 | 6.43 | -0.13 |
| Maryland | trump | 50.3 | 54.1 | -3.8 |
| Maryland | cruz | 23.8 | 18.97 | 4.83 |
| Maryland | rubio | 0 | 0 | 0 |
| Maryland | kasich | 23.9 | 23.22 | 0.68 |
| Massachusetts | trump | 48.9 | 48.99 | -0.09 |
| Massachusetts | cruz | 9.2 | 9.5 | -0.3 |
| Massachusetts | rubio | 20.5 | 17.75 | 2.75 |
| Massachusetts | kasich | 16.9 | 17.94 | -1.04 |
| Michigan | trump | 36.8 | 36.55 | 0.25 |
| Michigan | cruz | 23 | 24.68 | -1.68 |
| Michigan | rubio | 14.1 | 9.34 | 4.76 |
| Michigan | kasich | 23.7 | 24.26 | -0.56 |
| NorthCarolina | trump | 42.7 | 40.23 | 2.47 |
| NorthCarolina | cruz | 32.6 | 36.76 | -4.16 |
| NorthCarolina | rubio | 9.9 | 7.73 | 2.17 |
| NorthCarolina | kasich | 12.9 | 12.67 | 0.23 |
| NewHampshire | trump | 26.8 | 35.23 | -8.43 |
| NewHampshire | cruz | 12 | 11.63 | 0.37 |
| NewHampshire | rubio | 15.7 | 10.52 | 5.18 |
| NewHampshire | kasich | 15.2 | 15.72 | -0.52 |
| NewYork | trump | 53.8 | 59.21 | -5.41 |
| NewYork | cruz | 19.1 | 14.53 | 4.57 |
| NewYork | rubio | 0 | 0 | 0 |
| NewYork | kasich | 24.9 | 24.68 | 0.22 |
| Nevada | trump | 37.1 | 45.75 | -8.65 |
| Nevada | cruz | 21 | 21.3 | -0.3 |
| Nevada | rubio | 27.1 | 23.77 | 3.33 |
| Nevada | kasich | 7.6 | 3.59 | 4.01 |
| Ohio | trump | 34.4 | 35.87 | -1.47 |
| Ohio | cruz | 18.1 | 13.31 | 4.79 |
| Ohio | rubio | 3.9 | 2.34 | 1.56 |
| Ohio | kasich | 41.8 | 46.95 | -5.15 |
| Oklahoma | trump | 33.3 | 28.32 | 4.98 |
| Oklahoma | cruz | 21.9 | 34.37 | -12.47 |
| Oklahoma | rubio | 25.2 | 26.01 | -0.81 |
| Oklahoma | kasich | 7.9 | 3.59 | 4.31 |
| Pennsylvania | trump | 47.7 | 56.61 | -8.91 |
| Pennsylvania | cruz | 27.1 | 21.67 | 5.43 |
| Pennsylvania | rubio | 0 | 0 | 0 |
| Pennsylvania | kasich | 23.4 | 19.44 | 3.96 |
| RhodeIsland | trump | 59.7 | 62.92 | -3.22 |
| RhodeIsland | cruz | 12.5 | 10.29 | 2.21 |
| RhodeIsland | rubio | 0 | 0 | 0 |
| RhodeIsland | kasich | 25.2 | 24.01 | 1.19 |
| SouthCarolina | trump | 30.5 | 32.51 | -2.01 |
| SouthCarolina | cruz | 19.5 | 22.33 | -2.83 |
| SouthCarolina | rubio | 19.8 | 22.48 | -2.68 |
| SouthCarolina | kasich | 9.3 | 7.61 | 1.69 |
| Tennessee | trump | 42.2 | 38.94 | 3.26 |
| Tennessee | cruz | 18.6 | 24.71 | -6.11 |
| Tennessee | rubio | 22.4 | 21.18 | 1.22 |
| Tennessee | kasich | 6.2 | 5.29 | 0.91 |
| Texas | trump | 26.6 | 26.75 | -0.15 |
| Texas | cruz | 37.9 | 43.76 | -5.86 |
| Texas | rubio | 20.2 | 17.74 | 2.46 |
| Texas | kasich | 7.1 | 4.25 | 2.85 |
| Utah | trump | 9.9 | 14.03 | -4.13 |
| Utah | cruz | 58 | 69.17 | -11.17 |
| Utah | rubio | 0 | 0 | 0 |
| Utah | kasich | 30.3 | 16.81 | 13.49 |
| Virginia | trump | 36.7 | 34.8 | 1.9 |
| Virginia | cruz | 16.1 | 16.69 | -0.59 |
| Virginia | rubio | 30.1 | 31.98 | -1.88 |
| Virginia | kasich | 7.5 | 9.54 | -2.04 |
| Wisconsin | trump | 34.1 | 35.02 | -0.92 |
| Wisconsin | cruz | 41.8 | 48.2 | -6.4 |
| Wisconsin | rubio | 0 | 0 | 0 |
| Wisconsin | kasich | 21.8 | 14.1 | 7.7 |
What did we learn? FiveThirtyEight seems to be overestimating Establishment candidates' vote share (Clinton) and underestimated the outsiders' (Sanders, Cruz, Trump).
Note: While they have also been kind to Rubio and Kasich, it must be noted that one of them had too few observations and the other had low vote shares throughout the race, making forecasts difficult.
The distribution of their "error" i.e. their forecast vs. actual result is normally distributed around mean = 0 (Here a "Forecast Error" > 0 means FiveThirtyEight overestimated your vote share. In other words, the candidate did worse than the forecast)
But some candidates are more equal than others. Kasich and Rubio have benefitted from overestimation. Sanders and Trump have largely been underestimated, while Clinton has benefitted from some wild overestimates in some key states (notice the long tail - owing to states like Michigan and Indiana).
Of all the candidates, they love Clinton the most, and Sanders the least. Nate Silver and his lieutenant Harry "whiz kid" Enten don't try to hide it. Here are some of the things they have said in the past. Taken together, you have to wonder if they are data journalists or pro-Clinton hacks. Here are some of the memorable things they have said (paraphrasing)
No, the system isn't rigged against Sanders (which sort of contradicts with #7)
But lets take a closer look at their Clinton vs Sanders forecasts. What follows is an overlay of FiveThirtyEight's forecast vs actual outcome.
Each forecast has a probability distribution and a polling average. According to FivethirtyEight, there is an 80% chance that the actual outcome will fall within the blue-shaded area. A blue dot marks the polling average (also supplied by FiveThirtyEight)
A red line represents the actual result from that primary. A red shaded area indicates that the candidate did better than the forecast, a black shaded area indicated that the candidate underperformed the forecast.
As you can see, FiveThirtyEight has wildly overestimated Clinton's vote share and underestimated Sanders' in some key races.
What does all this mean? FiveThirtyEight is not being objective. They are letting their personal biases affect their work. Worse, they are using their platform to go after Clinton's opponents. That is not journalism, that is just hackery.
