FiveThirtyEight has an establishment bias

If you are like me, you had a lot of respect for FiveThirtyEight. They represented a new breed of political journalists who could explain the world with hard data. FiveThirtyEight aggregates polls, assigns weights to them based on historic accuracy and produces more accurate forecasts than any individual pollster could. Nate Silver shot to fame by forecasting the 2008 Presidential and Senate races with great accuracy.

This Primary season however, FiveThirtyEight has gone from punditry to hackery (not my words). What follows is an analysis of how every one of their forecasts compared to actual outcomes.

Note: Here is Nate Silver explaining how "Polls Plus" averages are calculated. They apparently include "endorsements, state and national fundraising totals, favorability ratings, ideology ratings and national polls".

See the bottom of this post for the raw data for probability distributions and confidence intervals for each democratic primary race. This data was manually transcribed from 538's website by hand. Ive checked them several times for errors, but if you do find one let me know.

State	Candidate	"Polls Plus" Average	Actual Result	Forecast - Actual Result
Alabama	sanders	23.2	19.2	4
Alabama	clinton	73.9	77.8	-3.9
Arkansas	sanders	32.2	29.7	2.5
Arkansas	clinton	64.3	66.3	-2
Connecticut	sanders	46.5	46.4	0.1
Connecticut	clinton	51.1	51.8	-0.7
Wisconsin	sanders	50.2	56.6	-6.4
Wisconsin	clinton	47.2	43.1	4.1
WestVirginia	sanders	49.6	51.4	-1.8
WestVirginia	clinton	46.6	35.8	10.8
Virginia	sanders	34.6	35.2	-0.6
Virginia	clinton	62.3	64.3	-2
Vermont	sanders	86.7	86.1	0.6
Vermont	clinton	11	13.6	-2.6
Texas	sanders	32.4	33.2	-0.8
Texas	clinton	64.6	65.2	-0.6
Tennessee	sanders	33.9	32.4	1.5
Tennessee	clinton	62.7	66.1	-3.4
SouthCarolina	sanders	28.7	26	2.7
SouthCarolina	clinton	67	73.5	-6.5
RhodeIsland	sanders	49.3	54.6	-5.3
RhodeIsland	clinton	47.9	43.6	4.3
Pennsylvania	sanders	39.7	43.6	-3.9
Pennsylvania	clinton	57.8	55.6	2.2
Oklahoma	sanders	47.3	51.9	-4.6
Oklahoma	clinton	47.4	41.5	5.9
Ohio	sanders	43.1	42.7	0.4
Ohio	clinton	54.1	56.5	-2.4
Nevada	sanders	46	47.3	-1.3
Nevada	clinton	52.4	52.6	-0.2
NewYork	sanders	41.3	42	-0.7
NewYork	clinton	56.3	58	-1.7
NewHampshire	sanders	57.2	60.4	-3.2
NewHampshire	clinton	39.9	38	1.9
NorthCarolina	sanders	36.2	40.8	-4.6
NorthCarolina	clinton	60.8	54.6	6.2
Missouri	sanders	48	49.4	-1.4
Missouri	clinton	48.8	49.6	-0.8
Mississippi	sanders	14.5	16.5	-2
Mississippi	clinton	79.2	82.6	-3.4
Michigan	sanders	37.5	49.8	-12.3
Michigan	clinton	60	48.3	11.7
Massachusetts	sanders	44.8	48.7	-3.9
Massachusetts	clinton	52.3	50.1	2.2
Maryland	sanders	39.8	33.2	6.6
Maryland	clinton	57.5	63	-5.5
Louisiana	sanders	17.9	23.2	-5.3
Louisiana	clinton	75	71.1	3.9
Iowa	sanders	45.1	49.6	-4.5
Iowa	clinton	48.3	49.9	-1.6
Indiana	sanders	43.3	52.5	-9.2
Indiana	clinton	54.2	47.5	6.7
Illinois	sanders	44.1	48.7	-4.6
Illinois	clinton	51.8	50.5	1.3
Georgia	sanders	28.9	28.2	0.7
Georgia	clinton	68	71.3	-3.3
Florida	sanders	32.1	33.3	-1.2
Florida	clinton	64.9	64.4	0.5
Alabama	trump	41.4	43.42	-2.02
Alabama	cruz	16.8	21.09	-4.29
Alabama	rubio	22.7	18.66	4.04
Alabama	kasich	5.6	4.43	1.17
Arkansas	trump	32.4	32.79	-0.39
Arkansas	cruz	27.9	30.5	-2.6
Arkansas	rubio	24.9	24.8	0.1
Arkansas	kasich	4.6	3.72	0.88
Arizona	trump	45.2	45.95	-0.75
Arizona	cruz	33.9	27.61	6.29
Arizona	rubio	0	0	0
Arizona	kasich	18.4	10.57	7.83
Connecticut	trump	56.1	57.87	-1.77
Connecticut	cruz	12.2	11.71	0.49
Connecticut	rubio	0	0	0
Connecticut	kasich	29.3	28.36	0.94
Florida	trump	42	45.72	-3.72
Florida	cruz	18.8	17.14	1.66
Florida	rubio	27.7	27.04	0.66
Florida	kasich	9.7	6.77	2.93
Georgia	trump	36.8	38.81	-2.01
Georgia	cruz	20.9	23.6	-2.7
Georgia	rubio	23.8	24.45	-0.65
Georgia	kasich	8	5.59	2.41
Illinois	trump	32.7	38.8	-6.1
Illinois	cruz	28.4	30.23	-1.83
Illinois	rubio	16.7	8.74	7.96
Illinois	kasich	20.2	19.74	0.46
Indiana	trump	43.6	53.25	-9.65
Indiana	cruz	37.6	36.64	0.96
Indiana	rubio	0	0	0
Indiana	kasich	16.2	7.57	8.63
Iowa	trump	25.6	24.3	1.3
Iowa	cruz	24.3	27.64	-3.34
Iowa	rubio	18.1	23.12	-5.02
Iowa	kasich	2.8	1.86	0.94
Kansas	trump	33.1	23.35	9.75
Kansas	cruz	31.8	47.5	-15.7
Kansas	rubio	19.5	16.83	2.67
Kansas	kasich	14.4	11.07	3.33
Louisiana	trump	44.1	41.45	2.65
Louisiana	cruz	30	37.83	-7.83
Louisiana	rubio	17	11.22	5.78
Louisiana	kasich	6.3	6.43	-0.13
Maryland	trump	50.3	54.1	-3.8
Maryland	cruz	23.8	18.97	4.83
Maryland	rubio	0	0	0
Maryland	kasich	23.9	23.22	0.68
Massachusetts	trump	48.9	48.99	-0.09
Massachusetts	cruz	9.2	9.5	-0.3
Massachusetts	rubio	20.5	17.75	2.75
Massachusetts	kasich	16.9	17.94	-1.04
Michigan	trump	36.8	36.55	0.25
Michigan	cruz	23	24.68	-1.68
Michigan	rubio	14.1	9.34	4.76
Michigan	kasich	23.7	24.26	-0.56
NorthCarolina	trump	42.7	40.23	2.47
NorthCarolina	cruz	32.6	36.76	-4.16
NorthCarolina	rubio	9.9	7.73	2.17
NorthCarolina	kasich	12.9	12.67	0.23
NewHampshire	trump	26.8	35.23	-8.43
NewHampshire	cruz	12	11.63	0.37
NewHampshire	rubio	15.7	10.52	5.18
NewHampshire	kasich	15.2	15.72	-0.52
NewYork	trump	53.8	59.21	-5.41
NewYork	cruz	19.1	14.53	4.57
NewYork	rubio	0	0	0
NewYork	kasich	24.9	24.68	0.22
Nevada	trump	37.1	45.75	-8.65
Nevada	cruz	21	21.3	-0.3
Nevada	rubio	27.1	23.77	3.33
Nevada	kasich	7.6	3.59	4.01
Ohio	trump	34.4	35.87	-1.47
Ohio	cruz	18.1	13.31	4.79
Ohio	rubio	3.9	2.34	1.56
Ohio	kasich	41.8	46.95	-5.15
Oklahoma	trump	33.3	28.32	4.98
Oklahoma	cruz	21.9	34.37	-12.47
Oklahoma	rubio	25.2	26.01	-0.81
Oklahoma	kasich	7.9	3.59	4.31
Pennsylvania	trump	47.7	56.61	-8.91
Pennsylvania	cruz	27.1	21.67	5.43
Pennsylvania	rubio	0	0	0
Pennsylvania	kasich	23.4	19.44	3.96
RhodeIsland	trump	59.7	62.92	-3.22
RhodeIsland	cruz	12.5	10.29	2.21
RhodeIsland	rubio	0	0	0
RhodeIsland	kasich	25.2	24.01	1.19
SouthCarolina	trump	30.5	32.51	-2.01
SouthCarolina	cruz	19.5	22.33	-2.83
SouthCarolina	rubio	19.8	22.48	-2.68
SouthCarolina	kasich	9.3	7.61	1.69
Tennessee	trump	42.2	38.94	3.26
Tennessee	cruz	18.6	24.71	-6.11
Tennessee	rubio	22.4	21.18	1.22
Tennessee	kasich	6.2	5.29	0.91
Texas	trump	26.6	26.75	-0.15
Texas	cruz	37.9	43.76	-5.86
Texas	rubio	20.2	17.74	2.46
Texas	kasich	7.1	4.25	2.85
Utah	trump	9.9	14.03	-4.13
Utah	cruz	58	69.17	-11.17
Utah	rubio	0	0	0
Utah	kasich	30.3	16.81	13.49
Virginia	trump	36.7	34.8	1.9
Virginia	cruz	16.1	16.69	-0.59
Virginia	rubio	30.1	31.98	-1.88
Virginia	kasich	7.5	9.54	-2.04
Wisconsin	trump	34.1	35.02	-0.92
Wisconsin	cruz	41.8	48.2	-6.4
Wisconsin	rubio	0	0	0
Wisconsin	kasich	21.8	14.1	7.7

What did we learn? FiveThirtyEight seems to be overestimating Establishment candidates' vote share (Clinton) and underestimated the outsiders' (Sanders, Cruz, Trump).

Note: While they have also been kind to Rubio and Kasich, it must be noted that one of them had too few observations and the other had low vote shares throughout the race, making forecasts difficult.

The distribution of their "error" i.e. their forecast vs. actual result is normally distributed around mean = 0 (Here a "Forecast Error" > 0 means FiveThirtyEight overestimated your vote share. In other words, the candidate did worse than the forecast)

But some candidates are more equal than others. Kasich and Rubio have benefitted from overestimation. Sanders and Trump have largely been underestimated, while Clinton has benefitted from some wild overestimates in some key states (notice the long tail - owing to states like Michigan and Indiana).

Of all the candidates, they love Clinton the most, and Sanders the least. Nate Silver and his lieutenant Harry "whiz kid" Enten don't try to hide it. Here are some of the things they have said in the past. Taken together, you have to wonder if they are data journalists or pro-Clinton hacks. Here are some of the memorable things they have said (paraphrasing)

"I will dunk ice water on myself and say "Feel the Bern" if Sanders passes Clinton in the national polling average"

Trump has a better chance of playing in the NBA finals than becoming the Republican nominee 

On Clinton's poor polling against Trump - don't worry about it 

Sanders is doing really well in Iowa. Interesting... 

Sanders could lose everywhere else after Iowa and New Hampshire 

Yes, Sanders won New Hampshire. But thats it for him

Sanders can't win because Establishment Democrats will squash him 

This could be the end of Sanders (February 2016)

Hillary Clinton's got this (Also February 2016)

Sanders is just not going to win, okay?

Hillary Clinton is the favorite (February 2016)

No, the system isn't rigged against Sanders (which sort of contradicts with #7)

Sanders isn't doing well with true Independents

Sanders is hurting Clinton by staying in the race 

Sanders is even less competitive than he appears

But lets take a closer look at their Clinton vs Sanders forecasts. What follows is an overlay of FiveThirtyEight's forecast vs actual outcome.

Each forecast has a probability distribution and a polling average. According to FivethirtyEight, there is an 80% chance that the actual outcome will fall within the blue-shaded area. A blue dot marks the polling average (also supplied by FiveThirtyEight)

A red line represents the actual result from that primary. A red shaded area indicates that the candidate did better than the forecast, a black shaded area indicated that the candidate underperformed the forecast.

As you can see, FiveThirtyEight has wildly overestimated Clinton's vote share and underestimated Sanders' in some key races.

What does all this mean? FiveThirtyEight is not being objective. They are letting their personal biases affect their work. Worse, they are using their platform to go after Clinton's opponents. That is not journalism, that is just hackery.

Note: I've also uploaded the raw data for these probability distributions here.