People’s trust in political polling is on the decline. But don’t pin all, or even most, of the blame on pollsters, says polling guru Nate Silver.
In a Tuesday evening blog post on his statistical analysis website, FiveThirtyEight, Silver defended his reputation and delivered a stern warning to news organizations, urging them to do “a better job” after a 2016 election season in which many newsrooms used polling data to convey an easy win for Hillary Clinton in the 2016 presidential election before she ultimately lost to Donald Trump.
“Here’s the thing,” Silver wrote. “The loss of trust mostly isn’t the pollsters’ fault. It’s the media’s fault.”
“Oh, yes, I’m going there,” he continued. “The loss of trust in polls was enabled, in large part, by reporting and analysis that incorrectly portrayed the polls as showing an almost-certain Clinton win when in fact they showed a close and highly uncertain Electoral College race, especially after FBI Director James B. Comey’s letter to Congress on Oct. 28.”
That polling, Silver asserted, was largely accurate, but the media failed to properly report the data.
“The polls were about as accurate as they’d been, on average, in presidential elections since 1968,” Silver said. “They were somewhat more accurate than they’d been in the most recent federal election, the 2014 midterms. But they were enough to tip the election to Trump because Clinton had been in a precarious position to begin with.”
Moving forward, Silver urged journalists to “do a better job of reporting on uncertainty when they report on polling data.”
“Not only do polls have a margin of sampling error — for instance, the margin of sampling error on CNN’s poll of 1,000 adults is plus or minus 3 percentage points — but they also have other types of errors, such as nonresponse bias,” Silver wrote.
“The people who respond to polls — often under 10 percent of the population contacted — may not be representative of the population as a whole, and that creates a lot of challenges.”
Silver said that the media’s downfall came from leaning solely on the traditional margin of sampling error, when there are a number of different types of error, some of which are better suited for different kinds of polling.
He said if the polls were spot on, Clinton would have won, but factoring all the types of error would show that polling analysis, including that performed by FiveThirtyEight (which on the eve of the election gave Clinton a 71.4 percent change of winning), was rather accurate.
Other types of error Silver explained are “margin of methodical error” and “true margin of error” (which requires the sum of squares formula, according to Silver).
For those not inclined to run the numbers, Silver provided two benchmarks:
- “For a high-quality, 1,000-person national poll, a good estimate of the true margin of error is about plus or minus 4 percentage points.”
- “For a national polling average, meanwhile, the true margin of error is about plus or minus 3 percentage points. It’s true that polling averages can greatly reduce sampling error, by aggregating thousands of interviews together. But polling averages don’t necessarily eliminate methodological error. When the polls are wrong, they tend to miss in the same direction.”
He also said that while a traditional margin of error can be applied to one candidate, a true margin of error for a poll about two candidates tends to be about twice as high. While national polling showed Clinton ahead with a 3-4 percentage point lead, Silver said the true margin of error would have shown a range of error putting Clinton on top by up to 10 percentage points or Trump in the lead by up to 3 percentage points.
“[S]o many of the articles I read toward the end of last year’s campaign didn’t convey any sense of uncertainty at all,” Silver said.
“A small Clinton lead was misreported as a sure thing. And then a small polling error was misreported as a massive failure of the data,” Silver wrote. “It’s a fairly minor part of the puzzle, but if journalists want to rebuild trust in their reporting, ending the boom-and-bust cycle in how they report on polling — first overrating its precision and then being shocked when it’s even a couple of percentage points off — would be one way to start.
“Doing so would make it harder for Trump, or other politicians, to undermine confidence in polls they don’t like,” Silver said.
