What if there were no polls?
No, this isn’t a Bill Mitchell fever dream—it’s a real question that comes up when you’re trying to build a primary polls-based Senate forecast model, which is what I’ve been working on for the last few months. Pollsters don’t pay equal attention to every Senate race: They spend more time (and money) trying to figure out what who’s going to win in marquee races toss-up races than in races that should be a cinch. This cycle, that means we’re likely to see some good polling from Missouri, West Virginia, Arizona, Nevada, Indiana, and Montana, but comparatively little polling on Nebraska or Rhode Island.
But my goal is to forecast every Senate race—Rhode Island and Nebraska included. And that’s one area where fundamentals can be helpful.
Fundamentals encompass basically everything outside head-to-head polls—presidential approval, generic ballot results, past election results within the state, information about the candidates, campaign finance data, etc. The idea is to take information we have on broader questions (e.g. how popular is the president, who are the candidates) and use it to predict what’s going to happen to supplement polling data (or to stand in for it when it’s not there). The New York Times Upshot and FiveThirtyEight both include fundamentals in their forecasts.
So rather than reinvent the wheel, I’ve used the New York Times fundamentals forecast as the basis for my model. I’ve used their historical polling data (presidential approval and net approval of incumbent senators), created my own measure of candidate experience, plugged in incumbency (sitting senators tend to do better than non-incumbents, when all else is equal), and the partisan lean in the previous presidential election. The Upshot uses a version of basically all these variables (they use generic ballot polls rather than presidential approval—more on that in a minute) plus campaign finance (which I’ve left out for now but may add later). So my initial fundamentals forecast is pretty similar to theirs.
This model wasn’t very accurate from an objective standpoint (fundamentals-only models of races often aren’t—these projections were off by 10 points on average), but it performed reasonably well given the constraints. It was only a point or two less accurate on average than what FiveThirtyEight came up with, and that’s worth something in races with little or no polling.
It’d be easy to run with this reasonably accurate version of the fundamentals. But first, I want to try two other things in order to try to look beyond some of the simpler regression techniques and think about which data is most relevant to projecting midterm results.
So I used two different methods to look at how well the fundamentals did at predicting outcomes. I used multiple linear regression and multivariate adaptive regression splines (MARS). If you’re a stat head, you know what that means. But even if you’re not, the ideas behind these regressions are pretty simple.
You may not know it, but you’re probably already familiar with linear regression. This is basically the “line of best fit” that gets drawn through data basically everywhere. There are a lot of assumptions that go into using this technique, but one of the most important ones is that there are straight-line relationships between whatever we’re seeing now and whatever we’re trying to predict.
That can be a problem in politics, because not everything fits on a straight line. For instance, the relationship between the House popular vote and the probability of Republican control hasn’t been a straight line. Or, if a president becomes exceedingly popular, his popularity will eventually have diminishing returns down-ballot. And sometimes seemingly obvious straight-line projections simply don’t pan out (for instance, the theory that megalopolises would eventually dominate our politics).
Which leaves us with MARS. You can see the difference between MARS and linear regression by looking at the first two graphics here. The basic intuition is simple—MARS looks at the data, starts to draw a line through the data but then lets itself draw a hockey stick instead of a straight line, if that’s what the data looks like. In other words, if a president’s approval rating goes from 45 percent to 65 percent in an general election year, his party’s probability of taking the House will go way up. But at some point partisan polarization creates a ceiling, and gaining 20 more percentage points (i.e. going from 65 to 85 percent approval) wouldn’t change the odds as much. I thought MARS might pick up on complicated relationships like that.
So I applied multiple linear regression and MARS to the fundamentals data—but before doing that, I tried one other trick.
I had been playing around with the data and noticed that if I divided it up into two time periods—before 2002 and after 2002. The year 2002 is a strange year because it captured George W. Bush’s abnormally high approval rating post-9/11. In both eras, incumbents tended to outperform non-incumbents, but in the Clinton and H.W. Bush period, the incumbency advantage was a few points higher than it was in the post-2002 period.
Intuitively, that made sense: Over time, politics has become more nationalized and voters have become less willing to cross party lines to vote for a competent, well-known legislator from the other party.
This observation led me to slice my data in half and try to predict more recent results by only using the post-2002 data. My thinking is that if Clinton-era incumbents were, on average, stronger than Bush and Obama-era incumbents, then maybe I shouldn’t be feeding them into my model. Models are what they eat and a model with too much historical information might overrate incumbents.
So I tried linear regression and MARS on both complete and post-2002-only versions of the dataset. And my fancy ideas failed to make the forecast any more accurate.
Disappointing, right?
Well, not really. Part of building a model—and working with numbers and, well, life—is learning what doesn’t work. Nobody bats a thousand at anything and if they did, the world would be a boring place. So I’m actually kind of excited to see all of the interesting ideas I have that don’t work out. Because that’s part of the process. (And I’ll try to share as many of those failures as possible in this series.)
In any case, this part of the model is good enough for the time being. It left me with a couple workable versions of the fundamentals forecast—some of which differ from what’s already out there. It’s not perfect, of course—there are other variables I would like to add in, regressions I would like to try, and other aspects that I might want to mess around with. But I need to focus on the (more complicated) poll-based forecast now part of the model now and will circle back to this later.
