How a data scientist beat the polls by nailing the Georgia senate race

The week of the Georgia senatorial elections, I’ve been following the forecasts by Northwestern University data scientist Tom Miller, who bested most of the polls in the presidential race by missing the margin of Joe Biden’s victory by just 12 electoral college votes. In fact, Miller’s only major wrong call was forecasting a Trump win in Georgia. This time, Miller learned from his misreading of the Peach State and the nailed the outcomes of the two senatorial races. Just after midnight on January 5, Miller predicted Democratic challengers Raphael Warnock and Jon Ossoff would sweep both races, defeating Senators would Kelly Loeffler and David Perdue by 2% and 1% respectively.

In the final count, Warnock prevailed 50.9% to 49.1%, a margin of 1.8 points, while Ossoff won 50.5% to 49.5%, or by 1 point. Miller was just .2% off on Warnock vs Loeffler, and hit the precise mark on Ossoff-Perdue. Miller’s coup was showing a sudden shift towards the two Democrats in the final days of the election that lifted both from underdogs to moderate favorites. Polling was light in Georgia, but the few surveys didn’t perceive the strength of that shift as accurately as Miller. The last Fox5/InsiderAdvantage reading had both races a tie, and final Trafalgar Group poll called Ossoff-Perdue dead heat, and showed Warnock trailing by two points.

Miller says that it was original analytics he developed specifically for the Georgia runoffs that yielded 100% accuracy in the Ossoff race, and 90% in the Warnock win. “I was attempting to fix what’s broken in election forecasting,” Miller told Fortune. “The predictions as reported in the media are broken.” The problem, he says, is that forecasting comes in two main categories, and both suffer from substantial drawbacks. The first type are the polls that canvas carefully-chosen samples of probable voters. “Those surveys have two problems,” says Miller. “First, one pollster’s sample is different from another’s, even though both are trying to choose the most representative populations. Second, polls tend to lean Democratic. Pollsters then need to adjust the results for that political bias, and also tweak the numbers for example, if they don’t get enough Hispanic voters in their latest survey.” For Miller, those differing adjustments cause lots of variability both across different polls, and even in the same surveys over time, rendering them highly unreliable guides to which candidates lead or trail by what margin.

The second category encompasses the modelers. The two most influential practitioners are the Economist and Nate Silver’s FiveThirtyEight. “The modelers start with everything they know about previous elections, and then incorporate a single number from each of the polls showing where the candidates stood on the day of the survey,” as Miller. The rub, he says, is that the modeling in largely based on polling done by somebody else. “They use one summary number from each poll––one candidate’s leading 51% to 49%, say––but that’s the only data they get from the polls. That limits what they can do with their models.”

Miller’s goal was to combine polling and modeling under one roof into a single system incorporating far more data than either deploy individually. In the past, he’d relied on betting odds instead of either polls or models, specifically the prices posted on the single major venue available in the U.S., Predictit.org. “The betting odds are much more accurate predictors than surveys or models, but they’re dominated by higher-income folks who are mainly male, and often bet on sports. So they have a Republican bias.” The gambling sites were mostly right-on in predicting a much narrower Biden win than the polls, but got Georgia wrong because of that GOP tilt. It was an over-reliance on those odds that caused Miller’s misfire on the Peach State in the presidential race.

Miller created a new methodology where he first conducted a conventional poll to ask whom voters expected to back, and what variables, notably race, sex and where they lived, influenced their preference. That’s the polling part. Then, he united that survey with a model that incorporated an element of political gaming, but at the same time eliminated the Republican bias. Here’s how his “magic beans” operated. To organize the survey, Miller partnered with polling specialists Isometric Solutions, Panel Consulting Group, and Prodege LLC. In the six days from December 30 to January 4, he canvassed 1,200 potential voters. Once again, his questions fell into two categories, first for the regular poll, called the Preference Survey, and second for the model designed to generate a more accurate, final result, tagged as the Prediction Survey.

In Preference Survey, Miller included only people who’d lived in Georgia at least a year, and who said they were likely to vote. “That’s really asking whom they hope will win, which candidate they’re big fans of,” he says. In the Prediction survey, he posed a different question: Which candidate do you expect to win? He added a financial carrot. Respondents were paid $1.50 for participating, but they’d get an additional $1.50 after the election for each race they called correctly, or $4.50 instead of $1.50 if they got both contests right. “That’s a small amount, but it worked,” says Miller. “The incentives turned folks into bettors. Only they weren’t the same mainly male sports bettors as on the gaming sites. That removed the GOP bias, while at the same time encouraging them to think hard about their choice.” In the Prediction Survey, Miller also included Georgians who didn’t intend to vote. His view was that everyone had lived in Georgia at least a year, so even the non-voters saw the brewing trends in their communities and counties. Including everyone, he reckoned, would provide a more complete picture of how Georgians on-the-ground were thinking.

Over the six day window, the Preference and Prediction Surveys showed a widening gulf. The former, the regular poll, had Ossoff and Warnock winning by commanding margins of over 5%, while the Prediction survey that incorporated the payments also showed a steady trend towards the Democrats, but much more competitive races. Miller was banking that the model, the Prediction Survey, was the best guide. “My view was that across all black male voters or white female voters, for example, their overall bets on which candidate would win were a better indicator of how that group would actually vote than their answer to the question, ‘to whom do you intend to vote for,” he says. “We learned more by asking, ‘Whom do you expect to win.'”

The results from the Prediction Survey told Miller how each of four groups, non-white men and women, and white men and women, were most likely to vote. But getting the final estimate required one last step. “I had to estimate another crucial factor, the size of the turnout,” he says. Miller predicted the total numbers who’d vote early or on Election Day would fall halfway between the presidential race of 2016 and the mid-term contest of 2018. He already knew the probabilities of how the four groups would vote from the Prediction Survey. He then forecast how many people in each of the categories would cast ballots in every one of Georgia’s 159 counties. “That’s a total of 636 separate forecasts,” he says.

On the eve of the election, Miller worked until past midnight combining his estimates of how many people in each group would vote with the likelihood of whom they’d vote for, once again taken from his model. Just hours before the polls opened, he posted his forecast that Ossoff and Warnock would notch one and two point victories respectively. The Miller model needs more testing, but its remarkable performance in Georgia may mean that he’s charting a whole new direction through the fog of forecasting elections.

More politics coverage from Fortune:

• “We will never concede”: How Donald Trump incited an attack on America
• Democrats plan to use Senate win to pass $2,000 stimulus checks
• Photos: A look at the nationwide riots on Wednesday
• Attempted coup at Capitol presents key opportunity for cyberattack, experts warn
• Betting odds heavily favored Georgia’s GOP candidates, then suddenly collapsed. What went wrong?