David M Rothschild on Posted on

I am very excited to continue to help analyze polling data for MSN on important topics of the day. MSN’s opt-in polling is very cost-effective and fast to run, and this column helps explain how it is also accurate.

MSN runs polls on both its front-page (and some back-pages) and that data is collected by MSN. MSN first reports the raw counts for everyone see as they answer. But, we know that the respondents to the poll do not exactly match the US population, so we take the raw data and run it through a quick process. Step 1: model what the breakdown of the US adult population is by: age, gender, party identification, and location (e.g., what percentage of US adults are 18-29 year old, women, independents, from Kansas). Step 2: model how those groups of respondents would answer each questions (e.g., what percent of 18-29 year old, women, independents, from Kansas would answer a, b, c, or do to a given question). Step 3: project the model of how they answer the poll onto who is in the population.

This method can work with any type of question from political outcomes to sentiment around hamburgers or hot dogs. We just need a few answers to the poll question and a few key demographics of the person that answered it. We do not report margin of error because we do not believe it can be accurately estimated for either this method or traditional public polling. But, we believe the error to be comparable.

We know this method works, because we tested it in 2016 on MSN with the presidential election. In 2016, the MSN polls had binary accuracy that was similar to top public polling (46 of 51 states “correct”) and consistently pointed to Clinton’s trouble in the rust belt that ultimately cost her the election to Trump, with: a more Trump leaning voter population and more support for Trump from key demographics. We just could not figure out why Wisconsin, Michigan, and Pennsylvania looked to different from the public polling; turns out, it was because we were right!

PredictWise20161110c

The target population that my colleagues and I created for the MSN polling in 2016 more closely resembled the true voting population than the estimates from the public polling. This is best demonstrated by Nate Cohn’s interesting experiment of giving four different pollsters (and himself) the exact same polling data and see what topline numbers they generate. I was happy to be in this experiment and used our model of the voting population for Florida. We ultimately had a more Republican/Trump make-up state-by-state: older, whiter, and less educated, than the what the polls estimated. That is why for the same polling data, our projections were constantly more Trump (Trump won by 1 point in Florida).

PredictWise20161109c

As we move into 2017, we will focus more on the US adult population, but revert to the voting population when relevant.

Further, we had support levels for Trump and Clinton that closely resembled the exit polling. While it dipped at points, towards the end we consistently had Republicans in the MSN poll supporting Trump at a rate of 2 pp more than Democrats supported Clinton. To be honest, I was concerned; what had we done wrong? But, the MSN data was solid there. Further, we had strong support for Trump among older males (not surprising), but also surprising tightness in older females. We did not have the ability to parse this data by race, but we anticipate that if we could, it would have shown a strong than expected return from white females driving that. That is probably reflected in our state estimates from states with higher percentage white populations. All of this is backed up by the polling data and why our estimates from MSN were throwing rust-belt states towards Trump.

As we move into 2017, we will use the same methods that got 2016 so right, on the varied questions of the day. One more thing MSN brings to the table: I believe that MSN can reach folks in the middle of America that are missed by traditional polling. We can control for many things, but having a wide sample of Americans is certainty a great starting point.