Aug.20

electoral-vote.com

Dem 51

Ties 1

GOP 48

Why the Polling in Wisconsin Was So Wrong

Yesterday we had an item on why the polls for the Wisconsin gubernatorial election were so far off. As noted yesterday, one of the polls was completely fake, just made up. But there were other polls by legitimate pollsters, that also showed progressive Democrat Francesca Hong way, way ahead. G. Elliott Morris wanted to find out why the legitimate polls were so far off, so he made a deal with pollster State Navigate. The deal was they would give him all their internal data to analyze to find out what they did wrong (something they very much wanted to know). In return, instead of asking for money, he wanted permission to publish his report on his blog, which it granted and he did. Here it is, but be warned it is quite technical.

First, a bit of background. After the 2000 election fiasco in Florida, Congress passed the HAVA (Help America Vote Act). Among its other provisions, it requires states to maintain a proper database of voters, informally called the voter list. What is in the voter list varies by state, but often contains the voter's name, address, date and place of birth, race, gender (identity), precinct, party affiliation, telephone number, e-mail address, Social Security number, driver's license number, voter number, father's name, mother's name, maiden name, political donations, elections voted in, and sometimes more. All states sell their voter lists. Sometimes any legitimate entity can buy them (e.g., political campaigns, pollsters, etc.), sometimes anyone can buy them but only for specific purposes set out in law, sometimes there are more restrictions, and sometimes there are no restrictions at all. Here is a map showing how different states deal with these lists:

Prices vary from free (e.g., Florida) to $12,500 (Wisconsin). Most states charge less than $1,000.

The first thing Morris learned was how the poll was done. Pollsters rarely use random-digit dialing anymore because nobody answers the phone for unknown callers anymore. What State Navigate did bite the bullet and buy the Wisconsin voter list. This gave it a list of every registered voter in the state. Then it made a random sample of the actual registered voters and contacted enough of them to get a statistically significant number to agree to take the survey. It did a survey in July and then contacted the same voters in August. This is very significant because then they knew how many individual voters changed their mind in that month, which gave them useful information about trends.

The first good thing about working with a voter list is you know everyone is a registered voter in the state being polled for sure. With any other method of collecting respondents, you have to trust them when they say they are a registered voter. The sample State Navigate got didn't conform to their model of the electorate. It never does, though, so they corrected it in many ways, based on the voter list data and their own demographic questions at the end. If they wanted a sample with 22% Catholics and they got a sample with 19% Catholics, they counted each Catholic as 22/19 = 1.158 people. Likewise for all the other demographic characteristics they cared (and asked) about. With these corrections, they got a sample that matched what they thought the electorate would be—that is, the right number of Catholics, college graduates, progressives, old people, poor people, Black people, rural voters, and so on for every category.

Based on the raw data, the corrections, the polling results, and the election results, Morris determined that there were three sources of error, from largest to smallest:

• Bad Demographics: The July poll had Hong at 44% and David Crowley at 15%, a 29-point lead. The August poll had Hong ahead 44% to 22%, a 22-point lead. Five days later, Crowley won by half a point. Being off by more than 20 points is a massive error, even by primary standards, where the absolute value of the polling error is 13 points. Morris verified all the math to make sure there wasn't a bug in the software's arithmetic. There wasn't. The problem was elsewhere.
  
  The model was wrong. Based on the exit polls of previous elections, State Navigate assumed that 9% of the voters would be 18-29. The voter file showed only 2% in this range. The July poll sample showed 17% under 30—more than 8x more than their share of actual registered voters. The July sample had 17% as actual Democratic Socialists and 43% as progressive. This is way more than the 2016 exit poll showed (25%). State Navigate corrected for the very skewed sample, but not enough. It is very possible that every young socialist contacted was excited to spend hours discussing politics with the pollster whereas older moderates were more hit or miss, depending on how busy they were.
  
  State Navigate knew about the discrepancy, and while counting a voter as 1.158 voters is fine, counting a voter as 0.118 of a voter (in other words, taking that 17% and shrinking it down to 2%) is a huge correction. It is almost like making up numbers, and honest pollsters are hesitant to make such enormous corrections. So, State Navigate didn't. When Morris actually made the corrections, his corrected version of the poll was off by only 3 points. He thinks the bad sample gave Hong 5-10 extra points.
  
  
• Late Movement: The difference between the July and August polls was a hint that the voters were changing their minds. With candidates dropping like flies, it is not surprising that there was real movement. Comparing the August poll to the July one, the Hong-Crowley margin fell 7 points in 10 days. Crowley was picking up support from the candidates no longer running and the undecideds were deciding. They went 40% to Crowley, 20% to Hong, and 40% to the minor candidates. It is almost certain that this movement continued until Primary Day. So a second source of error above the bad sample and insufficient correction was genuine change in how people were going to vote that was missed because there were too few polls close to the election. Late movement away from her cost Hong 6-10 points.
  
  
• Non-response: It is hard to get a good measure of how the people who were contacted and refused to answer voted. You could ask them, but given that they won't respond, that won't work. What Morris did was change the demographics by fudging correction factors to force the poll predict the exact election outcome to see how the demographics changed. What he discovered is that the mix that predicted the correct results had more Crowley voters across demographic categories. His conclusion is that Crowley voters were more likely to refuse to be polled than the super enthusiastic Hong voters who were bursting at the seams to explain to the world why Hong was so great. Morris thinks non-response overweighted Hong by 2-5 points.

So there you have the anatomy of a polling fail. Morris' feedback to State Navigate was to be leery of a sample that contradicts the voter list too much. He also advised doing a sensitivity analysis. That consists changing some parameters up and down and seeing what it does to the results. If small changes in some parameters (e.g., the number of 18-29 year olds in the assumed electorate) changes the results a lot, be very wary. if small changes to all the parameters don't change the result much, you can be more confident in the results. All in all, an interesting study. (V)