ArchPundit

Is to analyze the polling—Charles Franklin has long called for more transparency in polling and uses this is an excellent example of where it could be positive.

If the polls were systematically flawed methodologically, then we’d expect similar errors with both parties. Almost all the pollsters did simultaneous Democratic and Republican polls, with the same interviewers using the same questions with the only difference being screening for which primary a voter would participate in. So if the turnout model was bad for the Democrats, why wasn’t it also bad for the Republicans? If the demographics were “off” for the Dems, why not for the Reps?

This is the best reason to think that the failure of polling in New Hampshire was tied to swiftly changing politics rather than to failures of methodology. However, we can’t know until much more analysis is done, and more data about the polls themselves become available.

A good starting point would be for each New Hampshire pollster to release their demographic and cross tab data. This would allow sample composition to be compared and for voter preferences within demographic groups to be compared. Another valuable bit of information would be voter preference by day of interview.

In 1948 the polling industry suffered its worst failure when confidently predicting Truman’s defeat. In the wake of that polling disaster, the profession responded positively by appointing a review committee which produced a book-length report on what went wrong, how it could have been avoided and what “best practices” should be adopted. The polling profession was much the better for that examination and report.

I won’t contradict Charles because he has forgotten more about public opinion than I’ve ever known, but one thing to keep in mind is that polls don’t necessarily fail when the results are different from the latest polling. If there is an intervening event between the final poll and the election, polling may have been right on the money, but an event caused public opinion to change. Charles says this is some sense above, but the notion that polling is wrong in this case is far from certain to me.
The problem is more how we interpret the polling. Polling is a snapshot in time and as such when the context changes, the underlying phenomenon is changing, not the earlier measure of it.

Whether this is exactly what happened in New Hampshire is rightfully an empirical question and the steps Charles is recommending are exactly the best way to address it. However, it isn’t just about addressing the methodology, but how people interpret polls in the first place.