Research 2000 Called into Question
The Daily Kos Website has been commissioning pollster Research 2000
to run polls for it. Now the founder, Markos Moulitsas, has
Research 2000 and told everyone to delete all its tracking polls from their
data bases due to a statistical analysis
that suggests they have been curbstoning (making up the data).
Kos, as he is known, didn't say anything about Senate polls, but if Research 2000 is
unreliable with tracking polls, it is probably unreliable about everything.
This leaves us with a big problem. While we are not going to start daily posts of the
polling data until September, when the primaries are largely over and people begin
paying attention to the election, at that point we will be faced with the problem
that very few pollsters are left and of those, some have other issues. SurveyUSA and
PPP are probably pretty solid but the largest "nonpartisan" pollster is Rasmussen,
who now works for Fox News. While Fox is theoretically a news organization, it has a
definite slant on the world, that say, ABC News does not have. There is no evidence
that Rasmussen makes up numbers, but there are other issues here. In particular,
most pollsters, including Rasmussen, have a model of the electorate and normalize
their polls to it. Very briefly, suppose Rasmussen believes that the set of likely
voters (which is very different from the set of registered voters and very, very
different from the set of people over 18) is 40% Republican, 30% Democratic, and 30%
independent. If an actual poll turns up 200 Republicans, 200 Democrats, and 200 independents,
he will weight each Republican respondent by 40/33, each Democratic respondent by
30/33 and each independent by 30/33 to compensate for the bad mix in his small sample.
In principle, such weighting is not only legitimate, but a good idea. But if the
model has too many of one party and too few of the other, it will affect the results
of course since the people in the underrepresented groups will count more. If the
pollsters released the raw data, then others could undo this effect, but most pollsters
do not release the raw data. In the case of Research 2000, it wouldn't even release
it to Kos, its own client, which is very fishy. Weighting is also done for gender,
age, and other demographic characteristics. Again, doing this is actually a good thing,
but a lot depends on how good the model is and whether that has been fudged at all.
One might think that it is easy to tell which pollsters are accurate and which are
not by comparing their polls to the final election results, but such is not the case.
For the final polls a few days before the election, this is true, but for general election
polls in June (which are probably not worth much anyway), there is no way to check.
It is not inconceivable that a polling organization with an ulterior motive could issue
biased polls most of the year (to encourage supporters of its favored party and discourage
the opposition) and then as the election approached, gradually remove the bias and
try to be as accurate as possible, to acquire a good reputation for next time.
Research 2000 has hired a law firm to try to intimidate the media (including the
blogs) into not talking about this issue. Nate Silver of 538.com has already
I'll check my mailbox later today expectantly.
Charles Franklin, a professor of political science at the University of Wisconsin
at Madison, is worried that scandals like this may hurt the polling industry as a whole.
is that everyone commissioning a poll should write into the contract that the pollster
must deposit the raw data in the Roper Archive at the University of Connecticut, which
collects them. This would allow third parties to analyze the raw data, do statistical
checks on them, like using Benford's Law,
and renormalize them using different models of the electorate.
We'll keep an eye out for further developments in the Research 2000 affair.
For the time being, they have been marked as suspect in the internal data base.
If you like this Website, tell your friends. You can also share by clicking this button
-- The Votemaster
Your donation is greatly appreciated. It will buy ads to publicize the site.