John Lott’s book has been published with his new and supposedly improved survey. For those not following the John Lott train wreck concerning a survey he claims to have completed in 1998 check out Tim Lambert’s page in the hyperlink. Lott said he was replicating the survey and had this to say about it:
"let me note the most important bottom line: the survey that was done last fall produced very similar results. The earlier results were replicated."
Now that Lott’s survey has gone to press, Tim Lambert has revealed that the number of DGUs in the new sample is 7. Lott claims that in his previous study, he had 25 DGUs, but cannot provide any documentation for that survey.
Tim Lambert points out the
95% confidence interval is from 3 percent to 50 percent using Wilson’s method. This range would include the results from eight surveys done previously, all of which are better surveys.
Tim calculates using the Wilson method which I believe is the most appropriate method to understand these numbers. Tim or I or anyone with a decent statistical background can provide the technical aspects of the calculation. To put it in more familiar terms, the Margin of Error for the 7 cases of DGUs in the most recent survey is +/- 37.8 % with 95% confidence. Given there is one time the weapon is fired out of 7 DGUs, that means the range of possible results within the 95% confidence interval is 0 (You can’t have negative numbers here) and 52%. This is assuming that the survey’s methodology is adequate, and this is not likely the case.
Even if one were to use the results Lott claims from the first survey (and one shouldn’t since there is no data to analyze) the total number of DGUs would be 32 giving a margin of error of +/- 17.7%. With 1 weapon being fired in the 2002 sample and probably 2 being fired in the 1998 lost sample, that is 9.375% rate of firing the gun used in the DGU meaning up to 27% of cases having a weapon fired would be within the MOE. Using the Wilson method on both samples one observes the 95% confidence interval producing results between 3% and 24% of cases producing a weapon being fired. To insist that 98% of the time the weapon doesn’t have to be fired doesn’t actually fit his own results. Now Lott might try and claim that he uses weighting of the sample that would change this, but if he weights the sample his MOE inflates. All of this assumes unbiased sampling and effective survey methodology which is not likely given Lott’s survey techniques. IOW, all of these calculations are generous.
So Lott claims to have replicated the original survey. Tim Lambert disagrees. I agree with Lott on the point that he has replicated the original survey. Lott is correct, he has replicated a worthless survey with another worthless survey. Congratulations, Dr. Lott, you ought to be proud! The question Lott won’t answer is why he bothered with either survey. Anyone with a minimal grasp of statistics understands how useless the survey(s) he conducted are to answer the questions he wishes to answer. Others have insisted this survey would be much better, but it is another example of wasted time and effort.
I imagine Lott will release his numbers as well and that should provide for all sorts of fun!