The question John Lott won’t answer is why do a survey incapable of providing valid and reliable results for the question he is trying to ask. Lott has been shopping around the number that 98% of DGU (Defensive Gun Uses) occur without a shot being fired. This is significantly different than the results of others studies. No other study shows a figure of less than 21% of DGUs involving the weapon being fired. I’ve asked him three times and he has yet to provide a substantive response.
If one thinks there is a major methodological flaw that is inflating the rate of DGUs with a discharge, one would design a study that improved upon previous designs. His does not. It offers an instrument that is rudimentary, sampling methods that are suspect to be generous, and a sample size smaller than at least one previous study. In addition, the execution of both the first survey and second survey can kindly be called amateurish. The question I have asked him three times is how does his study hope to improve upon the 1995 paper by Kleck and Gertz. His only on topic response was:
There are significant problems with using a five year window. Sure it helps you get a bigger sample of defensive gun uses, but there is also a lot more error. For example, using five years is likely to results in respondents including cases that go back even further than five years. Answers to questions about what happened are also likely to contain more errors.
This is a terribly misleading response given that Kleck and Gertz’s 1995 piece looked at both 5 year and 1 year time frames because of the issues involved in 5 year recollections. In the single year time period, the Kleck and Gertz survey recorded 56 respondent DGUs and 68 Household DGUs, both numbers over twice the number in Lott’s first survey. Now in both surveys these samples are too small to make too strong of claims regarding the subsamples of DGUs. However, given Lott had 25 DGUs, his MOE is +/- 20 percent compared to Kleck and Gertz with +/- 13.4 percent (using individuals since Lott didn’t check on households–this is also problematic–see Lamber from the 12th). Even if Lott is foolish enough to try and claim both surveys as his sample size, he would end up with fewer cases than Kleck and Gertz. The newer survey being conducted has provided about 1015 responses and so if it is truly consistent with the mysterious first survey, the responses would only produce 10.5 more DGUs for a total of 35.5. This is still lower than the K & G survey sample of one year reports. And I made a mistake yesterday. Tim Lambert found the 1 year time frame produced identical rates for firing of the 5 year time frame respondents. Mea Culpa. As a note, Tim has been on this issue for far longer and while I’m hoping that I’m adding something, he has a far greater amount of information.
His other responses were complaining about what he feels is a misrepresenation by me:
As for the 2002 survey, a number of calls (form the surveyors end) were indeed randomly listened to by me. In all defensive gun uses, the surveyors were debriefed that night or the following morning about the call. All the respondents in these cases volunteered extensive details of what happened with the defensive gun use. None of the defensive gun uses recorded involved defensive uses by police. A couple of our surveyors had previous experience and I asked them to talk to the other surveyors before surveying began. As a result of call backs, over 50 percent of telephone numbers produced completed interviews.
Given the picture emerging of John Lott’s understanding of survey research, I’ll stand by my comment that the second survey isn’t an improvement and there was no effective supervision of callers. He does claim that a couple of the callers had some experience and they talked to the others. My, how comforting. Apparently he feels that his surveys are of the quality of the Kleck and Gertz survey, but can’t explain why.
John Lott is apparently uninterested in explaining how his survey is an improvement of better executed surveys in the past. Given this, one has to question whether Lott has a goal besides covering himself. Neither survey would be an improvement upon the Kleck and Gertz work or other works and so one is left with the uncomfortable feeling that John Lott is in the business of producing results that fit what he wants and not what he observes. The alternative is that he doesn’t understand the extent of the problem. Ignorance is not a flattering excuse in this case.
But to take this one step further, while Kleck and Gertz are limited in their one year conclusions, their survey is inadequate to make strong conclusions from and they point this out in their article. To add to the literature, Lott would need to not just meet that survey’s quality, he would need to improve upon it with a higher number of DGU respondents. Instead, he will have fewer, but he expects the public to accept the results. That’s chutzpah.
Jacob Levy added a comment regarding the incident. The IRB issues has been dismissed because the Chicago Law School apparently ignored such rules. This is disturbing, but arguably Lott was a part of a larger problem in that case. What is most disturbing, as I’ve said before, is that Lott had survey research with connecting information being stored and entered in dorm rooms. This isn’t a technical violation of the treatment of human subjects, this is a serious violation of how such material should be stored to protect human subjects’ privacy. Of all people, gun rights advocates should understand the importance of this.
In a larger sense, my views on gun control have largely been shaped by works like that of Kleck and Gertz. I don’t necessarily have a problem with concealed carry, I dislike HCI for many of the same reasons I dislike Lott, and I think guns should be generally available to the public though preferably with an FOID system as Illinois has. Why would some gun rights advocates work so hard to protect a charlatan when other credible researchers are out there that make solid arguments concerning the defensive use of guns that are overwhelmingly positive for gun rights advocates? Clearly, not all gun rights advocates take Lott seriously, but it seems that he is considered by some advocates as more credible and impressive than people like Kleck. I don’t get it.