From John Lott:
Here is a suggestion. If you have a question about whether something was done in the survey, it might make more sense to ask the question whether it was done rather than asserting that it must not be true. I had simply asked James to write up specific points, particularly how the survey sample was gathered. The discussion that I sent you dealt with the issues that had previously been raised.
There are significant problems with using a five year window. Sure it helps you get a bigger sample of defensive gun uses, but there is also a lot more error. For example, using five years is likely to results in respondents including cases that go back even further than five years. Answers to questions about what happened are also likely to contain more errors.
So Lott wants to claim that his survey ‘improves upon’ Kleck and Gertz (1995) because he limits it to one year. That might have some merit if he used the same or better survey techniques and his sampling was anything close to the quality of Kleck and Gertz’s.
He didn’t. In fact, the Kleck and Gertz survey called back to verify results on all DGUs, instituted several criteria to ensure accuracy in relation to actual civilian defensive use, and significant screening was used to determine the nature of the reported DGU unit.
Anytime anyone is asked to report a memory, the farther back the event, the lest trustworthy the memory. Lott is correct in this and this is why one would use probe the memory and attempt to establish credibility. Additionally, self-reporting can often inflate the reports, something Kleck was very concerned with, something Lott, despite protestations of being provided extensive details by all DGU respondents, did not do in the instrument.
Of course, what Lott is not mentioning and is highly relevant, is that Kleck and Gertz asked about DGUs within the last year as well. In fact, they have more cases of that subsample than does Lott per Table 2 (scroll down). Kleck and Gertz don’t break down the one year rate of a weapon being fired, but it is reported as being higher than in the 5 year reports. Of course, the sample size is small and has a larger maring of error than the 222 total (or 213 depending on which sample they are discussing). That MOE is still smaller than Lott’s and it is from a far better constructed survey.
For a good discussion of the issues surrounding 1 or 5 year samples see Kleck and Gertz. Lott’s complaint about the 5 year window is especially curious in this case given Kleck and Gertz explicitly cover the issue in detail and account for it.
As for the 2002 survey, a number of calls (form the surveyors end) were indeed randomly listened to by me. In all defensive gun uses, the surveyors were debriefed that night or the following morning about the call. All the respondents in these cases volunteered extensive details of what happened with the defensive gun use. None of the defensive gun uses recorded involved defensive uses by police. A couple of our surveyors had previous experience and I asked them to talk to the other surveyors before surveying began. As a result of call backs, over 50 percent of telephone numbers produced completed interviews.
In a pattern that is becoming all too apparent, Lott tries to equate a fly by the seat of your pants approach to doing his survey to having trained surveyors.
This speaks to Lott’s poor understanding of surveys. Even individuals who do a lot of survey research generally rely upon survey experts to conduct and help construct the surveys. The relevant passages in Kleck and Gertz are:
The present survey is the first survey ever devoted to the subject of armed self-defense. It was carefully designed to correct all of the known correctable or avoidable flaws of previous surveys which critics have identified. We use the most anonymous possible national survey format, the anonymous random digit dialed telephone survey. We did not know the identities of those who were interviewed, and made this fact clear to the Rs. We interviewed a large nationally representative sample covering all adults, age eighteen and over, in the lower forty-eight states and living in households with telephones. [42] We asked DGU questions of all Rs in our sample, asking them separately about both their own DGU experiences and those of other members of their households. We used both a five year recall period and a one year recall period. We inquired about uses of both handguns and other types of guns, and excluded occupational uses of guns and uses against animals. Finally, we asked a long series of detailed questions designed to establish exactly what Rs did with their guns; for example, if they had confronted other humans, and how had each DGU connected to a specific crime or crimes.
We consulted with North America’s most experienced experts on gun-related surveys, David Bordua, James Wright, and Gary Mauser, along with survey expert Seymour Sudman, in order to craft a state-of-the-art survey instrument designed specifically to establish the frequency and nature of DGUs. [43] A professional telephone polling firm, [Page 161] Research Network of Tallahassee, Florida, carried out the sampling and interviewing. Only the firm’s most experienced interviewers, who are listed in the acknowledgements, were used on the project. Interviews were monitored at random by survey supervisors. All interviews in which an alleged DGU was reported by the R were validated by supervisors with call-backs, along with a 20% random sample of all other interviews. Of all eligible residential telephone numbers called where a person rather than an answering machine answered, 61% resulted in a completed interview. Interviewing was carried out from February through April of 1993.
The quality of sampling procedures was well above the level common in national surveys. Our sample was not only large and nationally representative, but it was also stratified by state. That is, forty-eight independent samples of residential telephone numbers were drawn, one from each of the lower forty- eight states, providing forty-eight independent, albeit often small, state samples. Given the nature of randomly generated samples of telephone numbers, there was no clustering of cases or multistage sampling as there is in the NCVS; [44] consequently, there was no inflation of sampling error due to such procedures. To gain a larger raw number of sample DGU cases, we oversampled in the south and west regions, where previous surveys have indicated gun ownership is higher. [45] We also oversampled within contacted households for males, who are more likely to own guns and to be victims of crimes in which victims might use guns defensively. [46] Data were later weighted to adjust for oversampling.
Each interview began with a few general "throat-clearing" questions about problems facing the R’s community and crime. The interviewers then asked the following question: "Within the past five years, have you yourself or another member of your household used a gun, even if it was not fired, for self-protection or for the protection of property at home, work, or elsewhere? Please do not include military service, police work, or work as a security guard." Rs who answered "yes" were then asked: "Was this to protect against an animal or a person?" Rs who reported a DGU against a person were asked: "How many incidents involving defensive uses of guns against persons happened to members of your household in the past five years?" and "Did this incident [any of these incidents] happen in the past twelve [Page 162] months?" At this point, Rs were asked "Was it you who used a gun defensively, or did someone else in your household do this?"
All Rs reporting a DGU were asked a long, detailed series of questions establishing exactly what happened in the DGU incident. Rs who reported having experienced more than one DGU in the previous five years were asked about their most recent experience. When the original R was the one who had used a gun defensively, as was usually the case, interviewers obtained his or her firsthand account of the event. When the original R indicated that some other member of the household was the one who had the experience, interviewers made every effort to speak directly to the involved person, either speaking to that person immediately or obtaining times and dates to call back. Up to three call- backs were made to contact the DGU-involved person. We anticipated that it would sometimes prove impossible to make contact with these persons, so interviewers were instructed to always obtain a proxy account of the DGU from the original R, on the assumption that a proxy account would be better than none at all. It was rarely necessary to rely on these proxy accounts– only six sample cases of DGUs were reported through proxies, out of a total of 222 sample cases.
While all Rs reporting a DGU were given the full interview, only a one-third random sample of Rs not reporting a DGU were interviewed. The rest were simply thanked for their help. This procedure helped keep interviewing costs down. In the end, there were 222 completed interviews with Rs reporting DGUs, another 1,610 Rs not reporting a DGU but going through the full interview by answering questions other than those pertaining to details of the DGUs. There were a total of 1,832 cases with the full interview. An additional 3,145 Rs answered only enough questions to establish that no one in their household had experienced a DGU against a human in the previous five years (unweighted totals). These procedures effectively undersampled for non-DGU Rs or, equivalently, oversampled for DGU-involved Rs. Data were also weighted to account for this oversampling.
Questions about the details of DGU incidents permitted us to establish whether a given DGU met all of the following qualifications for an incident to be treated as a genuine DGU: (1) the incident involved defensive action against a human rather than an animal, but not in connection with police, military, or security guard duties; (2) the incident involved actual contact with a person, rather than merely investigating suspicious circumstances, etc.; (3) the defender could state a specific crime which he thought was being committed at the time of the incident; (4) the gun was actually used in some way–at a minimum it had to be used as part of a threat against a person, either by [Page 163] verbally referring to the gun (e.g., "get away–I’ve got a gun") or by pointing it at an adversary. We made no effort to assess either the lawfulness or morality of the Rs’ defensive actions.
An additional step was taken to minimize the possibility of DGU frequency being overstated. The senior author went through interview sheets on every one of the interviews in which a DGU was reported, looking for any indication that the incident might not be genuine. A case would be coded as questionable if even just one of four problems appeared: (1) it was not clear whether the R actually confronted any adversary he saw; (2) the R was a police officer, member of the military or a security guard, and thus might have been reporting, despite instructions, an incident which occurred as part of his occupational duties; (3) the interviewer did not properly record exactly what the R had done with the gun, so it was possible that he had not used it in any meaningful way; or (4) the R did not state or the interviewer did not record a specific crime that the R thought was being committed against him at the time of the incident. There were a total of twenty-six cases where at least one of these problematic indications was present. It should be emphasized that we do not know that these cases were not genuine DGUs; we only mean to indicate that we do not have as high a degree of confidence on the matter as with the rest of the cases designated as DGUs. Estimates using all of the DGU cases are labelled herein as "A" estimates, while the more conservative estimates based only on cases devoid of any problematic indications are labelled "B" estimates.
The question remains, what does John Lott think his survey is going to produce of value? Comparing the two methods of conducting a quality survey identifies how unconcerned Lott is with his research’s accuracy.