First, clearly I was confused on the categorizations of variables. His response makes it clear how one gets to 36. Mea Culpa. Second, he has released tax records, though it is impossible to garner any useful information from them as to whether a survey was conducted. Conveniently, as with any other direct evidence of the survey, all supporting documentation is gone.
It appears that I significantly overestimated the amount of time for each call as well. The reason for this? I assumed the survey was at least attempting to be of some quality. For some context, one should examine the methodology of Kleck and Gertz’s (1995) survey found in Section C-1 of the paper.
So in the first case, Lott paid and accepted volunteer efforts by some undergraduates in performing a survey that:
1) was performed in dorm rooms
2) used untrained personnel
3) used mediocre sampling methods based on a commercial CD
4) had no Call-Backs for verification
5) had no effective supervision
6) had no questions regarding the context of the attack in relation to civilian versus police distinctions
7) had a question at the end whether the interviewer trusted the respondent with no objective guidelines
The list could go on…
Why was this survey ever done? It clearly couldn’t improve upon work already done and published by Kleck and Gertz in 1995. There are reasons to attempt different surveys after the work was done by another researcher.
1) Situation has changed.
Not relevant in the two years since Kleck and Gertz.
2) Replicate the results
Given how primitive the survey instrument and methodology are Lott could have no real hope of replicating Kleck and Gertz.
3) Improve on the findings by using some sort of innovation.
Again, this would rely on the survey using an alternative to the original study that could leverage more information out of respondents. Lott’s survey would not do that in any way.
So why do it? Good question. Not only did Lott use his own money, he wasted it to the tune of several thousand dollars. It would be impossible to account for the actual costs because:
a) Lott has no effective records of paying anyone. He paid some $8,000, but there isn’t any supporting documentation
b) apparently accepted students as volunteers and employees
c) no one can locate a student who did the research
Even better, Lott decided to replicate the missing survey using AEI interns in the same slipshod manner except this time the calls were made at AEI. He says the results will be available in his upcoming book and the sample size is ~1000. Other than covering himself, this survey serves no purpose.
In his first survey the sample of DGUs was 25 meaning a margin of error of +/- 20 percent. Compared to the Kleck and Gertz survey where oversampling produced 222 DGUs this is assanine. Their margin of error would be +/- 6.7 percent within the DGUs.
In responding to me:
Between the two surveys 3,439 people have been interviewed. Can more
information be obtained? Sure, but given my personal resources and
that these surveys are such a trivial portion of my overall interests I have spent about as much time as I plan on spending on this issue.
Actually, without the first survey, around 1,000 people have been surveyed. Lott appears to think that his previous results are worth something because they reside in his memory. I would point out that without the data, he won’t be able to use those results at all and given he doesn’t even have the original instrument, one can’t even tell if he was asking the same thing.
Ultimately, there is no point in this survey if he wishes to establish the rate of firearm use versus brandishing. Any claim to the contrary is statistical malpractice. Even more troubling, is that Lott is apparently disputing the results of well done surveys by others without even bothering to do a survey that might improve upon their results.
As Kieran Healy pointed out the other day, it seems quite clear one shouldn’t be going around spouting a number when all of the underlying research is now gone. To make matters worse, the data is utterly meaningless in regards to the 98% matter so why would one repeat it?
If you take Lott at his word and the first survey was done, I’m not sure that really adds to his credibility. It might change the reason for one’s conclusion that he isn’t credible. In one case he would be a fraud, in the other he could be an absolute incompetent or just someone selling a bunch of malarky.