FRANKFURT. It looked like a routine decision that Steven Levitt had to make. The co-editor of the "Journal of Political Economy" (JPE), who is most renowned for his bestseller "Freakonomics", had to accept or reject a comment written by the Dallas-economist Stan Liebowitz on an article which had been published in the JPE.
Perhaps Levitt should have simply used his far-reaching powers as editor and reject the comment without much ado. He did not do this, He did something instead, which could potentially taint his own good reputation and the reputation of the JPE, and which exemplifies the relatively lax procedural standards at top-flight economics journals.
On closer look, everything about the case was unusual. The comment was phrased very strongly. It was a thinly veiled assertion of data-manipulation. This is quite remarkable, given that the JPE, edited at the University of Chicago where Professor Levitt teaches, is one of the top five economics journals globally.
The subject of the criticised article was a hot topic. The authors, Felix Oberholzer-Gee (Harvard) and Koleman Strumpf (University of Kansas), claimed to prove that file-sharing websites on the internet have not been responsible for the sharp decline in music sales since the turn of the millennium. Music labels might disagree. They have been busy suing file-sharing sites and their users for copyright infringement.
Professor Liebowitz disagreed strongly He had told Stephen much in a letter even before the article was published in the JPE. Liebowitz had pointed to what he considered numerous mistakes and inconsistencies in the paper and complained that the authors would not share their data for replication purposes. Levitt had forwarded the letter to the authors of the study, but had published the study more or less as submitted.
Ignoring Liebowitz was not possible any more, however, after he submitted his counter-study officially as a comment to the JPE in September 2007. Levitt started by asking one of the authors, Koleman Strumpf, for his opinion. Strumpf handed in his reply in November. He defends the study and retaliates by pointing to alleged mistakes in Liebowitz? comment.
In addition, Levitt asked for a report from an impartial referee. The referee recommends publishing the comment in order to "save subsequent researchers from building on a flawed research foundation." While he advises Liebowitz to rephrase his comment such that it would not contain any overt assertions of data manipulation he sides with him on almost all the critical points and comes to a damming conclusion regarding the file-sharing article: "I would suggest that the authors? conclusions are not warranted given the analysis and evidence that they provide."
However, Levitt is not inclined to publish the comment. He anonymizes the reply by Strumpf and uses it as a second referee-report on which he bases his rejection of Liebowitz? comment. "There is no doubt you raise some reasonable points. Nonetheless, I think the negative referee (negative toward the comment, N.H.) is correct in most of what he says", Levitt writes to Liebowitz. The only point he takes up from the impartial referee is the advice to moderate the tone, should Liebowitz wish to submit the comment to some other, lesser journal.
"The authors were being dishonest"
"I wrote the comment because I thought the authors were being dishonest, not just sloppy" is Liebowitz? comment to Levitt?s decision and advice. "There appears to be no way within the profession to adjudicate such claims. In fact, the profession tends to operate as if all economists are truthful all the time." He finds such an assumption highly naïve, as the profession does not reward replication and only very recently have a handful of journals even required that authors make replication possible, "I am supposed to hide my belief that the authors were essentially fabricating results. You are generally not allowed to be published saying such things", Liebowitz says and adds that the professions lack of care is highly troubling to him.
That Levitt?s re-labelling of the author?s reply became known is due to some special circumstances. One of the authors of the article had provided Handelsblatt with Strumpf?s reply earlier. This was in response to our request for comment on the dispute with Liebowitz. It was clear that the second "referee"-report differed from Strumpf?s reply to Levitt only by the missing letterhead and signature.
The rejection of the comment at the end of May was one of Levitt?s last acts as an editor of the JPE before his term expired. Confronted with the assertion that he had "misrepresented" an author?s reply as an anonymous referee report, Professor Levitts replies: "It is common practice at the JPE to solicit referee reports from authors of competing papers, or of an original paper when a comment is submitted. Then it is up to the editor, aware of potential biases, to make a judgment on the suitability of a paper for publication", he adds, pointing out that all referee reports are done anonymously.
He rejects the assertion that he "misrepresented" the author?s reply: "I did not in any way indicate that any report was or was not written by any particular scholar." Professor Levitt downplays the sensitivity of the dispute and sees no problem in handling biases. : "The issues on this paper are no more difficult or contentious than the ones I face on many papers. I handle papers of my friends and colleagues all the time", he writes.
When asked, the JPE would not confirm that it would indeed conform to the journals standards to use an author?s reply in lieu of an anonymous referee report. JPE-editor Robert Shimer answers the specific question evasively: "The decision to publish a paper or a comment in the journal ultimately belongs to the editorial board. We use referees to improve our own understanding of the paper and its contribution to the field. We may also consult with colleagues and other experts who may or may not provide a written report."
Professor Shimer did not want to comment on whether the editorial board might draw any conclusions for the journal?s review procedures. "In general, we believe that our refereeing and editorial procedures do a good job of selecting articles of current relevance that will have a long-term impact on economics research", he writes and argues that occasional mistakes should not cause a problem: "Since there are many other journals, those papers that were rejected because of a faulty decision process should be able to find a good home elsewhere."
Professor Liebowitz is not impressed by this consolation, which had been offered to him by Levitt before: "That statement is true for papers, but not so for comments. Most journals are loath to publish a comment on a paper appearing in another journal. It?s almost unheard of."
Lack of clear rules
The impression that procedural standards of economics journals are not particularly strict is widely shared in the profession. Zurich-based economist Ernst Fehr, an associate editor of the top-five journal "Quarterly Journal of Economics" and of "Science" points to a lack of clear rules as to when an editor should recuse himself because of potential prejudice. Science journals also seem to deal more openly with the competition among scientists. "Authors who submit an article to a science journal can say who they do not want to review their article", praises Fehr, a choice which is typically not given to economists.
One internationally renowned economist, who did not want to be named, expresses the complaint more bluntly: "Little scandals and big scandals are commonplace: editors who publish articles in their own journals, referees or editors who decide about articles submitted by their own doctoral students."
The internet provides Liebowitz with a different venue, to voice his doubts. His paper "How Reliable is the Oberholzer-Gee and Strumpf Paper on File-Sharing?" has been viewed almost 6000 times and downloaded more than 1200 times on the Social Science Research Network. This puts it close to the top-one-percent most downloaded papers on the network.
Links zum Thema im Web:
Link to Stan Liebowitz? » http://ssrn.com/abstract=1014399
Link to the paper by Felix Oberholzer-Gee and Koleman-Strumpf (working paper version): » http://www.unc.edu/~cigar/papers/FileSharing_March2004.pdf