Electronic referees

Judging a paper’s quality may be hard for human referees, and people are looking for alternatives. For instance, this recent PhysicsWeb news gives an overview of P. Chen et al. article Finding Scientific Gems with Google, where the authors take advantage of Google’s page rating algorithm to assess the relative importance of all publications in the Physical Review family of journals from 1893 to 2003. Since the rating algorithm weights pages by number of referrers [0], there’s in principle no value added to traditional citation indexes: both popularity measures are linearly correlated. The catch is that there are exceptions: papers that are not widely cited but that, judging for the number of web pages linking to them, seem to be much more influential than one would think (the article mentions quite a few, Feynman, Murray and Gell-Mann’s one on Fermi interactions being an example). Amusing; although i must confess that this kind of democratic assessments of our scientific endeavours remind me somewhat of a well-known Planck dixit:

Max PlanckA new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.

Max Planck, 1858–1947

(I remember i jotted down this same quotation some twenty years ago, together with a note showing my skepticism… nowadays i think i’m much more of a planckian than i used to be.)

Returning to our electronic referees, over at PhysOrg there’s a story about how computer science may help us in detecting bogus papers, where by bogus i mean automatically generated ones (looks like our human referees do sometimes find their task really hard!). Probably the most popular case of such a prank was the article accepted at WMSCI 2005 whose author came out to be SCIgen, an automatic paper generator created by the guy in portrait on the right. And our field is not immune to similar problems, as exemplified by the amusing Bogdanoff Affair (besides, as you’ll see, most probably no computer program would be of much help in this case).


[0] Of course, i’m oversimplifying: see here for the complete history behind Google’s PageRank.

Update: Andrew Jaffe, in his excellent blog (recommended, but you probably already knew it), has some interesting thoughts about peer review, and a recent initiative by Nature to open a debate on the issue and looking for ways of improvement.

Advertisements

3 Responses to “Electronic referees”

  1. srinivasa ramanujam Says:

    Is not the SCIgen picture actually on the left?

  2. koantum Says:

    I understand some of the change you underwent during the last 20 years. Something similar happened to me. I started out with a lot of faith in the honesty of scientists (particularly physicists), their determination to deceive neither others nor themselves. I had to change my mind. The extent to which they are willing to defend paradigms that have passed their expiry date is staggering.

    Tipler has an interesting article on peer review:
    http://anti–matters.blogspot.com/2006/06/tipler-on-refereed-journals.html

  3. jao Says:

    Srinivasa, is it? ;-)

    koantum, well, in my case it’s been a matter of realizing that physics is just a human endeavour, and, as such subject to all too human virtues and vices:

    The folly of mistaking a paradox for a discovery, a metaphor for a proof, a torrent of verbiage for a spring of capital truths, and oneself for an oracle, is inborn in us. -Paul Valery, poet and philosopher (1871-1945)

    I agree with Valery, and as a result don’t have a great problem with self-deception: it’s a fact of life. Besides, i’m very wary of calling a paradigm or idea expired. To begin with, the history of physics is full of circles; most importantly, i try to remember than i’m so prone to self-deception as anyone. Trying to deceive others (for instance, in the form we call hype) is, of course, a totally different matter.

    I don’t buy Tipler’s arguments at all. I dislike his ‘Geniuses vs. Stupid Ones’ ideas. Taking oneself for a martyr seems to be also inborn in us. (As an aside, i think that pre-WWII physics didn’t need peer review simply because it was a simpler world: the active physics community was so small that everyone knew each other personally… sadly, that didn’t prevent geniuses like Pauli truncating careers of stupid colleagues…). Tipler’s article strikes me as too simplistic, even manichean, and, as Wilde was fond of saying, the truth is rarely pure, and never simple :).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: