How scientists think — and fight
March 29, 2010
Today’s guest blogger is the best science writer in the country named Steve Easterbrook. Steve
is a professor of computer science at the University of Toronto. He wrote a much admired
comment on RealClimate, which offers a rare look into the scientific mindset.
In the interest of bridging the two cultures, I asked if I could reprint it, and Steve expanded it to
add context and links. It has been posted on his blog with the title, “Academics always fight over
the peer-review process.”
Note: This started as a comment on a thread at RealClimate about the Guardian’s investigation
of the CRU emails fiasco. The Guardian has, until recently, had an outstandingly good record on
it’s climate change reporting. It commissioned Fred Pearce to do a detailed investigation into the
emails, and he published his results in a 12-part series. While some parts of it are excellent,
other parts demonstrate a complete misunderstanding of how science works, especially the
sections dealing with the peer-review process. These were just hopelessly wrong, as
demonstrated by Ben Santer’s rebuttal of the specific allegations. In parallel, George Monbiot,
who I normally respect as one of the few journalists who really understands the science, has
been arguing for Phil Jones to resign as head of the CRU at East Anglia, on the basis that his
handling of the FOI requests was unprofessional. Monbiot has repeated this more recently, as
can be seen in this BBC clip, where he is hopelessly ineffective in combating Delingpole’s
nonsense, because he’s unwilling to defend the CRU scientists adequately.
The problem with both Pearce’s investigation, and Monbiot’s criticisms of Prof Jones is that
neither has any idea of what academic research looks like from the inside, nor how scientists
normally talk to one another. The following is my attempt to explain this context, and in particular
why scientists talking freely among themselves might seem to rude or worse. Enough people
liked my comment at RC that I decided to edit it a little and post it here (the original has already
been reposted at ClimateSight and Prof Mandia’s blog). I should add one disclaimer: I don’t
mean to suggest here that scientists are not nice people – the climate scientists I’ve gotten to
know over the past few years are some of the nicest people you could ever ask to meet. It’s just
that scientists are extremely passionate about the integrity of their work, and don’t take kindly to
people pissing them around. Okay, now read on…
Once we’ve gotten past the quote-mining and distortion, the worst that can be said about the
CRU emails is that the scientists sometimes come across as rude or dismissive, and say things in
the emails that really aren’t very nice. However, the personal email messages between senior
academics in any field are frequently not very nice. We tend to be very blunt about what appears
to us as ignorance, and intolerant of anything that wastes our time, or distracts us from our work.
And when we think (rightly or wrongly) that the peer review process has let another crap paper
through, we certainly don’t hold back in expressing our opinions to one another. Which is of
course completely different to how we behave when we meet one another. Most scientists
distinguish clearly between the intellectual cut and thrust (in which we’re sometimes very rude
about one another’s ideas) and our social interactions (in which we all get together over a beer
and bitch about the downsides of academic life). Occasionally, there’s someone who is unable to
separate the two, and takes the intellectual jabs personally, but such people are rare enough in
most scientific fields that the rest of us know exactly who they are, and try to avoid them at
conferences.
Part of this is due to the nature of academic research. Most career academics have large egos
and very thick skins. I think the tenure process and the peer review process filter out those who
don’t. We’re all jostling to get our work published and recognised, often by pointing out how
flawed everyone else’s work is. But we also care deeply about intellectual rigor, and preserving
the integrity of the published body of knowledge. And we also know that many key career
milestones are dependent on being respected (and preferably liked) by others in the field, such
as the more senior people who might get asked to write recommendation letters for tenure and
promotion and honors, or the scientists with competing theories who will get asked to peer review
our papers.
Which means in public (e.g. in conference talks and published papers) our criticisms of others
are usually carefully coded to appear polite and respectful. For example, a published paper
might talk about making “an improvement on the methodology of Bloggs et al”. Meanwhile, in
private, when talking to our colleagues, we’re more likely to say that Bloggs’ work is complete
rubbish, and should never have been published in the first place, and anyway everyone knows
Bloggs didn’t do any of the work himself, and the only decent bits are due to his poor, underpaid
postdoc, who never gets any credit for her efforts. (Yes, academics like to gossip about one
another just as regular people do). This kind of blunt rudeness is common in private emails,
especially when we’re discussing other scientists behind their backs with likeminded colleagues.
Don’t be fooled by the more measured politeness in public: when we think an idea is wrong, we’ll
tear it to shreds.
Now, in climate science, all our conventions are being broken. Private email exchanges are being
made public. People who have no scientific training and/or no prior exposure to the scientific
culture are attempting to engage in a discourse with scientists, and neither side understands the
other. People misquoting scientists, and trying to trip them up with loaded questions. And,
occasionally, resorting to death threatst. Outside of the scientific community, most people just
don’t understand how science works, and so don’t know how to make sense of what’s going on.
And scientists don’t really know how to engage with these strange outsiders. Scientists normally
only interact with other scientists. We live rather sheltered lives; they don’t call it the ivory tower
for nothing. When scientists are attacked for political reasons, we mistake it for an intellectual
discussion over brandy in the senior common room. Scientists have no training for political
battles, and so our responses often look rude or dismissive to outsiders. Which in turn gets
interpreted as unprofessional behaviour by those who don’t understand how scientists talk. And
unlike commercial organisations and politicians, universities don’t engage professional PR firms
to make us look good, and we academics would be horrified if they did: horrified at the expense,
and horrified by the idea that our research might need to be communicated on anything other
than its scientific merits.
Journalists like Monbiot, despite all his brilliant work in keeping up with the science and trying to
explain it to the masses, just haven’t ever experienced academic culture from the inside. Hence
his call, which he keeps repeating, for Phil Jones to resign, on the basis that Phil reacted
unprofessionally to FOI requests. But if you keep provoking a scientist with nonsense, you’ll get
a hostile response. Any fool knows you don’t get data from a scientist by using FOI requests, you
do it by stroking their ego a little, or by engaging them with a compelling research idea that you
need the data to pursue. And in the rare cases where this doesn’t work, you do some extra work
yourself to reconstruct the data you need using other sources, or you test your hypothesis using
a different approach (because it’s the research result we care about, not any particular dataset).
So to a scientist, anyone stupid enough to try to get scientific data through repeated FOI
requests quite clearly deserves our utter contempt. Jones was merely expressing (in private) a
sentiment that most scientists would share – and extreme frustration with people who clearly don’
t get it.
The same misunderstandings occur when outsiders look at how we talk about the peer-review
process. Outsiders tend to think that all published papers are somehow equal in merit, and that
peer-review is a magical process that only lets the truth through (hint: we refer to it more often as
a crap-shoot). Scientists know that some papers are accepted because they are brilliant, are
others are accepted because its hard to tell whether they are any good, and publication might
provoke other scientists to do the necessary followup work. We know some published papers are
worth reading, and some should be ignored. So, we’re natural skeptics – we tend to think that
most new published results are likely to be wrong, and we tend to accept them only once they’ve
been repeatedly tested and refined.
We’re used to having our own papers rejected from time to time, and we learn how to deal with it
– quite clearly the reviewers were stupid, and we’ll show them by getting it published elsewhere
(remember, big ego, thick skin). We’re also used to seeing the occasional crap paper get
accepted (even into our most prized journals), and again we understand that the reviewers were
stupid, and the journal editors incompetent, and we waste no time in expressing that. And if there’
s a particularly egregious example, everyone in the community will know about it, everyone will
agree it’s bad, and some of us will start complaining loudly about the idiot editor who let it
through. Yet at the same time, we’re all reviewers, and some of us are editors, so it’s understood
that the people we’re calling stupid and incompetent are our colleagues. And a big part of calling
them stupid or incompetent is to get them to be more rigorous next time round, and it works
because no honest scientist wants to be seen as lacking rigor. What looks to the outsider like a
bunch of scientists trying to subvert some gold standard of scientific truth is really just scientists
trying to goad one another into doing a better job in what we all know is a messy, noisy process.
The bottom line is that scientists will always tend to be rude to ignorant and lazy people, because
we expect to see in one another a driving desire to master complex ideas and to work damn hard
at it. Unfortunately the outside world (and many journalists) interpret that rudeness as
unprofessional conduct. And because they don’t see it every day (like we do!) they’re horrified.
Some people have suggested that scientists need to wise up, and learn how to present
themselves better on the public stage. Indeed, the Guardian published an editorial calling for the
emergence of new leaders from the scientific community who can explain the science. This is
naive and irresponsible. It completely ignores the nature of the current wave of attacks on
scientists, and what motivates them. No scientist can be an effective communicator in a world
where those with vested interests will do everything they can to destroy his or her reputation.
The scientific community doesn’t have the resources to defend itself in this situation, and quite
frankly it shouldn’t have to. What we really need is for newspaper editors, politicians, and
business leaders to start acting responsibly, make the effort to understand what the science is
saying, make the effort to understand what really driving these swiftboat-style attacks on
scientists, and then shift the discourse from endless dissection of scientists’ emails onto useful,
substantive discussions of the policy choices we’re faced with.
– Steve Easterbrook
http://climateprogress.org/2010/03/29/how-scientists-think-peer-review-global-warming/
My thanks to Dave Haith.
March 29, 2010
Today’s guest blogger is the best science writer in the country named Steve Easterbrook. Steve
is a professor of computer science at the University of Toronto. He wrote a much admired
comment on RealClimate, which offers a rare look into the scientific mindset.
In the interest of bridging the two cultures, I asked if I could reprint it, and Steve expanded it to
add context and links. It has been posted on his blog with the title, “Academics always fight over
the peer-review process.”
Note: This started as a comment on a thread at RealClimate about the Guardian’s investigation
of the CRU emails fiasco. The Guardian has, until recently, had an outstandingly good record on
it’s climate change reporting. It commissioned Fred Pearce to do a detailed investigation into the
emails, and he published his results in a 12-part series. While some parts of it are excellent,
other parts demonstrate a complete misunderstanding of how science works, especially the
sections dealing with the peer-review process. These were just hopelessly wrong, as
demonstrated by Ben Santer’s rebuttal of the specific allegations. In parallel, George Monbiot,
who I normally respect as one of the few journalists who really understands the science, has
been arguing for Phil Jones to resign as head of the CRU at East Anglia, on the basis that his
handling of the FOI requests was unprofessional. Monbiot has repeated this more recently, as
can be seen in this BBC clip, where he is hopelessly ineffective in combating Delingpole’s
nonsense, because he’s unwilling to defend the CRU scientists adequately.
The problem with both Pearce’s investigation, and Monbiot’s criticisms of Prof Jones is that
neither has any idea of what academic research looks like from the inside, nor how scientists
normally talk to one another. The following is my attempt to explain this context, and in particular
why scientists talking freely among themselves might seem to rude or worse. Enough people
liked my comment at RC that I decided to edit it a little and post it here (the original has already
been reposted at ClimateSight and Prof Mandia’s blog). I should add one disclaimer: I don’t
mean to suggest here that scientists are not nice people – the climate scientists I’ve gotten to
know over the past few years are some of the nicest people you could ever ask to meet. It’s just
that scientists are extremely passionate about the integrity of their work, and don’t take kindly to
people pissing them around. Okay, now read on…
Once we’ve gotten past the quote-mining and distortion, the worst that can be said about the
CRU emails is that the scientists sometimes come across as rude or dismissive, and say things in
the emails that really aren’t very nice. However, the personal email messages between senior
academics in any field are frequently not very nice. We tend to be very blunt about what appears
to us as ignorance, and intolerant of anything that wastes our time, or distracts us from our work.
And when we think (rightly or wrongly) that the peer review process has let another crap paper
through, we certainly don’t hold back in expressing our opinions to one another. Which is of
course completely different to how we behave when we meet one another. Most scientists
distinguish clearly between the intellectual cut and thrust (in which we’re sometimes very rude
about one another’s ideas) and our social interactions (in which we all get together over a beer
and bitch about the downsides of academic life). Occasionally, there’s someone who is unable to
separate the two, and takes the intellectual jabs personally, but such people are rare enough in
most scientific fields that the rest of us know exactly who they are, and try to avoid them at
conferences.
Part of this is due to the nature of academic research. Most career academics have large egos
and very thick skins. I think the tenure process and the peer review process filter out those who
don’t. We’re all jostling to get our work published and recognised, often by pointing out how
flawed everyone else’s work is. But we also care deeply about intellectual rigor, and preserving
the integrity of the published body of knowledge. And we also know that many key career
milestones are dependent on being respected (and preferably liked) by others in the field, such
as the more senior people who might get asked to write recommendation letters for tenure and
promotion and honors, or the scientists with competing theories who will get asked to peer review
our papers.
Which means in public (e.g. in conference talks and published papers) our criticisms of others
are usually carefully coded to appear polite and respectful. For example, a published paper
might talk about making “an improvement on the methodology of Bloggs et al”. Meanwhile, in
private, when talking to our colleagues, we’re more likely to say that Bloggs’ work is complete
rubbish, and should never have been published in the first place, and anyway everyone knows
Bloggs didn’t do any of the work himself, and the only decent bits are due to his poor, underpaid
postdoc, who never gets any credit for her efforts. (Yes, academics like to gossip about one
another just as regular people do). This kind of blunt rudeness is common in private emails,
especially when we’re discussing other scientists behind their backs with likeminded colleagues.
Don’t be fooled by the more measured politeness in public: when we think an idea is wrong, we’ll
tear it to shreds.
Now, in climate science, all our conventions are being broken. Private email exchanges are being
made public. People who have no scientific training and/or no prior exposure to the scientific
culture are attempting to engage in a discourse with scientists, and neither side understands the
other. People misquoting scientists, and trying to trip them up with loaded questions. And,
occasionally, resorting to death threatst. Outside of the scientific community, most people just
don’t understand how science works, and so don’t know how to make sense of what’s going on.
And scientists don’t really know how to engage with these strange outsiders. Scientists normally
only interact with other scientists. We live rather sheltered lives; they don’t call it the ivory tower
for nothing. When scientists are attacked for political reasons, we mistake it for an intellectual
discussion over brandy in the senior common room. Scientists have no training for political
battles, and so our responses often look rude or dismissive to outsiders. Which in turn gets
interpreted as unprofessional behaviour by those who don’t understand how scientists talk. And
unlike commercial organisations and politicians, universities don’t engage professional PR firms
to make us look good, and we academics would be horrified if they did: horrified at the expense,
and horrified by the idea that our research might need to be communicated on anything other
than its scientific merits.
Journalists like Monbiot, despite all his brilliant work in keeping up with the science and trying to
explain it to the masses, just haven’t ever experienced academic culture from the inside. Hence
his call, which he keeps repeating, for Phil Jones to resign, on the basis that Phil reacted
unprofessionally to FOI requests. But if you keep provoking a scientist with nonsense, you’ll get
a hostile response. Any fool knows you don’t get data from a scientist by using FOI requests, you
do it by stroking their ego a little, or by engaging them with a compelling research idea that you
need the data to pursue. And in the rare cases where this doesn’t work, you do some extra work
yourself to reconstruct the data you need using other sources, or you test your hypothesis using
a different approach (because it’s the research result we care about, not any particular dataset).
So to a scientist, anyone stupid enough to try to get scientific data through repeated FOI
requests quite clearly deserves our utter contempt. Jones was merely expressing (in private) a
sentiment that most scientists would share – and extreme frustration with people who clearly don’
t get it.
The same misunderstandings occur when outsiders look at how we talk about the peer-review
process. Outsiders tend to think that all published papers are somehow equal in merit, and that
peer-review is a magical process that only lets the truth through (hint: we refer to it more often as
a crap-shoot). Scientists know that some papers are accepted because they are brilliant, are
others are accepted because its hard to tell whether they are any good, and publication might
provoke other scientists to do the necessary followup work. We know some published papers are
worth reading, and some should be ignored. So, we’re natural skeptics – we tend to think that
most new published results are likely to be wrong, and we tend to accept them only once they’ve
been repeatedly tested and refined.
We’re used to having our own papers rejected from time to time, and we learn how to deal with it
– quite clearly the reviewers were stupid, and we’ll show them by getting it published elsewhere
(remember, big ego, thick skin). We’re also used to seeing the occasional crap paper get
accepted (even into our most prized journals), and again we understand that the reviewers were
stupid, and the journal editors incompetent, and we waste no time in expressing that. And if there’
s a particularly egregious example, everyone in the community will know about it, everyone will
agree it’s bad, and some of us will start complaining loudly about the idiot editor who let it
through. Yet at the same time, we’re all reviewers, and some of us are editors, so it’s understood
that the people we’re calling stupid and incompetent are our colleagues. And a big part of calling
them stupid or incompetent is to get them to be more rigorous next time round, and it works
because no honest scientist wants to be seen as lacking rigor. What looks to the outsider like a
bunch of scientists trying to subvert some gold standard of scientific truth is really just scientists
trying to goad one another into doing a better job in what we all know is a messy, noisy process.
The bottom line is that scientists will always tend to be rude to ignorant and lazy people, because
we expect to see in one another a driving desire to master complex ideas and to work damn hard
at it. Unfortunately the outside world (and many journalists) interpret that rudeness as
unprofessional conduct. And because they don’t see it every day (like we do!) they’re horrified.
Some people have suggested that scientists need to wise up, and learn how to present
themselves better on the public stage. Indeed, the Guardian published an editorial calling for the
emergence of new leaders from the scientific community who can explain the science. This is
naive and irresponsible. It completely ignores the nature of the current wave of attacks on
scientists, and what motivates them. No scientist can be an effective communicator in a world
where those with vested interests will do everything they can to destroy his or her reputation.
The scientific community doesn’t have the resources to defend itself in this situation, and quite
frankly it shouldn’t have to. What we really need is for newspaper editors, politicians, and
business leaders to start acting responsibly, make the effort to understand what the science is
saying, make the effort to understand what really driving these swiftboat-style attacks on
scientists, and then shift the discourse from endless dissection of scientists’ emails onto useful,
substantive discussions of the policy choices we’re faced with.
– Steve Easterbrook
http://climateprogress.org/2010/03/29/how-scientists-think-peer-review-global-warming/
My thanks to Dave Haith.
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
