The Union of Concerned Scientists has received a lot of attention over the past few days for a study they released in which they claimed, according to the NY Times that :
it is common for scientists to be pressured to eliminate references to climate change, for their work to be changed to misrepresent their findings, and for climate-related materials to disappear from Web sites.
Almost 60 percent of the scientists who responded to the survey said they had personally experienced such an incident in the last five years, the report says, and those who said their work was most closely related to climate change experienced the most interference.
That sounds bad, you have to wonder if the study is scientific. It has to be scientific because they are the Union of Concerned Scientists and because they say they are scientists they must be produce scientific work, right? Wrong. The Wall Street Journal reports:
The survey is here (PDF), and a closer look at it ought to raise some doubts.
The relevant questions, 19-33, appear on pages 4-5. Questions 19-30 list 12 “types of activities affecting climate science” and ask the respondent if he has “perceived in others and/or personally experienced” them. (Question 31, a catch-all “other” category, can be ignored, since few bothered even responding to it.)
One problem is that of these 12 questions, only three–Nos. 20, 24 and 25–clearly indicate that the scientist responding agrees with the Union of Concerned Scientists on climate issues. A scientist who reports “self-induced pressure to change research or reporting in order to align findings with agency policy or to avoid controversy” (No. 23), for example, could feel such pressure to avoid raising doubts about global warming.
Note, too, the wording of that question: “self-induced pressure.” The scientists who answered “yes” to this question are reporting on their own state of mind, not any objective facts that may bear on it. The same is true of questions 24 and 25, which refer to “fear of retaliation,” and 27, which refers to “implicit expectation.”
Many of these questions, too, simply reflect the realities of working in any bureaucracy, public or private. No. 19 asks if the scientists have perceived of experienced “changes/edits during review that change the meaning of scientific findings.” Some have, but we have no basis on which to judge the merits of the disagreements between the scientists and their editors.
The biggest problem with the survey, though, is its basic methodology, explained on the first page of the PDF:
Following is the text of the survey UCS mailed to 1,630 federal climate scientists at seven federal agencies and departments, along with response data for the 279 scientists who completed and returned surveys.
That is, only about 17% of the scientists who received the survey actually filled it out and returned it. There is no reason to think this is a representative sample of the total population, and it seems reasonable to surmise that people who would go to the trouble of completing such a survey are more likely than those who wouldn’t to perceive themselves as under political pressure–i.e., to agree with the UCS.
To put it much more simply, this was an unscientific survey. If this is how these guys do social science, how can we trust them with the hard stuff?
Well said. These guys aren’t scientists, they are charlatans.
