The report of the InterAcademy Council on the IPCC is a real step forward. The council, representing academies of science and equivalent bodies all over the world, assembled a review committee on the IPCC at the request of the UN and the IPCC itself. Its team was chaired by Princeton's Harold Shapiro. How refreshing that it surveyed opinion very widely, and has grappled intelligently with the most persuasive criticisms that the panel's adversaries have been pressing. I think it struck just the right balance between recognizing the panel's accomplishments and criticizing its methods and organization. The best part, though, is that its judicious stance may well achieve some useful results, because its recommendations are going to be difficult to ignore.
Climate-change skeptics are mostly celebrating a report that, they say, slams the IPCC. A little to my surprise, champions of the panel and its work see a report that, in the words of RealClimate, appears mostly sensible and has a lot of useful things to say. (The Washington Post, in one of the strangest pieces of commentary I have read in some time, sees the IAC report as an actual endorsement of IPCC procedures, as though the report calls on the panel to keep up the good work.) Contrary to what you might think, these apparently conflicting interpretations are encouraging. They mean that neither side is dismissing the report's recommendations -- which (whatever the Post may imagine) are substantive and far-reaching. Finally, consensus. Governments should get on with it, and make the recommended changes forthwith.
In most media accounts, attention has focused on what appears to be the implicit recommendation that Rajendra Pachauri, the IPCC's current head, should step down. The committee says the chief (and other key people) should serve one term, and Pachauri is already into his second. I am no fan of Pachauri, and I think it would be far better if he went, but this is less important than repairing other aspects of the panel's governance and methods. Among the new report's many recommendations, I would stress the following.
Recommendation: The IPCC should establish an Executive Committee to act on its behalf between Plenary sessions. The membership of the Committee should include the IPCC Chair, the Working Group Co-chairs, the senior member of the Secretariat, and 3 independent members, including some from outside of the climate community. Members would be elected by the Plenary and serve until their successors are in place. [My emphasis.]
Broadening the range of perspectives is vital. Aside from creating an executive committee with outsider representation, one of the best ways to do this is through the IPCC's review process. The report looks at this in detail, recognising that many of the panel's critics have focused on it. Review Editors play a key role in the IPCC's work -- or ought to. They oversee the IPCC's Lead Authors' treatment of reviewers' comments on their drafts. The IAC explains:
Two or more Review Editors for each chapter oversee the review process, ensuring that review comments and controversial issues are handled appropriately. However, the Lead Authors have the final say on the content of their chapter.
With the tight schedule for the revision process, authors do not always consider the review comments carefully, potentially overlooking errors in the draft report that might have been caught...
A near-universal observation -- made in presentations, interviews, and responses to the questionnaire -- was the need to strengthen the authority of the Review Editors to ensure that authors consider the review comments carefully and document their responses. With the tight schedule for completing revisions, authors do not always do an adequate job of revising the text and Review Editors do not always require them to explain why they rejected a comment... Although a few such errors are likely to be missed in any review process, stronger enforcement of existing IPCC procedures by the Review Editors could minimize their numbers. This includes paying special attention to review comments that point out contradictions, unreferenced literature, or potential errors; and ensuring that alternate or dissenting views receive proper consideration.
Recommendation: The IPCC should encourage Review Editors to fully exercise their authority to ensure that reviewers' comments are adequately considered by the authors and that genuine controversies are adequately reflected in the report. [My emphases.]
Enabling and requiring Review Editors to do their job is critical. Partly, as the report says, this is a matter of resources. But the report also acknowledges that encouraging Review Editors to fully exercise their authority is not enough by itself to make the review process fully independent. Further thought will need to be given to this.
Although implementing these recommendations would greatly strengthen the review process, it would not make the review process truly independent because the Working Group Co-chairs, who have overall responsibility for the preparation of the reports, are also responsible for selecting Review Editors. To be independent, the selection of Review Editors would have to be made by an individual or group that is not engaged in writing the report, and Review Editors would report directly to that individual or group (NRC, 1998, 2002).
Despite the desirability of an independent review, it is not clear what scientific body has the recognized legitimacy and capacity to carry out such a large task. At the NRC, a special group called the Report Review Committee carries out this function on behalf of the institution. The Report Review Committee is made up of approximately 30 members of the National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, is staffed by individuals from outside the program units, and reports directly to the NRC Governing Board. One option for the IPCC would be to appoint a small group of experts who would report directly to a new Executive Committee (see "IPCC Management Structure" in Chapter 4) to serve a similar function for the IPCC. Another option would be to engage an international scientific body to provide such services for the IPCC.
An important related issue is the selection of authors.
Although the individuals who corresponded with the Committee were generally supportive of the author teams chosen, few knew why some authors are chosen and others are not, and the selection criteria seemed arbitrary to many respondents. The absence of a transparent author-selection process or well-defined criteria for author selection can raise questions of bias and undermine the confidence of scientists and others in the credibility of the assessment (e.g., Pielke, 2010a). The IPCC has no formal process or criteria for selecting authors, although some Working Group Co-chairs established their own for the fourth assessment, considering factors such as scientific expertise and excellence, geography, gender, age, viewpoint, and the ability to work in teams. Establishing such criteria and applying them in a transparent manner to all Working Groups would alleviate some of the frustrations voiced by the community.
Recommendation: The IPCC should establish a formal set of criteria and processes for selecting Coordinating Lead Authors and Lead Authors. [My emphasis.]
The report has this to say about "handling the full range of views".
An assessment is intended to arrive at a judgment of a topic, such as the best estimate of changes in average global surface temperature over a specified timeframe and its impacts on the water cycle. Although all reasonable points of view should be considered, they need not be given equal weight or even described fully in an assessment report. Which alternative viewpoints warrant mention is a matter of professional judgment. Therefore, Coordinating Lead Authors and Lead Authors have considerable influence over which viewpoints will be discussed in the process. Having author teams with diverse viewpoints is the first step toward ensuring that a full range of thoughtful views are considered.
Equally important is combating confirmation bias -- the tendency of authors to place too much weight on their own views relative to other views (Jonas et al., 2001). As pointed out to the Committee by a presenter [John Christy of the University of Alabama, Huntsville] and some questionnaire respondents, alternative views are not always cited in a chapter if the Lead Authors do not agree with them. Getting the balance right is an ongoing struggle. However, concrete steps could also be taken. For example, chapters could include references to all papers that were considered by the authoring team and describe the authors' rationale for arriving at their conclusions.
Recommendation: Lead Authors should explicitly document that a range of scientific viewpoints has been considered, and Coordinating Lead Authors and Review Editors should satisfy themselves that due consideration was given to properly documented alternative views. [My emphases.]
The report has a lot to say about the way the IPCC has characterised and communicated uncertainty. For instance, commenting on one document (Working Group 2 Summary for Policymakers), it says:
The Working Group II Summary for Policy Makers has been criticized for various errors and for emphasizing the negative impacts of climate change. These problems derive partly from a failure to adhere to IPCC's uncertainty guidance for the fourth assessment and partly from shortcomings in the guidance itself. Authors were urged to consider the amount of evidence and level of agreement about all conclusions and to apply subjective probabilities of confidence to conclusions when there was high agreement and much evidence. However, authors reported high confidence in some statements for which there is little evidence. Furthermore, by making vague statements that were difficult to refute, authors were able to attach "high confidence" to the statements. The Working Group II Summary for Policy Makers contains many such statements that are not supported sufficiently in the literature, not put into perspective, or not expressed clearly. [My emphasis.]
The IPCC has no formal policy on conflicts of interest. The report says this should change.
Recommendation: The IPCC should develop and adopt a rigorous conflict of interest policy that applies to all individuals directly involved in the preparation of IPCC reports...
In developing such a policy, the IPCC may want to consider features of the NRC policy. These include:
- Distinguishing between strong points of view (i.e., biases) that can be balanced and conflicts of interest that should be avoided unless determined to be unavoidable
- Differentiating between current conflicts, where the candidate's current interests could be directly and predictably affected by the outcome of the report, and potential conflicts of interests
- Considering a range of relevant financial interests...
- Judging the extent to which an author or Review Editor would be reviewing his or her own work, or that of his or her immediate employer
- Examining indications of a fixed position on a particular issue revealed through public statements (e.g., testimony, speeches, interviews), publications (e.g., articles, books), or personal or professional activities
- Maintaining up-to-date confidential disclosure forms and participating in regular, confidential discussions of conflict of interest and balance for the major components of each report. [My emphasis.]
The IPCC's communications efforts come in for criticism too.
IPCC's mandate is to be policy relevant, not policy prescriptive. However, as noted above, IPCC spokespersons have not always adhered to this mandate. Straying into advocacy can only hurt IPCC's credibility. [My emphasis.]
I think it is a pity that the IAC will not be publishing the questionnaire responses from the hundreds of experts they approached, except apparently in aggregated form with names withheld. That hardly conforms to a spirit of openness. Perhaps somebody will assemble a list of links to those responses that have already found their way on to the web. That would be a useful service.