We conducted a framed field experiment23 at the 2015 United Nations Climate Change Conference, COP21, held in Paris. We recruited 217 participants, representing more than 100 countries (the sample composition is described in Supplementary Table 9) and elicited their expectations for global temperature increases by 2100 before testing their responses to climate models’ projections. More than half of our respondents were climate negotiators, including eight heads of delegations. The others were non-negotiator policymakers from different communities.
In individual in-person interviews, we prompted policymakers for their prior probability distribution of four different intervals of year 2100 global temperature increases (<2 °C, 2–3 °C, 3–4 °C, >4 °C), following implementation of current NDCs. We provided policymakers with response scales using the IPCC numeric–verbal format (see Supplementary Information 8).
After eliciting policymakers’ prior distributions, we presented them with a specific extrapolation of the NDCs beyond 2030, where global emissions remained roughly constant throughout the century. We then presented policymakers with predicted 2100 temperature increases given that specific emission trajectory that were based on the transient climate response of all 30 climate models included in the 5th Assessment Report of the IPCC, WGI (Table 9.5)24. We presented policymakers with the results, shown in either one of the three boxplot formats in Fig. 3. These were introduced as follows: ‘the projections (in °C) as estimated by all climate models whose results on transient climate response are reported in the IPCC latest assessment report’. We then elicited the policymakers’ projections of long-term temperature conditional on the specified emission scenario (‘Based on the projections we have just shown you, and for each of the 4 ranges presented in the table below, could you please indicate the probability (or probabilities) that the temperature will be in that range.’). For this second round, we used again the response template shown in Supplementary Information 1 (in Supplementary Information 8 we report the full questionnaire used in the survey).
Figure 2 shows different ways of communicating the uncertainty in predictions across climate models. Subjects were randomly assigned to one of the three formats. This provides greater accuracy but lower treatment effects than a within-subjects design. When we asked policymakers for the second round of estimates of the probability distribution over possible 2100 temperature increases, we instructed them to consider the specified emission pathway as given, to isolate the impact of climate uncertainty alone. In both rounds of probability elicitation, we asked policymakers to report their level of confidence in their estimates.
In May 2016, a two-day simulation of a post-COP21 climate change negotiation (Climate Change Strategy Role Play held through CEMS – The Global Alliance in Management Education) took place in Erasmus University Rotterdam. This event involved MBA students from seven major European business schools who had received briefings in climate change science and UNFCCC climate negotiations. MBA students were playing the role of delegates to the COP21 process for a representative set of countries. These students had been preparing for this event for several months with documents including detailed background papers. We replicated the key portions of the experiment with a sample of 113 respondents. This MBA student sample is far more knowledgeable, in the content matter of the study, than any usual sample of students, or online survey subjects (because of their selection and preparation for the meeting). However, the students are less driven/influenced in their beliefs by national needs/agendas than actual climate negotiators, as they only play/act or simulate national roles.
For both the Rotterdam and Paris experiments, informed consent was obtained from participants, consistent with procedures of a protocol approved by the Institutional Review Board at Columbia University.
Analysis of priors.
We used the STATA command ‘sureg’ to perform the seemingly unrelated regression25. Demographic controls in the regressions are gender, age, number of children, and education (dummy for each category), as responses to questions 1, 2, 3 and 7 in the questionnaire (see Supplementary Information 8).
Description of regional coding.
The coding of country/region clusters is based, primarily, on self-reported country represented. Of the 217 subjects, 84 did not provide enough information to allow us to code the country they represent, reporting ‘None’, ‘UN’, ‘University’ or simply nothing. We coded those who did not fill in information according to their reported nationality. In this way, we coded the country/region cluster for 21 more observations.
The sample size is smaller than the total sample as some respondents did not fill either the country they represented or their demographic information.
Vulnerable countries/regions in our sample are: Afghanistan, Antigua and Barbuda, Bangladesh, Barbados, Bhutan, Burkina Faso, Central African Republic, Chad, Comoros, Congo RDC, Equatorial Guinea, Ethiopia, Fiji, Gabon, Ghana, Guatemala, Kenya, Latvia, Lebanon, Maldives, Marshall Islands, Mongolia, Morocco, Mozambique, Myanmar, Nepal, Pakistan, Palau, Panama, Papua New Guinea, Philippines, Somali, Salvador, Sudan, Swaziland, Tunisia, Togo, Tonga, Uganda, Vanuatu, Vietnam and Zambia.
Emerging economy countries/regions in our sample are: Argentina, Bangladesh, Brazil, Chile, China, Colombia, Hungary, India, Indonesian, Malaysia, Mexico, Pakistan, Panama, Peru, Philippines and Poland.
Energy exporter countries/regions in our sample are: Algeria, Australia, Brazil, Canada, China, Colombia, Georgia, Iraq, Latvia, Lebanon, Mongolia, Norway, Netherlands, Qatar, Russia, South Africa, Vietnam and United States of America.
High-emitter countries/regions in our sample are: Brazil, China, European Union, India, Japan, Russia and United States of America.
OECD members in our sample are: Australia, Austria, Belgium, Canada, Chile, Denmark, Finland, France, Germany, Hungary, Ireland, Israel, Italy, Japan, Korea, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Poland, Switzerland, United Kingdom and United States of America.
Analysis of conditional probabilities.
We consider four metrics to quantify the difference between reported conditional probabilities and the scientific information. The metrics used in the analyses performed are based on two factors: whether they are based on the differences bin-by-bin or aggregated across the four temperature bins (Overall); and whether they measure the magnitude of change (Continuous) or its direction (Dichotomous).
The following are the four metrics we used as dependent variables in the regressions, the first two continuous, the last two dichotomous.
Overall_dis (continuous, overall): Euclidean distance between overall probability distributions.
Bin_dis (continuous, bin-by-bin): Bin-by-bin absolute distance between probabilities.
Overall_closer (dichotomous, overall): Dummy variable indicating whether the Euclidean distance between the overall distribution of conditional probability and information is smaller (or greater) than the Euclidean distance between the overall distribution of prior and information.
Bin_closer (dichotomous, bin-by-bin): Bin-by-bin dummy variable indicating whether the absolute distance between conditional probability and information is smaller (or greater) than that between prior and information.
Raw versus normalized probabilities.
Only 18% of respondents reported probabilities for the four ranges of temperature increases that summed up to 100%.
A large literature has studied ‘binary additivity’, that is, testing whether P(Event) + P(Not Event) = 1. In most cases, and on average, this condition is satisfied. However, studies that have looked at partitions of discrete distributions with more than two outcomes, as in our case, all find a different behaviour. Indeed, results from ref. 26 show that additivity in such cases is much harder to achieve and in fact quite rare, while subadditivity is more common. Studies find evidence of subadditivity in judgements made by doctors27, by lawyers28, and by option traders29. Finding that n > 2 events sum to a probability > 1 may be driven by a bias toward the ‘case partition’ ignorance prior of 1/2 for each event, see ref. 30.
We found no significant differences between the COP21 and MBA students’ samples in terms of the additivity of their probability estimates of either distributions (priors and conditional probabilities).
For the purpose of our analyses we normalized the four subjective probabilities given by each individual to add up to 100%. Our main findings are robust to the exclusion of subadditive observations for either priors or conditional probabilities. For more information, see Supplementary Table 5, where we test the robustness of results presented to the use of raw data rather than normalized data.
Difference across formats.
Figure 4 and Supplementary Fig. 7 report for each temperature bin the proportion of respondents whose reported conditional probability is closer to the scientific information than the corresponding prior.
Respondents were asked to judge the provided information along two dimensions, credibility and informativeness. The range of scales for both variables credibility and informativeness associated with each format is from 1 to 7. There is no difference in credibility across formats (Kruskal–Wallis χ2 = 2.99, df = 2, p value = 0.22). Informativeness, however, is marginally different across the formats (Kruskal–Wallis χ2 = 5.00, df = 2, p value = 0.08).
Post hoc Dunn’s test with Bonferroni correction for two tests reveals that Format 3 is marginally more informative than Format 1 (p = 0.08) but there is no difference between Format 1 and Format 2 (p = 1.00) or between Format 2 and Format 3 (p = 0.16). Note that credibility and informativeness are measured in a between-subject design, so the identified difference in perceptions across formats could be bigger had the subjects been able to see multiple formats. Results are presented in the Supplementary Information (Supplementary Table 6).
Data availability.
The authors declare that data supporting the findings of this study are available online. Further information regarding the code used and the data produced are available from the corresponding author on request.
