A Survey Instrument to Measure Skeptics’ (Dis)Trust in Climate Science

Abstract

Existing survey instruments of trust in science and scientists that focus on the general public are potentially insufficient to assess climate skeptics’ perspectives towards climate science. They may miss important aspects of climate science about which skeptics raise concerns, and may not accurately measure climate skeptics’ distrust in climatology. We introduce a new survey instrument developed using data gathered from interviewing 33 self-identified climate change skeptics in Idaho. The survey items capture skeptics’ beliefs regarding climate scientists’ trustworthiness and credibility, their deference to scientific authority, and their perceptions of alienation from the climate science community. We validate our survey instrument using data from an online survey administered to 1000 residents in the U.S. Pacific Northwest who are skeptical of climate change. By employing standard survey design principles, we demonstrate how our new (dis)trust in climate science instrument performs in tandem with well-known predictors of science attitudes and pro-environmentalism.

1. Introduction

In 2019, climate scientists reached unanimity in their acceptance of anthropogenic global warming [1]. Yet, approximately 20% of the American public continues to remain skeptical about the realities and negative effects of climate change [2]. In recent years, social scientists have identified several scientific topics that have raised similar doubts and concerns among the public (e.g., vaccines, stem cell research, genetically modified food). On these topics, people who generally demonstrate receptiveness to scientific authority still question their findings and recommendations [3]. This phenomenon occurs despite the fact that general trust in science and support for scientific research has remained relatively high since the 1970s, compared to trust in other social institutions in the United States [4].

Trust in science and scientists is especially pertinent to climate change, a topic that has become increasingly politicized and polarized in the United States [5]. Lacking first-hand knowledge or information about climate change, reliance on experts becomes a critical determinant of public understanding of climate change as well as public support for adaptation and mitigation action [6]. Research suggests that people rely on trust or credibility of the messenger as a mental shortcut when determining whether they should accept novel information [7]. Particularly in situations where people are presented with conflicting scientific claims (such as the case of the politicized climate change debate in the United States), trust in relevant experts provides citizens with the ability to evaluate conflicting claims and map out courses of action [8].

Extant scholarship on trust in science and scientists typically rely on survey instruments to measure trust. Commonly used survey measures include the General Social Survey (GSS) question on confidence in the scientific community [9,10,11,12], trust in scientists as information sources [13,14,15], deference to scientific authority [16], trustworthiness of scientists [17], trust in science’s ability to solve problems [18,19], trust in production scientists versus impact scientists [20], trust in the scientific methodology [21], and general support for science [22,23].

Research examining how trust in science and scientists affect climate change attitudes and beliefs specifically have also relied on survey instruments, often operationalizing trust as the extent to which the public relies on scientists as a source of information on climate change [8,13,14,24,25,26]. In a 2009 study, Malka et al. found that, for people who trust scientists to provide credible information on environmental issues, knowledge about global warming leads to increased concern about the issue [8]. This effect was not observed for those who were distrusting of scientists. Hamilton et al. found a strong positive effect of education on trust in scientists as a source of information about climate change for Democrats, whereas the effect was close to zero for respondents who identified as Republicans and Tea Party supporters [24]. Examining the impacts of the “Climategate” scandal that directly challenged perceptions of trustworthiness of climate scientists, Leiserowitz et al. found a nine-point decline in “trust in scientists” between 2008 and 2010 [14].

Beyond trust in science and scientists, extant literature on climate change skepticism has identified several other factors that shape skeptical attitudes. While the present study focuses on trust, it is important to note that U.S. climate skepticism occurs within a cultural context in which the issue is increasingly politically polarized [5]. An active minority of contrarian climate scientists have continued to express skepticism about human-caused climate change [27]. Furthermore, deliberate and influential “disinformation campaigns”, coupled with elite cues perpetuated through some conservative media outlets, have continued to create climate change counter narratives that continue to challenge the scientific consensus on the issue [28,29,30].

Because of the key role of trust in communicating scientific information, operationalizing trust as whether or not respondents rely on scientists as a source of information has logical consistency. However, in the case of climate skepticism, recent qualitative research suggests that the reliance on this measurement alone may miss important nuances in climate skeptics’ concerns about climatalogy. Our prior qualitative research suggests that climate skeptics’ distrust about scientists as information sources might stem from a number of underlying factors: (1) concerns about incentive structures in climate science, (2) perceived motivations of climate scientists, (3) distrust of climate data and climate modeling, and (4) perceived exclusion of outside voices by the climate science community (i.e., alienation) [31].

Building on the themes above as identified by interviews with self-declared climate change skeptics, we posit that there is a need for a more nuanced and comprehensive survey instrument to measure climate skeptics’ (dis)trust in climate science. In the remainder of this article, we describe the methodology we employed to develop and validate such an instrument, uniquely focused on the approximately 20% of the American public who doubt the realities and negative effects of climate change [2]. Our analysis uses interview and survey data collected from self-identified climate change skeptics in the U.S. Pacific Northwest. Our findings show that (dis)trust in climate science explains skeptics’ environmental attitudes, and is shaped by predictors of climate skepticism among the general public, such as political ideology and gender [32].

2. Materials and Methods

2.1. Development of Survey Items for the “(Dis)Trust in Climate Science” Instrument

Between May 2017 and May 2018, we conducted 33 interviews with self-identified climate change skeptics in Idaho (One interview participant was from eight miles across the Idaho/Washington border). To identify and recruit skeptics for the study, we hung fliers in grocery stores in northern and southern Idaho and posted advertisements in commercial Facebook groups throughout Idaho. Recruitment ads used explicit language indicating that we were seeking self-declared skeptics: i.e., “If you are skeptical about human caused climate change, we would love to interview you. We are seeking to better understand information and arguments against climate change as well as distrust in science”.

In these interviews, we asked participants several questions related to trust in general and trust in climate science specifically (e.g., “Are there any entities that you think contribute to false narratives about climate change?”, “Do you believe that scientists are generally trustworthy?”, “Do they disseminate accurate information about health and environmental risks?”, “When it comes to information about climate change, how trustworthy are scientists?”, “How much and how often should politicians rely on scientists and experts when designing health and environmental policy?”). Interview data revealed several important nuances of climate skeptics’ distrust in climate science. In particular, we found that skeptics tend to question the legitimacy of climate models and data, incentive structures that could bias climatology (e.g., tenure pressures), and certain scientific practices (e.g., peer review procedures, academic conferences) that are seen as excluding outside voices. For further details of the methodology and themes identified, see Sarathchandra and Haltinner [31].

When considering survey items for our new “(dis)trust in climate science” instrument, we combined findings from the above qualitative work with insights from previous work on climate skepticism [33] and social science scholarship more broadly on science trust [20,34,35]. We then developed a series of Likert-scale items that tap into climate scientists’ trustworthiness and credibility (item # 1, 2, 3, 4, 9, 11, 12, 13, 16, 17), deference to scientific authority (item # 5, 10, 14, 15, 18), and perceptions of alienation from the climate science community (item # 6, 7, 8) among climate change skeptics. See

Table 1. Full list of survey items on skeptics’ (dis)trust in climate science.

Item #	Survey Item	Mean	Standard Deviation
1	Climate scientists do not have enough data to know that human-caused climate change is happening	4.44	1.50
2	Climate scientists are not open about their research	4.24	1.43
3	Climate scientists aren’t doing real science	3.70	1.51
4	Climate modeling isn’t science	3.91	1.43
5	Doing climate science is the same as making educated guesses	3.63	1.64
6	Climate scientists ignore those who disagree with them	4.54	1.49
7	Only people with PhDs are welcome to participate in discussions about climate change	3.21	1.57
8	Scientists write in a way that makes things difficult to understand	4.47	1.42
9	Scientific journals only publish papers that conclude climate change is happening	4.25	1.45
10	The fact that scientists have been wrong before suggests they are wrong about climate change	3.97	1.49
11	Climate science is driven by politics	4.68	1.61
12	Climate scientists are influenced by funding	4.83	1.49
13	Climate scientists are pressured to make certain claims in order to get tenure	4.53	1.45
14	Most climate scientists believe that climate change is caused primarily by human activities	4.58	1.36
15	There is no scientific consensus that human-caused climate change is happening	4.29	1.48
16	Climate scientists know what they are talking about	4.03	1.34
17	Climate scientists can’t be trusted	3.85	1.49
18	Farmer’s Almanac records show different trends than those of climate scientists	4.34	1.11

Survey question: Please indicate whether you DISAGREE or AGREE with the following statements about climate change; strongly disagree = 1 to strongly agree = 7.

for full list of items, their means and standard deviations.

In addition to focusing on the three theoretical dimensions of trust and credibility, deference to authority, and alienation, we employed standard survey design principles to reduce measurement errors. We used a combination of forward-worded and reverse-worded items to reduce acquiescence bias. Our overall survey question—“Please indicate whether you DISAGREE or AGREE with the following statements about climate change”—capture the full range of bivalent opinions from “strongly agree” to “strongly disagree”. When gathering online survey data for validation of the instrument, we employed item order randomization to minimize potential item order effects, which we also encourage for future research that employ these survey items.

2.2. Data Collection and Measurements

Our survey data come from an online survey of adults living in Washington, Oregon, and Idaho in the U.S. Pacific Northwest. The survey was distributed via Qualtrics, a firm that specializes in representative online surveys. We obtained ethical clearance for the survey procedure from the University of Idaho Institutional Review Board. Quota constraints were applied for gender and education for the sample to be representative of U.S. census data.

The survey was first “pilot tested in September 2019 and a soft launch was conducted in November 2019 (n = 50). Based on the results of the soft launch, a median time to completion of 20 min (measured as one-third the median soft launch time) was added as a speeding check, automatically removing from our final sample any respondents who completed the survey rapidly. The full survey data collection occurred between November 2019 and January 2020, subsequent to data quality checks and adjustments. Our final sample (N = 1000) consists of only respondents who completed all survey items, passed speeding checks, and met the above two quota constraints” [36].

The survey began with two screening questions. “In the screening questions respondents reported whether they believe (1) ‘climate change is happening’ and (2) ‘climate change is caused by human activities’ (response categories: yes = 1; no = 2; not sure = 3). Respondents who said that they believed climate change was happening and it was caused by human activities were screened out of our final sample, limiting our final sample to only those who expressed uncertainty regarding the realities and human causes of climate change (i.e., climate skeptics). Participants who met the screening criteria then proceeded to answer our full survey, which consisted of 45 questions” [36] on attitudes towards climate change, environment, policy, energy, information, and trust.

Forty-nine percent of the participants in our sample identified as men, 49.9% labeled themselves as women and 1.1% indicated they were neither men nor women. Approximately 42% revealed that they were slightly to very conservative. Roughly 35% were under 40 years of age. The majority of our participants had high school diplomas or some college level education. Eighty-nine percent categorized themselves as white. In terms of religiosity, approximately 42% of our participants selected that they “never” attend religious services while 17% stated that they attend religious services “more than once a week”. In addition, 12% of respondents attended religious services “about once a year”, while another 12% attended these services “a few times a year”. See

Table 2. Sample Characteristics ¹.

Characteristic	Mean (SD) or % (Frequency)
Gender
Women	49.9% (499)
Men	49.0% (490)
Other	1.1% (11)
Political ideology (very liberal = 1 to very conservative = 7)	4.55 (1.50)
Age (18–19 = 1 to >80 = 8)	4.54 (2.04)
Education (less than high school diploma or equivalent = 1 to doctoral degree = 8)	3.70 (1.83)
Race
White	89%
Other	11%
Religiosity (never attend religious services = 1 to attend services more than once a week = 7)	3.03 (2.19)

¹ Source: Adapted from Haltinner and Sarathchandra [36].

for a summary of these sample characteristics.

2.3. Statistical Analysis

To develop our (dis)trust in climate science scale, we used the results of a principal component analysis (PCA) with Direct Oblimin rotation and Kaiser normalization on the 18 items in Table 1. The PCA results suggested a three-factor solution (

Table 3. Rotated component matrix from factor analysis with construct reliability.

Item #	Scale Items	Component 1 (Distrust Scale)	Component 2	Component 3
1	Climate scientists do not have enough data to know that human-caused climate change is happening	0.787	−0.051	0.145
2	Climate scientists are not open about their research	0.779	−0.053	0.009
3	Climate scientists aren’t doing real science	0.769	0.303	0.019
4	Climate modeling isn’t science	0.711	0.296	0.022
5	Doing climate science is the same as making educated guesses	0.558	0.433	−0.124
6	Climate scientists ignore those who disagree with them	0.781	−0.222	−0.025
8	Scientists write in a way that makes things difficult to understand	0.470	−0.166	−0.382
9	Scientific journals only publish papers that conclude climate change is happening	0.727	−0.129	−0.160
10	The fact that scientists have been wrong before suggests they are wrong about climate change	0.744	0.272	0.001
11	Climate science is driven by politics	0.784	−0.244	0.169
12	Climate scientists are influenced by funding	0.746	−0.331	0.147
13	Climate scientists are pressured to make certain claims in order to get tenure	0.770	−0.294	0.050
15	There is no scientific consensus that human-caused climate change is happening	0.741	0.030	0.203
17	Climate scientists can’t be trusted	0.762	0.204	0.042
18	Farmer’s Almanac records show different trends than those of climate scientists	−0.243	0.611	−0.106
7	Only people with PhDs are welcome to participate in discussions about climate change	0.365	0.547	−0.478
14	Most climate scientists believe that climate change is caused primarily by human activities	−0.424	0.321	0.544
16	Climate scientists know what they are talking about	0.487	0.224	0.493
	Cronbach’s Alpha	(0.93)	(0.61)	(0.67)

Note: Factor loadings in bold indicate the highest loadings and the corresponding component/subscale in which the items were included.

) with 14 of the 18 items loading on component 1 (eigenvalue = 8.36; 53.9% of total variance explained), indicating that participants did not clearly distinguish between different theoretical dimensions of trust in climate science such as credibility, deference to authority, and alienation. Eigenvalues for components 2 and 3 were 0.78 and 0.74, respectively.

Given a majority of the items loaded on the first component with high internal reliability (Cronbach’s alpha 0.93), we use only these items in our final (dis)trust in climate science scale. All items were coded so that higher values indicate more distrust in climate science and scientists. In the final scale we combined these 14 items and calculated the average score for each respondent’s overall distrust level.

To validate this instrument, we conducted regression analysis employing two key outcome variables: environmental concern and pro-environmental policy support. Environmental concern (Table S1) measures the extent to which skeptics say they are concerned about a range of environmental issues (“not at all concerned” = 1 to “very concerned” = 7; Cronbach’s alpha = 0.96). Pro-environmental policy support (Table S2) measures skeptics’ support for selected policy initiatives (“not at all support” = 1 to “support a great deal” = 7; Cronbach’s alpha = 0.87). In creating these two outcome scales, we drew from and extended the New Environmental Paradigm (NEP) scale previously employed and updated by social scientists [37].

In terms of key predictors, we measured political ideology on a seven-point scale from very liberal (1) to very conservative (7). Gender is categorized as men = 1 and women/other = 0. Age is represented by eight categories: 18–19 = 1 to 80 or older = 8. Education is represented by the highest degree earned: less than high school diploma or equivalent = 1 to doctoral degree = 8. Religiosity asks respondents how often they attend religious services; never (1) to more than once a week (7). Race is a dummy variable where white = 1. See Table 2 above for descriptive statistics on these variables.

We employed multivariate OLS regression analysis to test the influence of our key predictors on skeptics’ (dis)trust in climate science (

Table 4. Unstandardized coefficients from OLS regression models explaining (dis)trust in science, environmental concern, and pro-environmental policy support among skeptics (standard errors in parenthesis).

Predictors	Model 1 (Dis)Trust in Climate Science	Model 2 Environmental Concern	Model 3 Environmental Concern	Model 4 Policy Support	Model 5 Policy Support
Liberal–conservative ideology	0.22 *** (0.02)	−0.08 ** (0.03)	0.01 (0.06)	−0.16 *** (0.03)	−0.12 (0.07)
Gender (men = 1)	0.27 *** (0.06)	−0.31 *** (0.09)	−0.30 *** (0.09)	−0.06 (0.09)	−0.06 (0.09)
Religiosity	0.05 *** (0.01)	−0.03 (0.02)	−0.04 (0.02)	−0.01 (0.02)	−0.01 (0.02)
Education	0.02 (0.02)	0.07 ** (0.02)	0.07 ** (0.02)	0.13 *** (0.03)	0.13 *** (0.03)
Race (white = 1)	0.04 (0.10)	−0.23 (0.14)	−0.23 (0.14)	0.06 (0.14)	0.06 (0.14)
Age	0.02 (0.01)	0.05 (0.02)	0.05 (0.02)	0.06 (0.02)	0.06 (0.02)
(Dis)trust in climate science	-	−0.20 *** (0.04)	−0.09 * (0.09)	−0.29 *** (0.04)	−0.24 ** (0.09)
(Dis)trust in climate science * political ideology	-	-	−0.05 (0.03)	-	−0.02 (0.03)
Constant	2.74 *** (0.14)	5.74 *** (0.22)	5.35 *** (0.35)	5.06 *** (0.23)	4.89 *** (0.36)
F Adjusted R2 N	32.3 *** 0.16 999	10.73 *** 0.07 999	9.66 *** 0.07 999	18.43 *** 0.12 999	16.16 *** 0.12 999

Note: * p < 0.05 ** p < 0.01 *** p < 0.001.

—Model 1). We also examined the effect of (dis)trust in climate science on pro-environmentalism by using the above two outcome scales, environmental concern and pro-environmental policy support, as dependent variables (Table 4—Models 2 and 4), controlling for the effects of key theoretical predictors of climate skepticism such as gender, political ideology, and race [32]. Finally, we included an interaction effect to test the potential political moderation of science distrust on pro-environmentalism among skeptics: “(dis)trust in climate science * political ideology” (Table 4—Models 3 and 5). The interaction term was centered on the mean prior to entering into multiple regression models, in order to reduce collinearity. We performed all analyses using IBM SPSS 24.0 software.

3. Results

Results from multivariate OLS regression models using (dis)trust in climate science as a dependent variable is reported in Table 4, Model 1. This model includes key sociodemographic predictors of general trust in science. This analytical model shows that, among climate change skeptics, gender, political ideology, and religiosity are significant predictors of (dis)trust in climate science, with moderate effects for gender (b = 0.27) and political ideology (b = 0.22) and a small effect for religiosity (b = 0.05). Results indicate that men who are skeptical of climate change are less likely to trust climatology compared to women who are also skeptical of climate change. Further, increasing degrees of political conservatism and religiosity both lead to intensifying levels of distrust in climate science among skeptics. These findings affirm prior literature on the effects of gender, political ideology, and religion on trust in science [38,39], but operates differently for education. Extant scholarship suggests a positive association between education and trust in science [21]. However, when other relevant predictors are held constant, we did not observe a significant effect of education on dis(trust) in climate science.

Results from multivariate OLS regression models using environmental concern and pro-environmental policy support as dependent variables are reported in Table 4, Models 2 to 5, that includes our new (dis)trust in climate science scale as one of the predictors. As mentioned above, Models 3 and 5 includes an additional interaction effect (distrust in climate science * political ideology) to test the potential political moderation of science distrust on climate beliefs.

Model 2 shows that when other important predictors of pro-environmentalism such as political ideology, education, gender, age and race [15] are held constant, (dis)trust in climate science has a negative and statistically significant effect on environmental concern (b = −0.20) among climate skeptics. Increasing distrust in climatology leads to decreasing levels of concern, suggesting that (dis)trust in climate science may be a more important predictor of environmental concern among people who doubt climate change than even political ideology. In this model, political ideology (b = −0.08), gender (b = −0.31), and education (b = 0.07), still emerge as statistically significant predictors of environmental concern, in line with prior literature.

Similarly, and in line with prior literature, Model 4 shows that when other key theoretical predictors of pro-environmentalism are held constant, (dis)trust in science leads to a statistically significant negative effect on pro-environmental policy support (b = −0.29). Additionally, political ideology (b = −0.16) and education (b = 0.13) emerge as significant predictors of policy support in the expected directions.

Prior literature on U.S. climate change skepticism suggests political orientation (conservative ideology) as the most robust predictor of skeptical attitudes [40], which appears to also moderate the relationship between education, scientific literacy [41,42] and climate change attitudes [43]. However, for our unique population of self-identified climate change skeptics, this political moderation effect does not appear to be consistent. As seen in our Models 3 and 5, we did not find a significant interaction effect of “distrust in climate science * political ideology” on environmental concern or policy support. The lack of a significant effect may be reflective of the unique nature of our sample (i.e., climate skeptics as opposed to the general public) and requires further research. Yet, (dis)trust in climate science appears to have a negative direct effect on environmental concern (b = −0.09) and pro-environmental policy support (b = −0.24) even when other key sociodemographic variables are held constant. Overall, results from our multivariate analysis shows that the new (dis)trust in climate science scale operates in the directions expected in tandem with predictors and outcome variables from prior literature, which is indicative of the validity of this new scale.

4. Discussion

Our new (dis)trust in climate science scale has a high Cronbach’s alpha value, demonstrating internal reliability. The scale also performs in expected directions with known predictors of science attitudes and pro-environmentalism, suggesting that the instrument has good construct validity. Men, political conservatives, and religious people have higher levels of scientific distrust regarding climate change. Further, distrust in climate science is associated with lower levels of environmental concern and lower support for pro-environmental policy among self-identified skeptics.

While Americans are now “as concerned as ever” about climate change, a significant fraction of the American public (about 20%) continues to remain skeptical [2]. According to Gallup surveys, this fraction of the public “worry little or not at all about global warming, do not think it will pose a serious threat in their lifetime, think it’s attributable to natural environmental changes and think the news exaggerates the problem”. These skeptical beliefs persist even while climate scientists have reached unanimity in their agreement on anthropogenic climate change [1]. Given the size of the skeptic population in the United States, it is important that scholars accurately capture the variability of attitudes among climate change skeptics and examine what factors explain these attitudes. It is also important that scholars accurately measure skeptics’ attitudes towards climatology and trust in climate science. We offer our new (dis)trust in climate science instrument as an initial step to better measure climate science attitudes among the unique population of U.S. climate skeptics.

In future research, this new science distrust scale should be incorporated alongside prior validated measures of climate skepticism such as Whitmarsh’s “Scepticism Scale” [33] to advance knowledge on climate change attitudes and beliefs. While Whitmarsh’s scale was developed and validated using a general population based in the U.K., our work uses self-declared groups of climate skeptics in the U.S. Comparisons across these scales may reveal important insights on geographic and temporal variations in climate change attitudes, with potential implications for communication and policy. Future research should also consider the extent to which the newly proposed distrust scale may be orthogonal to a trust scale, along with examining how climatology distrust and trust scales may operate across skeptic and non-skeptic subpopulations. Such comparisons will reveal the degree to which extant survey instruments on trust in science/scientists are adequate at measuring climate skeptics’ perspectives towards climate science.

Environmental studies scholars, science studies scholars, other educators and practitioners may employ our new (dis)trust in climate science instrument in a number of ways including (1) understanding climate skeptics’ views towards climatology, (2) examining effects of distrust on other environmental attitudes, pro-environmental policy support, and pro-environmental behavior, (3) assessing variability of attitudes within and among climate skeptics as opposed to the general public, and (4) measuring changes in distrust of climatology based on different interventions such as climate change messaging experiments and exposure to extreme weather events.

Extant scholarship on U.S. climate change skepticism identifies political orientation (conservatism) as the most robust predictor of skeptical attitudes [43]. As a result, conversations about U.S. climate skepticism have centered mainly on political polarization of climate change views. However, results from our study, which uniquely focuses on self-identified climate change skeptics, points to one additional and important aspect of skeptics’ attitudes, i.e., distrust in climate science, which seems to affect some of their environmental perspectives. This suggests that environmental and climate communication could benefit from refocusing attention on the scientific process itself.

In particular, to align skeptics’ beliefs about climate change with the scientific consensus on the issue, communication efforts should focus on climate skeptics’ specific concerns about climatology. For example, the climate science community may attempt to better communicate what climate modeling entails (e.g., How does modeling of complex environmental phenomena differ from “educated guessing”?). The climate science community should invite non-scientists into discussions about the science of climate change as well as appropriate action to mitigate its effects, and combat feelings of alienation and marginalization experienced by some skeptics. Furthermore, where skeptics seem to distrust climate science based on perceived nefarious motivations—e.g., “scientific journals only publish papers that conclude climate change is happening”—scientists may address those concerns by highlighting the valued role of “dissent” in science. Contradictory to what skeptics believe as a publication bias favoring anthropogenic climate change, dissent in science would suggest the alternative hypothesis, according to Powell: scientists with evidence against the phenomenon “would be sure to publish the findings and would have no trouble finding a journal to do so” [1].