Arizona Science Policy: Uncertainty Lies at the Heart of Proposition 127

Here’s one way to make sense of the claims leveled by advocates and opponents.

Advertisements

Note from the Authors: We wrote this post as an op-ed for Arizona newspapers about Proposition 127, a state-wide ballot initiative that would require utilities to source 50% of electricity from renewable sources. 

Ballot initiatives in Arizona are supposed to appeal to the good citizen in all of us by providing an opportunity for citizens to carefully consider issues facing the state. They reflect Arizona’s fiercely independent spirit and politics. But anyone seeing ads about Proposition 127 might lament the state of Arizona politics.

Groups on both sides spread competing claims about the initiative’s possible impacts: Opponents say it will raise electricity bills $1,000 per year for Arizona households, while advocates say it will reduce utility bills by harnessing Arizona’s abundant sunshine. Both sides point to the amount of money spent by the other, and the sources of that money, as they compete for the political high ground.

The disputes between the sides don’t inform voters. Arguments by advocates appeal to people likely to support the initiative anyway. Arguments by opponents target those likely to vote against the initiative. Everyone else is left to make sense of contradictory claims insisted on by one side or the other.

But each side overstates its case. Neither wants to lay bare the uncertainties at the heart of their claims: It’s difficult to predict the future of energy systems. Changing global markets, shifting policies, and changes in technology cloud the spreadsheets and computer models that appear to so confidently predict future energy demand and prices. Twenty years ago, few experts predicted cheap natural gas, driven by fracking technologies, would replace coal across much of the U.S.

Citizens should be empowered to consider the uncertainties surrounding the claimed impacts of Prop 127. To help get there, ASU Researchers affiliated with the Consortium for Science, Policy and Outcomes created a guide about the initiative. The guide highlights important features of Arizona’s energy systems, the challenges those systems may face in the future, and the uncertainties that make long-range planning for energy difficult.

We created the guide through interviews with energy and policy experts, as well as various government and industry reports. It’s not meant to settle whether or not utility bills will increase or decrease, but to show what we know and don’t know about the complex and sometimes unpredictable relationships between policy and technical systems like our power grid. We did our best to strip away partisanship to provide information and balanced perspectives that help voters explore uncertainties and identify their values and preferences as they prepare to vote on Prop 127.

How can you use it? Read and reflect on it or use it to guide discussions with others. We recommend inviting a few neighbors or co-workers, hopefully including people with different values than your own, to discuss what each of you thinks is important. You might be especially surprised by what you learn from people you disagree with.

We’re scholars and students of the politics and uncertainties surrounding technical topics and we felt it our civic duty to create this guide and in the words of the late Senator John McCain, get into the arena. We hope this guide, and discussions that might stem from it, can exemplify how we can think more seriously about how to plan for Arizona’s future energy system, as well as other technical systems like driverless cars. Price estimates and technologies change; voters’ values—and the civil discussions that can create mutual understanding—are the brighter guidestar for dealing with uncertainty.

 

– Nich Weller & Michelle Sullivan Govani

 

Online Comments Are a Terrible Way to Involve the Public in Policy-making

Michelle and I have mentioned problems with public comment periods in several past blog posts, usually ending those notes with “but that’s a post for another day.” Well I did write something about public comment periods and on the urging of Michelle and others I submitted it to Slate, where the article was published today!

The gist? Online public comment periods, the primary means for public participation in policy-making, face relentless and repetitive comments from organized interest groups, do little to engage broad perspectives, and are subject to manipulation and alleged manipulation that undermine their credibility and utility. The editor picked an aggressive title (The online public comment period is a fraudulent cesspool) worthy of any social media post. But the subtitle is the real takeaway: We need better ways of involving the public in policy-making.

If the article piques your interest, check out some of our other posts, like one we wrote about Rand Paul’s proposed “taxpayer advocates” at the National Science Foundation or our reflections on public forums about climate change in Phoenix and Boston.

– Nich Weller

Climate change in context: Local creation, use, and interaction with science

Climate change is often framed as a national and international concern, a framing that masks the efforts of cities, universities, and parks that are sensitive to local climate change impacts. Such entities must also contend with how, if at all, to effectively respond to those impacts and how to integrate scientific information into that response.

Joshua Tree National Park (JOTR) has become a poster child for the impacts of climate change on US National Parks. Desert bighorn sheep and desert tortoise ranges are shifting as average annual temperatures rise and precipitation becomes sporadic. And media outlets, including NPR and The LA Times, have latched on to the tragic story of a park without its namesake: climate change threatens the park’s population of Joshua trees. As read in front-page news, climate change portends the grand demise of this treasured landscape. But of course, there is so much more to the story.

First, the two foremost models of climate change in the park don’t entirely agree. The first study, which relies on globally-scaled models, predicts “elimination of Joshua trees throughout most of the southern portions of its current range,” which includes the entire range within JOTR. A second study argues that the unique, uneven terrain within the park could create micro-climates that will deviate from regional trends and provide suitable habitat for Joshua trees. The inputs, assumptions, and uncertainties inherent to each study present different management options.[1] Aware of the first set of models, JOTR managers could choose to abandon the Joshua tree as the park’s focal point, accepting that they will disappear from the landscape. Or managers might shift resources and efforts toward re-locating the species (and the park?) northeast, to areas expected to be suitable. The second model suggests that park managers could preserve potential refugia by mitigating other threats like fire and invasive species in those locations for the benefit of potential remnant populations.[2]

Park managers understand that predictions of climate change impacts are uncertain and varied. For example, JOTR protects multiple ecosystems within a relatively small area, so it’s difficult to model the response of the entire protected landscape. Park managers encounter uncertainties and knowledge gaps for management in each of the ecosystems: “How effective are changes in…management policies when you don’t have the data there to support them?”[3] Data are incomplete, but of course when the data are present, they’re often contingent and disputed. JOTR managers must account for competing scientific frameworks, assessments, and as we saw above, models, that muddle applications of science for park management.

Further complicating decision-making, Joshua Tree is one of over 400 sites managed by the National Park Service, each with its own culture of management and facing climate change impacts unique to its resources and purpose. When national policies are released, each park must spend time and resources to reflect on how to interpret and apply them in context. The dynamics of climate science and politics unfold quite differently at each park compared to on the national and even regional levels. Thus, national parks like JOTR are at once institutionally connected and independent, each creating, interpreting, using, and politicking around climate change science and models in their own context, while striving to meet national standards and abide by system-wide policies.

Beyond national parks, other local-scale entities, like cities, question if and how they should  prepare for the uncertain impacts of climate change. These entities draw on collaborations with researchers, in-house expertise, and a variety of tools to incorporate scientific information regarding their risks, vulnerabilities, and abilities to respond. At a recent conference hosted by the American Geophysical Union [4], Laurna Kaatz of Denver’s water utility presented on the challenges of using climate models and predictions and described how the utility has partnered with other utilities and research groups like the National Center for Atmospheric Research to understand potential climate change impacts and adaptation options. At Arizona State University, researchers (like Nich) studying the impacts of heat on communities across central Arizona have partnered with local governments to exchange knowledge and lay the foundations of future collaborations. Even the engagement work that we’ve frequently discussed on Muddling Through aims to link national assessments of climate change impacts to local priorities through deliberative forums.

Climate change is often framed as a national and international concern, a framing that masks the efforts of cities, universities, and parks that are sensitive to local climate change impacts. Such entities must also contend with how, if at all, to effectively respond to those impacts and how to integrate scientific information into that response. How are local entities using science to plan for climate change impacts and why? How do local entities adapt to varied access to resources and expertise regarding climate change science? How do local norms or policies interface with or influence the application of climate change impact assessments? Who is (un)welcomed in those conversations?

We believe that local responses to climate change present an opportunity for learning about decision-making under uncertainty, as well as in the face of each of the challenges detailed above. As such, we’re seeking submissions to a panel at the upcoming Society for Social Studies of Science Conference regarding how local entities (taking a variety of institutional shapes from across the globe) create, use, respond to, and interact with scientific knowledge in the context of climate change impacts. We hope this session will lead to practical insights and an exchange of best practices for decision-making under uncertainty, to better aid cities, parks, and other local entities as they tackle climate change impacts in context.

See the panel abstract (scroll down to #92) for more details. The deadline for submissions is February 1st.

 

Footnotes

[1] We could write a lengthy blog with a dissection of those differences, and thus won’t share them here (but if you insist on knowing now, email Michelle and she’ll bore you with the details).

[2] This, by the way, is the current approach.

[3] Anonymous JOTR employee, quoted from Michelle’s dissertation research interviews

[4] AGU, a large scientific organization primarily oriented towards the physical sciences, is doing a lot of innovative work to connect science to on-the-ground outcomes for communities. Most notably, their Thriving Earth Exchange program attempts to connect communities in need of scientific expertise to scientists who can provide technical assistance while helping both the scientists and community understand what the other can and cannot provide.

More than meets the eye? The value of Rand Paul’s BASIC Research Act

Rand Paul deserves credit for seeking to integrate more perspectives into the research funding process, but his vaguely defined taxpayer advocate would fail to better connect research to desired societal outcomes.

The newsbeat is, well, beating Michelle and I thanks to proposal defenses, holidays, and our own involvement in politics around the latest tax bill. But we’d like to turn back the news cycle to October and look at a proposal from Senator Rand Paul related to research funding decisions at federal agencies. The BASIC Research Act would require a “taxpayer advocate” on research proposal review panels, the groups of scientists who decide what proposals to fund. Senator Paul deserves credit for seeking to integrate more perspectives into the research funding process, but his vaguely defined taxpayer advocate would fail to better connect research to desired societal outcomes.

Congress has a history of skewering federal agencies and scientists for wasting taxpayer money on research that puts shrimp on tiny treadmills, asks questions of single people on speed dates, studies snail sex1, or examines farm-based tourism in rural China2. Senator Paul regularly complains about NSF, often through poorly written Waste Reports that attract the ire of science advocates. His bill invited similar reactions. For example, one reporter accused Senator Paul of politicizing or misunderstanding science because the bill proposes interventions in the peer review process that federal agencies, such as the National Science Foundation (NSF), use to dole out research money. At the hearing where the bill was introduced, Senator Paul justified the bill as a mechanism to weed out “bad science,” to create a more transparent funding system, and to fight the perceived cronyism of scientists approving funding for the work of friends and colleagues. The bill covers a lot of ground, including requirements to make grant applications to federal research agencies public. The most interesting pieces of the bill, however, pertain to proposal review panels, the groups of scientists who decide which requests get funded. If passed, the bill would forbid researchers from suggesting potential reviewers for their applications and would require agencies to add two people to their review panels: 1) an expert from outside the discipline and outside academia; and 2) someone to serve as a taxpayer advocate.

Senator Paul’s bill centers on long-standing debates about which projects should receive taxpayer funding and who should make those funding decisions. Vannevar Bush, the lauded architect of postwar American science policy, and Senator Harley Kilgore of West Virginia, engaged these questions 70 years ago at the inception of the National Science Foundation3. Senator Kilgore argued for increased federal investment in social sciences, more equity in the geography of federal science investment4, and a shift away from military research. Bush parlayed his success managing the wartime U.S. Office of Scientific Research and Development, which led the Manhattan Project, into a different vision for federal investment in research, one that placed scientific excellence as judged by (mostly physical) scientists at the forefront of funding decisions. Today’s NSF largely resembles Bush’s ideal: Scientists make decisions about which proposals merit funding within individual disciplines based on the scientific merit of those proposals (as opposed to other criteria, like regional economic development or potential technological applications). While Senator Paul’s bill is no present day incarnation of Kilgore’s science policy, it nonetheless attempts to bring non-scientists into the fold of science governance with the aim of improving science5; Bush might be appalled.

That bit of historical context in place, let’s dig through Senator Paul’s proposal for a taxpayer advocate on grant review panels. Frankly, I agree with the spirit of this proposal, though not Senator Paul’s word choice nor his analysis around the topic. Precedent for public involvement in federal decision-making exists in many agencies, often taking the shape of required public comment periods6. The myth that only scientists can govern science, argued by Bush, leaves no place for such external input in science funding decisions. But bringing non-scientists to the table could, in theory, better connect federal investment in research to the ‘goods’ scientists and science advocates parade about to justify that investment, such as economic development, new products, new medicines, better health, happier people, and so on. Non-scientists provide broader perspectives, ideas, and critiques of research decisions, forging research robust to conceptual and practical criticism. For example, non-scientists might question troubled research mechanisms like studies of brain health that rely on animal models shown to be disconnected, even irrelevant, from processes in the human brain. And by nature of being external to engrained research pathways, non-scientists could offer constructive insight and critique where institutional inertia has taken hold7.

That said, Senator Paul’s proposal contains many loose ends that cloud assessment of the bill: Who should taxpayer advocates be and how do agencies select them? Are taxpayer advocates on the federal payroll (would love to see Taxpayer Advocate on usajobs.gov) or is this a service expected of people external to the federal government? Are they paid at all (reviewers for NSF, for example, are generally not paid)? What, exactly, does it mean to advocate on behalf of the taxpayer? Is the taxpayer advocate just Rand Paul (we kid…)? Different answers to these questions could lead to very different outcomes. A member of the public could face a steep learning curve to understand different research fields. Yet having someone internal to federal agencies might fail to bring new perspectives to the table or could amount to a rubber stamp on projects8. Further, structural issues with Senator Paul’s proposal pose more questions. Is one person sufficient for establishing taxpayer interests on a review panel? Is having input external to the scientific community most useful at the individual grant level? The bill’s language fails to clarify any of these questions, meaning we might see little change in the role of public value in research funding decisions.

Let’s assume that the taxpayer advocate is someone disconnected from the scientific enterprise: they’re a member of broader society, which ostensibly sounds like a win for both the taxpayer and for research (at least in our view). But the involvement of non-experts brings up more issues.

Non-expert panelists could be too intimidated to question the rest of the panel members due to a lack of topical expertise9. Perhaps the logistics of getting public members to such review panels would mean the ‘usual suspects’ might show up: those already highly engaged in debates around science, like people from science advocacy organizations. Or perhaps other forms of political power would lead to competition for seats on panels, turning them into politically contested arenas; imagine a lobbying firm or an interest group pushing hard to get ‘friendly’ public members on the panels through one mechanism or another. Further, selecting members of the public to sit in as reviewers creates layer upon layer of difficult decisions about recruitment and preparation. Simply put, involving the public in this way is a big challenge, particular for agencies without experience in this type of engagement10.

Given these questions, Senator Paul’s proposal for a taxpayer advocate would do little to clear the air on research funding decisions. Whatever his intentions, Senator Paul’s bill avoids the important, broader question of how to better connect investments in research to public goods and outcomes used to justify research, despite nods to this ideal in his hearing. Perhaps he and other legislators have avoided (and continue to steer clear of) the question because it’s a hard nut to crack and would require substantial institutional changes at federal agencies.

But what can we take away from Senator Paul’s proposal that might be effective? And how could we build on and improve it?

Having diverse input as to which research programs merit federal investment is a laudable goal. Rather than involve the public on individual proposal review panels, non-scientists from the public could be brought into decision-making about federal research earlier in the funding process to set larger goals for a particular research program. At this stage, non-experts can interface with scientists and determine priorities for research investments without digging into the nitty-gritty of a particular method or theory for every proposal considered. Further, public involvement at this level allows for conversations about the ethical and political ramifications of research. Research on public participation in technical decision-making shows that non-expert people from all walks of life can thoughtfully and constructively contribute to technical decisions when given the opportunity. While large-scale involvement might be unduly expensive for any given research program, smaller panels (5-10 people) could spend a few days working with program managers to identify potential issues and societal outcomes and help set research priorities accordingly.

Senator Paul should look elsewhere in the federal government for inspiration; in DARPA, for example, research funding decisions revolve around specific problems and empowered program managers, not peer-review panels (or perhaps he should wait for Michelle’s dissertation to be published regarding the National Park Service). Alternatively, Congress could ask federal agencies to do a bit of experimentation. Invite non-scientists to serve on a few review panels, even if only in an advisory role. Create panels for various research initiatives and ask non-scientists, or better yet diverse members of the public, to help set research agendas and write calls for proposals. In short, play around with different mechanisms appropriate to the agency. Several agencies, including USGS, EPA, NSF, and NASA, support external research, each with distinct institutional shapes and sizes. A little experimentation could go a long way to creating new ways for non-scientists to contribute their expertise and values to funding decisions.

Would taxpayer advocates on review panels better connect research to the outcomes we use to justify funding research? Without more detail, we’re stuck with an ambiguous “maybe”. Perhaps Michelle and I should apply for NSF funding to test citizen review panels for NSF research programs (no really, that would be super cool to study). Or maybe that proposal would end up in Senator Paul’s next Waste Report.

– Nich Weller

Footnotes

  1. Michelle was part of this research team as an undergraduate.
  2. The National Science Foundation is used to congressional criticism and does its best to encourage researchers to communicate public values. When I received a small NSF grant for the project examining Chinese farm-based tourism linked above (a relic of a past dissertation avenue), NSF program officers encouraged me to clearly articulate the public value of my proposal in the summary for NSF’s website, ostensibly to keep congressional threats at bay. In our opinion, this does little to promote public value in research but that’s a topic for another day.
  3. See Gregg Pascal Zachary’s Endless Frontier: Vannevar Bush, Engineer of the American Century for more on Kilgore and Bush’s shared ideas and disputes.
  4. Notably, West Virginia lacked, and still lacks, major sites of federal science investment like national labs.
  5. Senator Paul sees external input, which Bush would have detested as interference in scientific excellence, as a path to ‘good’ science, or at least a path away from ‘bad’ science (in the words of the Senator’s testimony).
  6. Federal public comment periods, however, have glaring problems as 1) existing interest groups dominate the process and 2) they require substantial time, expertise, and resources to participate in.
  7. Are broader perspectives a panacea for what ails science? Of course not. Believing so is just as detrimental as believing that non-scientists and the public have nothing relevant or wise to say about research.
  8. An assessment of NSF’s policy requiring ‘broader impact statements’ on proposals might help answer these questions.
  9. This is a very real phenomena I must account for in my research.
  10. Agencies could try to do this independently, without a legislative mandate, and “see what works,” which might also provide them political clout if congress passes legislation on the topic.

Revisiting the Value of Public Forums for Building Consensus and Changing Perspectives

Over the summer, Nich wrote about the ‘power of conversation,’ reflecting on a public forum in Boston that brought together public participants and facilitators and staff from the Museum of Science to discuss the challenges of sea level rise and extreme precipitation in the Boston area. The Boston forum is part of a national program, funded by the National Oceanic and Atmospheric Administration (NOAA). And this past September Nich invited me to facilitate at the second forum hosted by the Arizona Science Center, with about 70 members of the public from the Phoenix metro area. Working from educational and discussion-prompting materials developed by Nich and his colleagues, participants shared their values, concerns, and questions, as well as proposed resilience strategies, all related to drought and heat. Each group was provided with real data and visuals to understand the impacts of their choices and to evaluate, and reevaluate, their resilience strategies.

In some ways, my experience reflected Nich’s experience in Boston. For instance, I similarly felt uplifted by the tone of discussion, or as one forum organizer put it, the fact that participants could “disagree without being disagreeable.” I frequently heard phrases such as, “I could see why you’d think that,” and questions like, “can you please explain why you feel that way?”

The participants at my table came from various jobs, age-groups, and locations across the valley, and likewise, they came with a range of motives. The most universal motive revolved around ownership of place, Phoenix: “Because this is my hometown.” This strong connection to and knowledge of place was evident in the afternoon exercise on resilience to extreme heat in Heattown.1 Participants moved quickly through sorting hypothetical stakeholder perspectives without much discord, though not due to lack of care. I heard multiple iterations of, “We live in Phoenix so we know heat and how we feel about it. We get the ‘heat jargon’.”

Although participants shared a connection to Heattown (in their minds, Phoenix) and asked about and acknowledged the values and concerns of others with respect and genuine interest, there were still contested choices and contradicting perspectives on the best path forward for resilience. I noticed that part of the reason we were able to reach ‘consensus’ was because participants were told that they didn’t have to agree with the majority at their table; their values and choices would be reflected in their personal worksheets if not on the consensus game-board. This led me to question the value of the forum for consensus building; I witnessed thoughtful consideration of shared perspectives, but I didn’t notice much in the way of changing minds.

First, I wondered, how is the consensus building format of the forum any better than voting or majority rules? According to the U.S. Office of Personnel Management, consensus means “finding a proposal acceptable enough that all team members can support it, with no member opposing it.” But that doesn’t mean a unanimous vote or majority rule, because voting doesn’t require opposing parties to listen effectively to each other as they discuss their differences.

This is why voting often proceeds more quickly than consensus building (e.g., a show of hands vs. an eight hour forum). For instance, at another table, forum participants resorted to voting whenever they felt their discussion was running behind the other tables or that their discussion was stagnant. But despite its efficiency, voting has it’s downsides. Primarily, it creates losers (to the majority winners), and if the losing minority feels misunderstood and unheard, distrust and disgust with the process can fester. On the other hand, consensus building takes time because participants have to know and trust each other, and in the case of the forum, learn together, to work constructively and listen effectively. The participants at my table openly commented on the arduous process of reaching consensus and believed that it demonstrated the robustness of their consensus. They felt that although some parties disagreed with the consensus, all parties were heard. But they couldn’t quite articulate why that made the consensus acceptable.

There were still majority and minority views, and there were certainly perspectives that were unaccounted for in the final resilience plans (even after a round of revision). Nich actually recorded data regarding this phenomenon that illustrates what I witnessed; we can celebrate that people were able to disagree without being disagreeable, but at my table they didn’t often change their minds about what resilience plan they thought best. One in five changed their mind for drought and three in five chose the final plan that matched the table’s consensus. For heat, no one changed their mind, and again, three in five agreed with the table consensus. But across all tables, the data captured some compromise (with the caveat that we had a lot of missing data points): 43% of people changed their minds in drought and 20% did not (37% were unknown, missing sheet, didn’t ‘follow the rules,’ etc).  And for heat, 45% changed their minds, while 23% did not (32% unknown).

Where we saw changed minds, were they only willing to compromise because the stakes were relatively low? That is perhaps something we cannot answer unless we performed the forum under different conditions (real-life scenarios, no personal worksheets to turn in, etc.) Most of all, however, I wondered about the times when we did not see changed minds: what’s the value of a mind unchanged but understood? To get at that I had to let go of numbers and return to what I saw.

First, under the conditions of the forum, there was a noticeable change to the tone and language of discussion. When I encounter people discussing climate resilience out in the world, it rarely involves respectful discourse and active listening. At the forum, forced and facilitated exposure to divergent perspectives pops participants’ perspective bubbles, so to speak. This was evident in the noticeable effort by participants to frame their opinions in terms of the moral values of participants who disagreed with them, or at the very least to notice that differences of opinion were, beyond being “right” or “wrong,” rooted in different values. This is a remarkable and important accomplishment. In a joint study from the University of Toronto and Stanford, researchers found that people often struggle with this feat in political discourse; it’s difficult to set aside personal reasoning and step into the perspectives and values of opponents. “Moral reframing,” as they call it, is a true exercise is perspective taking, and participants had their fair share of practice with it throughout the forum.

Further, because moral reframing helps you understand your opposition in terms of what they value, it thus alerts you to what they are giving up when they compromise. And if you understand the stakes of their concession to build consensus, then perhaps in the future you might be willing to concede something in return (i.e., an exchange in political capital). In future forums it would be interesting to have note takers observe the language and patterns of consensus building. Did compromises on drought policy earn a participant leverage when negotiating heat resilience?

So maybe this post just ends up with me realizing that, to borrow from Churchill, public forums are the worst form of consensus building, except for all the others.2 But that doesn’t diminish the value of critical reflection, and for that I leave you with a quote from Sheila Jasanoff:

“Science and democracy at their best are modest enterprises because both are mistrustful of their own authority. Each gains by making its doubts explicit. This does not mean the search for closure in either science or politics must be dismissed as unattainable. It does mean that we must ask and insist on good answers to questions about the procedures and practices that undergird both kinds of authority claims.”

We spill a lot of ink on this blog encouraging healthy (and scientific!) questioning of the ability of Science alone to address issues in the public sphere like climate resilience. We should be willing to explore the limits of our democratic solutions as well.

-Michelle

1Heattown was based on real data from anonymized Louisville, KY.
2And Nich pointed out that we can also question whether seeking consensus is the best role for forums. Perhaps consensus is only instrumental– the real goal is to get people to talk and listen thoughtfully to other perspectives.

Image: Participants, planners, facilitators, and Arizona Science Museum staff enjoying some coffee-break sunshine during the eight-hour forum.

Evidence-Based Policy-Maker since 1998

From Santa to Climate Change

(Warning parents, this post contains Santa spoilers!)

As a student and researcher of evolution during my undergraduate thesis and now climate change in my dissertation, I am no stranger to debates over the proper use of science in policy-making as well as over the validity of science itself. But recently, as I was reflecting on my experience with the challenges of evidence-based policy-making, I realized that to get to the start we have to journey back to 1998, the height of my suspicions about Santa Claus.

No way did Santa fit that new bicycle down the chimney. And elves don’t make American Girl Dolls. But I couldn’t just say I didn’t believe anymore. I had to know. And to be certain, I had to do some serious research. This was before high-speed internet; it took an hour to log on to our dial-up system, and should someone call during my Google search, all would be lost. Plus, let’s be real. My mom wasn’t going to let seven-year-old Michelle surf the web without supervision.

So instead, I undertook some top-secret archival work in my mother’s basement office, and as is often not the case with archival research, it didn’t take long before I found what I was looking for. To this day, my mother meticulously prints, marks, and files receipts for everything she purchases. On November 29, 1998, Karen Sullivan spent $115.00 on “Kit, the American Girl Doll, with book and accessories.”

Truth hurts. And always one for drama, I ran upstairs to the dining room and tearfully screamed, “All adults are liars!!!!!!!” at my mother right in the middle of her lunch-date with friends.

A crusader for honesty, I shared my discovery with my friends on Monday at school. My mother paid for this dearly when my friends’ parents called to complain that I had unapologetically RUINED their daughters’ childhoods. Though most of my friends appreciated me (even if begrudgingly) for disseminating my research, one friend stopped me in my tracks.

“Maybe Santa just gave your mom a receipt in case you didn’t like the doll. That’s smart,”she said.

The evidence was…inconclusive?! How could she disagree with my findings? How could she criticize my policy plan to stop the generational cycle of deceit?

Early Lessons from “Santa Policy”

Now, maybe you’re thinking I just needed more information to convince her. I could have set up a video camera to capture her parents setting up her toys. (But maybe Santa wasn’t feeling well so he shipped the gifts to her parents?) Or what if I could have flown over the North pole to show her it’s just a barren ice-cap? (They probably live underground?). If she wanted to believe in Santa Claus, she was probably going to find a way to believe.

At the time, I was shocked. Fast forward about two decades, *almost* three degrees, a few eye-opening mentors, and some real world practice with this evidence-based policy stuff, and I can now say I encounter this all the time in my research of climate change politics and policy. Researchers often have noble intentions for sharing their work with decision makers, but this doesn’t always translate into the policy actions they propose. That some scientists even propose specific policy actions creates uncertainty around their science for those who disagree with their proposals. And decision makers may use evidence to justify their policy actions, only to find critique from colleagues gets louder.

Disagreements over issues like climate change are often argued in terms of lacking or contested knowledge, as well as conflicting notions of risk. But increasing the quality and amount of evidence doesn’t seem to dissolve dissension as much as you might expect. Why is that? What are the limits of scientific knowledge for addressing today’s pressing policy issues?

The Excess of Objectivity

First, particularly with climate change, there are myriad perspectives from which you could conduct your research due to various and overlapping natural and human-causes of climate change and an even wider array of potential environmental, economic, and socio-cultural impacts. Each perspective comes with its own body of knowledge, values, and action-items which may contradict those of another. And in the vast space of climate change research, those holding different views are sure to find some academic in some university who holds a hypothesis or theory that fits their perspective.1

My co-blogger, Nich has a helpful analogy for this: We have a dozen cupcakes, all of different flavors and decoration fitting one of twelve people’s preferences. Each person can choose the cupcake that best suits them based on taste and appearance. Even outside of that dozen, you’re bound to find a cupcake that suits you, considering the vast number of bakeries, recipes, and ingredients. Now, substitute preferences of flavor or appearance for “values/aims” and cupcakes for “evidence/science.” Because of the various perspectives that characterize the extensive amounts of climate change science and evidence out there, you can find contradicting facts to support contradicting value- or aim-based positions on climate change, and a whole host of other issues.

Scientific Uncertainty, Caught in the Middle

Scientific uncertainty often lies at the center of debates over climate policy. One side will prescribe a policy based on a scientific claim, while dissenters will invoke scientific uncertainty to rally against action.

Some scientists argue that the public and their elected officials simply don’t understand uncertainty.  But I have to say, I can hardly blame my grandmother2 for misunderstanding “scientific uncertainty.” First, it has different meanings in different fields, largely owing to the mathematical differences among studying electrons, atoms, cells, humans bodies, and human societies. Second, uncertainty is an abused concept in debates over climate change policy (among other controversial science policy arenas, e.g., GMOs, vaccines, etc.).

Case in point: earlier this year, New York Times columnist Bret Stephens used his first column to challenge climate change scientists and activists, noting that there are many unknowns and uncertainties when it comes to climate change, enough that proposed ‘abrupt and expensive changes in public policy’ should be delayed and conversation (i.e., debate) should continue. Former Times columnist and blogger, Andrew Revkin swiftly replied to the column, in which he was oft quoted. In his reply, he argues that the basics are clear; climate change is happening. Unclear are the scope and scale of impacts including answers to questions that extend far beyond the bounds of climate science, ‘how dangerous?’ and ‘what do we do?’ But, Revkin argues, such uncertainty is still actionable knowledge.

Uncertainty has been caught in the middle of this debate, with one side declaring that it is reason to forestall action and the other countering that it’s the reason to act urgently. Yet, Revkin and Stephens would likely both agree that climate change is a (mostly) political problem involving really difficult values questions that are consistently couched in terms of (un)certainty by advocates and opponents of action. With such debate, centered on competing interpretations of, misunderstanding of, or misuse of scientific uncertainty, it’s fair to see why my grandmother is sometimes skeptical of the facts.

Uncomfortable Knowledge

Some people cannot accept evidence for climate change because it is inconsistent with their social-cultural identity. To explain this phenomenon, Yale Law Professor, Dan Kahan suggests that there may be two ways people use reason, (1) to know what is known (e.g., the latest climate science), and (2) to be who we are. Sometimes who we are doesn’t align with what is known. Understandably, most people choose to protect their social-cultural identities; it’s what they have to live with everyday. Put another way, whether or not someone “believes” in the evidence for climate change may be less an expression of what they know and more an expression of who they are. Climate change is wrapped up in a host of cultural and socio-economic problems, so it isn’t surprising that many individuals and institutions find evidence for climate change to be “uncomfortable knowledge.”

For example, my conservative, Republican uncle, who works in steel, refuses to accept climate change is real. But given that proposals to address climate change threaten his work and his ideology, it makes sense that he would have a hard time accepting evidence for climate change. And it’s worth noting that he approves of my climate change work with the National Park Service. Perhaps this is because national parks are ideologically neutral (their 75% ‘favorable’ approval rating is only second to the US Postal Service among federal agencies3) compared to debates over energy, infrastructure, and lifestyle.

The Role of Science

So if evidence is so contested in political negotiations, what’s the use?

Well one idea, in the words of Philip Handler, President of the National Academy of Science from 1969-1981, is that “The estimation of risk is a scientific question… The acceptability of a given level of risk, however, is a political question, to be determined in the political arena.”4 In other words, the role of science is to understand how different policy choices can lead to different outcomes (or, in Handler’s example, different levels of risk). The role of politics, then, is to choose which outcomes (levels of risk) and thus which policy choices are acceptable. But Handler’s point is not entirely sound because even our tools of research and estimation can be politically subjective: the way scientists and policy analysts pose research questions can bias research programs toward certain conclusions and policy suggestions. In a recent National Affairs article, conservative pundit Oren Cass argues that this is one of the ways in which he believes evidence-based policy falls short. He uses the example of policy analyses around health care access to make his point:5

“The debate over how best to ensure that low-income Americans have access to health care in the most cost-effective way possible is one of the most controversial and complex policy quandaries in our politics. Yet the researchers providing the evidence on which to base policy were investigating whether the value of Medicaid is larger than zero…Proponents of Medicaid expansion understandably delighted in this framing, which established a bar of “not worthless” for the program.”

Cass argues that the research and results are biased because the experiment was designed without regard for alternative ways to spend Medicaid money, or some might say, with a liberal mindset. Cass then purports that the government philosophy should come before the research design: “…assessment should begin from a philosophical inquiry into the proper role of the state and its relationship to the development of healthy families and communities….”

Such an inquiry could lead to different measurements, different experimental designs, and the use of different research tools. If this is true, our ‘objective research’ can be politically biased from the outset because the questions we choose to ask, the frames we ask them in, and the tools and experiments we use to answer them can all be ideologically influenced. Cass even suggests that we should abandon the premise that policy-related science is objective: “…let’s couch that science in its political perspective upfront.”

It’s important to note that transparent alignment of a research program with a political perspective doesn’t mean the research is “false” or “wrong.” But it could limit contributions to bipartisan policy-making. In the case of the Medicaid research bearing the brunt of Cass’ criticism, the utility of the results was constrained because the research design neglected a host of other ways in which we might improve access to health care for low income Americans.

The Upshot

Before you lose your mind down a postmodern wormhole wondering about the (non)existence of “truth” or “objectivity,” let’s get back to what’s important here: Santa isn’t real. My friend could spend her whole life believing, but that doesn’t change reality. But of course, telling her this didn’t change her mind at the time.6 In retrospect (and this is what’s really important here), I was learning an important science policy lesson at the ripe-old-age of seven: Two people can look at the same facts and reach two different, even opposite, conclusions, and not because the facts aren’t true, but because the world and its problems are complicated and our ability to “know” is limited.

Awareness of this “Santa Policy” lesson, and all of the others above, is necessary when creating, acquiring, using, and sharing information. Plus, it invites us to question what we know and why we know it. After all, blind faith in the value of evidence isn’t scientific.

I’m personally still muddling through. First, how do I know when to stop questioning (i.e., how do I avoid that postmodern wormhole)? Perhaps, it has to do with improving the transparency of perspectives that contribute to research. I personally believe climate change is real because smart people who work on climate change and who demonstrate understanding of both sides of the political argument (Stephens and Revkin, for example) agree climate change is real, but still disagree about what to do about it. But that confidence in expertise is just confidence, founded or unfounded, in certain people’s opinions anyway, which can seem an insufficient justification for policy action.

My own research examines the role of science in decision making for the National Park Service, often concerning climate change. And through that work, I’ve started to understand why more or better information rarely solves disagreements over climate change. But how can such disagreements be solved? And how can we effectively use evidence to inform policy? I’m learning everyday. Stay tuned for another post, another time.

-Michelle

P.S. My Santa Policy has evolved, and I promise not to break the news to your small children. Also worth noting that my sister’s policy was to pretend she still believed because then you guarantee a consistent gift-flow…So of course there is more than one policy to craft based on the evidence!

Footnotes:

1ASU science policy professor and practitioner, Dan Sarewitz, calls this the “excess of objectivity.” He claims that it’s not for a lack of knowledge that we can’t all agree; rather it’s the excess of knowledge.

2My grandmother is my litmus strip for thoughts from the average American. I love you, Grandma!

3The latest data on this is from 2015, but the parks are only increasing in popularity year-to-year so I think it’s safe to assume this number is probably steady.

4Quoted in Risk and Culture, Douglas and Wildavsky 1982, 65

5And then there are other places where I disagree with his analysis, but that is outside the scope of this post.

6And similarly, recent work suggests that constantly barraging climate deniers with the “97.1% consensus” is a failing strategy.

A Tired Critique of a Tired Pro-Science Op-Ed

Reflecting on the Use of Science as a Political Tool

Although it may feel we are always on the science-policy news beat, fieldwork, summer jobs, new puppies, and Game of Thrones watch parties sometimes put us on a news delay. Lucky for us, we have friends (and guest bloggers) like Christian Ross who keep us on our toes. Last week, Christian brought an editorial to our attention; in the latest issue of Science, Delaware’s junior Senator, Chris Coons, sounds off on why “scientists can’t be silent.” Christian asked us, “does this strike you as just political posturing?,” and questioned some of Coons’ claims about the public benefits of the historic and current public support of science in the United States.

The rapid-fire emails that ensued led us to a reflection on science’s increasing use as a political tool. If our critique sounds hackneyed that’s because Coons’ piece is just another standard “pro-science” op-ed.1 Maybe we are cynical, but Senator Coons’ vague declaration of the importance of science for decision making makes him another politician playing the “science card” for political points. And the card game Coons and other “pro-science” politicians play increasingly draws a line between the “right” and “wrong” sides of science on issues like climate change, vaccines, and GMOs. Dangerously, these sides map onto political agendas, with democrats like Coons on the side of science and republicans painted as the opposition.

We all agreed the piece tasted strongly of political posturing. Coons includes lots of democratic talking points from immigration to climate change – and a sincere call for action. And he offers himself shameless self-congratulations for co-founding the Senate Chemistry Caucus, a bipartisan effort to “promote the use of sound science in policy-making.”2

We also questioned the misleading lack of context for Coons’ claims.3 For example, he laments the 17% cuts to research funding in Trump’s Skinny Budget but ignores the rest of the budget proposal and accompanying questions: What else is cut? Is scientific progress really being “threatened”?  And as Nich notes in a previous blog post: “Too many responses to Trump’s budget blueprint and its impacts on federally funded science rely on dubious connections between research and public value to justify funding.” To us, Coons’ outcry fails to address the real problems facing our country’s relationship with science.

Coons’ call for scientists to more widely publicize their work and reach out to elected officials is a noble one, but lacks nuance and detail. We know it’s imperative that scientists share and publicize their work, but it’s important to consider how. Are they openly advocating for a particular policy or agenda item? Are they presenting a set of policy options based on evidence? Are scientists setting their recommendations in the context of political realities? (For more on this, see the work of Roger Pielke, Jr.). Recent work on the failure of scientific consensus messaging in climate change policy points to the importance of these questions. Further, how will Coons’ recommendations for scientists help him accomplish his vague goal to elevate the role of science and fight ‘anti-science’ sentiments?

Finally, Coons fails to note that at the root of debates surrounding climate change, GMOs, and vaccines, are questions fundamental to political ideology: how much control should the government have over our personal choices (lifestyle, nutrition, health)?; how much government regulation should businesses tolerate?; etc. Science has become implicated as the justification for different answers to those questions. It’s concerning to us that being ‘pro-science ‘is becoming synonymous with a being a democrat. In op-eds like Coons’, science represents a political talking point to garner votes and exchange barbs with opponents instead of an effective tool for evidence-based governance. In doing so, ‘Science’ borrows at high interest against its future status and trust in the public sphere.

-Michelle, Nich, and Christian

1You can find very similar messaging in rhetoric surrounding the March for Science. Also, see Coons’ speech to the AAAS from earlier this year.

2We will give Coons credit for “his” effort being “bipartisan,” but he’s still posturing. The caucus actually started in the House, sponsored by the largest scientific society in the world, The American Chemical Society, to spotlight the role of the chemistry enterprise in the U.S. economy. The caucus is not the subject of our critique, but rather Coons’ seeming touting of it for political gain; our (admittedly short) web search did not uncover any other senators putting out press releases on the matter.

3Though we acknowledge you don’t typically have the luxury of a generous word count when writing editorials.