New Ways to Bolster Research and Its Impact

Evidence and Evaluation is making some changes this year. Where shall we start?

Let’s start with my sincere excitement about the direction of our team’s work. First, we have a tremendous opportunity to build upon the work that Arnold Ventures has undertaken over the past ten years as a funder of first resort for rigorous research. It’s remarkable: our team has supported about 120 randomized controlled trials (RCTs) across 12 policy areas since its inception. These studies represent a major investment in figuring out what can be effective to improve lives for kids, families, and communities. And now we are ready for the next step in thinking about how to leverage that research and achieve its full potential for impact.

Second, I am excited about how we are thinking about supporting new research this year. We are launching two new RFPs – one focused on RCTs and the other on quasi-experimental designs (QEDs). I really look forward to seeing the kinds of research projects that come through the door.

Tell us more about this shift in your thinking.

Historically, our team has only supported RCTs – and for good reason. They are often the best way to know if something is actually impactful. But we also know that some questions are difficult or impossible to answer through randomization. Going forward, our new RFP opens the door to other approaches like natural experiments, regression discontinuities, and difference-in-differences. That is a pivot, and we are hopeful we can answer important questions and build evidence in new places.

How does this form of research connect with policy?

Great question. We really aren’t interested in supporting research for research’s sake. We want to ask and answer questions that are top-of-mind for decision-makers. Our new RFPs place significant emphasis on making sure those questions are highly policy-relevant and aligned with AV strategic priorities. 

One key difference between RCTs and QEDs is that, due to design, the latter can focus on areas where there are pressing policy questions. QEDs can look back at data that has already been collected and offer insights. In that sense, there can be a faster turnaround to provide analyses, and retrospective data that provides an evidence-based view into how we might look at the policy question going forward. When feasible, those insights could lead to a test of the policy or program through randomization in the future.

How do you envision potential partnerships with portfolios at AV?

We’re really looking to increase opportunities for collaboration and partnership with other AV teams. In tandem with our teams in Infrastructure, Higher Education, Contraceptive Choice and Access, and Public Finance, we plan to solicit research that aligns with their top-of-mind questions. We want to look for ways to align and support their research agendas. 

Can we go back in time for a minute? If you remember your late 1980s cartoons, this is a real Voltron or ThunderCats kind of moment. We are envisioning a union of strengths coming together to advance their work in rigorous ways. And this gets to the second big part of how the Evidence and Evaluation team is transforming this year.

Say more.

The other change is a stronger focus on how to build from the evidence base that we have already developed – how to get the evidence into the hands of policymakers who shape our society. We are invested in supporting the development of a stronger ecosystem surrounding effective programs and policies so they can be more actionable and more likely to help policymakers decide how we spend scarce dollars and resources to improve societal outcomes.

Connecting the evidence base with policymakers to affect policy formation is a big challenge. At a 30,000 foot view, what are you seeing?

Some questions really rise to the top of our minds. How do we test the kinds of supports that translate learnings from one context to the other? What infrastructure is required to better support and incentivize the use of data and evidence? And how do we take what is learned and think about how to develop a more thoughtful agenda to scale evidence-based programs in the context of how public systems fund projects and operate?

Do you envision this connection happening at the federal, state, or local levels?

We believe this work has a role to play at all levels of government. Through investments and partnerships, we will be trying to develop proof points at the state and local levels where we have seen policy levers and opportunities. We’re also developing supports that create proof points where decisionmakers have been able to take up and use research or build in greater testing and learning in policymaking. We are also hopeful that we can support our partners’ work on evidence-based policy at the federal level. 

What do grantees and potential grantees need to know about these shifts?

That this team remains committed to being a funder of first resort for rigorous research – this is not a retreat, it is an expansion. Partners should know that we’re very open to ideas of how to better connect research with decisionmakers, and that we are committed to the process of testing and learning in all the work we do. We’d like to find ways to bring rigor to the implementation and scaling of effective programs and policies as much as possible. So, if there are ways to test approaches to how we think about supporting policymakers, that would be an exciting next step for us.

Do you see any threads between E&E’s shifts and what’s happening in the United States today?

If you think about the two things we are trying to do – support strong research, and then connect that research to decisionmakers to drive better use of our scarce resources – those two efforts relate to the strong note of distrust and dissatisfaction in our country. There are two surveys that tell an interesting story and point to the need for evidence-based policy. One Pew survey recently showed that fewer than two in 10 Americans trust the government to do the right thing most of the time – the lowest in seven decades. Not good. And yet, in a separate Pew survey, most Americans believe government should have a large role in improving outcomes for different groups and across most areas. When you view these findings together, I think you see a desire for government not to vacate its responsibilities, but to work more effectively. The way we can make a stronger case for that role to be effective is by figuring out what social programs work and expand what works at greater scale. We have to do a better job communicating why these studies matter and how they change lives.

Recently, AV co-founder and co-chair Laura Arnold spoke with Professor Esther Duflo, the Abdul Latif Jameel Professor of Poverty Alleviation and Development Economics at the Massachusetts Institute of Technology and Nobel laureate, about her approach to research stemming from a learning mindset. Could you speak more about that?

That was a terrific interview. For me, the phrase that comes to mind is epistemic humility. That means that we shouldn’t be overly sure about what we think we know at any time, and we are in the process of trying to build knowledge about the incredibly complicated world around us. The problems we want to address are complex and don’t have simple solutions. 

And yet, I think it is important to pair that humility with an investment in knowledge building and faith in the scientific process. By testing and learning, we can try to find things that ​‘work’ and ways to better improve the lives of kids and families. In my mind, that is what Arnold Ventures represents, what makes AV unique and makes it exciting to work here. It’s a long game, it’s incredibly complicated, contradictory, and that’s all part of the process. That’s how we grow as a society — we try to strive toward greater equity and prosperity and opportunity. That’s what it takes.