During the “Art and Science of Place-Based Evaluation” convening, there was substantial discussion about the role evaluators can play in shining a light on race, equity, and power in a community. Most conversations at the convening focused on highly developmental evaluation approaches, where the evaluator becomes part of the change strategy – testing different lines of inquiry, bringing different themes and data forward for community discussion, engaging the community in primary data collection to build local capacity, and playing an important role as an active contributor to community change efforts.
While a local evaluator who is embedded in the change strategy can be a huge asset, the conversation sparked me to consider the role that national or portfolio-level evaluators – or those who are charged with a more summative evaluation – can play in creating the space for open, honest conversations about race, equity, and power. Without a doubt, this approach requires the evaluator to understand and navigate the history of the community, balance formal and informal power dynamics, and identify perspectives that have traditionally dominated – including those that have been intentionally and unintentionally overlooked – with the intent of bringing these to the fore.
I’d like to share two examples from our recent work: an evaluation of the Aspen Forum for Community Solutions’ Opportunity Youth Forum (OYF) and the Lumina Foundation’s Community Partnership for Attainment (CPA) assessment. OYF is a 21-community partnership whose change strategy is to improve the structures and systems supporting “opportunity youth,” defined as 16-24 year olds who are not connected to school or meaningful employment. CPA is a 75-city investment intended to drive postsecondary attainment by increasing cross-sector collaboration within communities. Both initiatives elevate the principles of diversity, equity, and inclusion (DEI) as central to the investment.
So how does a national evaluation partner working with close to 100 communities create space for honest conversations about race, equity, and power? Here are a few suggestions:
- Engage “end users” (e.g., youth in the OYF investment) and various stakeholders in the development of data collection instruments. We ask these youth to help us understand what “system change” would look like to them. Similarly, we ask different community partners, as well as site leads, how they would know that systems are in fact changing to better support opportunity youth in the community. The result is a list of indicators that reflects very different perspectives on how one would know that “systems change” is happening to improve the lives of opportunity youth.
- Develop ways to share aggregate- and community-specific data in ways that help partners see where they fall against other communities with regard to the measures they helped define. And then help the communities digest and make sense of the information both across communities, and among partners and stakeholders within the communities. During a recent Aspen OYF convening, we discussed our emerging findings with community stakeholders from different sites, and we will share our individual site findings in one-on-one discussions with each community.
- Use targeted interview processes to help community members define what “DEI” means in the context of their work and their community. During recent interviews for our Lumina project, and for a forthcoming Issue Brief on how DEI manifests itself in a community-completion agenda, our questions required partners to specify how they interpret and apply a DEI lens in their everyday work. Many interviewees concluded the conversation by thanking us for pushing their individual thinking (and over time, their collective thinking) about diversity, equity, and inclusion – a topic that at times does not receive the deliberate focus it requires.
- Lead conversations with community members to test assumptions whenever the chance arises – for instance, at national convenings or during periodic data collection sessions. Never miss an opportunity to have the community member provide analysis and perspective on emerging evaluation findings. For the Aspen OYF initiative, we have posited hypotheses about why we think we are seeing certain trends to community members, site leaders, and funders. In many cases, perspectives were aligned about what might be driving the changes that we are seeing.
- Continuously integrate concepts of community history; power dynamics; and diversity, equity, and inclusion in evaluation frameworks, data collection tools and processes, and reporting tools. Making these concepts consistently visible over time also makes them more discussable. In both the Aspen and Lumina work, our evaluation frameworks call out the DEI principles explicitly, and all of our data collection instruments have corresponding measures.
- Allow community partners to adopt and adapt your data collection tools. We often share our survey and analysis instruments with communities, along with the findings. We have found that community partners have had productive conversations about how they interpreted (often differently) various survey questions and terms. This has the double-benefit of strengthening our instruments, and enabling community stakeholders to develop more aligned perspectives, shared definitions, and a shared language. A handful of Aspen OYF sites have informed us that they are using our survey items as standing discussion points with community stakeholders to help document how different partners are enacting and interpreting community change processes.
I offer these suggestions as fodder for deeper conversation about the role evaluators – whether from within or outside the community – can play in facilitating open and honest conversations about race, equity, and power. I look forward to continuing the discussion when we meet again at the next place-based convening, at the American Evaluation Association conference, or wherever our paths may cross.