Evaluation and documentation are essential to learning from, reporting on, improving, making the case for, and sustaining the work of artists in municipal settings. Evaluation is a systematic process to learn how effectively the work was carried out, and what the impacts are on partners and participants. It’s important to include evaluation in the project plan and scope of work, even if you only have limited resources, skills, and capacity for it.
“Understanding what other stakeholders might see as a success (or not) is...critical. It is only through these multiple prisms that the work conducted at the intersection of the arts and other fields will ever be understood in its fullness. By extension, it is only through such efforts that the role of the artist in relation to community will be more clearly appreciated.”
Maria Rosario Jackson, Urban Planner
Consider these basic questions to clarify the purpose and focus of the evaluation:
Who is the evaluation for? Is it municipal leaders, funders, artists, community partners and participants, the public, or a combination?
What does the work look like? What’s the nature of the creative work and the partnership?
What makes the work work? What factors contributed to and/or challenged the effectiveness of the partnership?
What are the outcomes? And who should be informed of the outcomes?
How can we improve program design and partnership practices?
Why is this project/partnership important? Talk through what a strong and effective partnership would look like, and what its desired effects on partners could be. Consider:
The extent to which partners share common goals;
The nature and quality of interactions between partners;
How power is shared;
How effectively partners managed conflict;
How collaborative decision-making happens;
What resources each partner contributes, and how they are shared;
How the partnership helps or hinders the work;
How each partner benefits from the partnership.
Documenting Your Partnership
Documentation creates a record of a project through media such as video or photography, meeting notes, interviews, correspondence, journals, press coverage, and other quantitative and qualitative data collection. It is crucial for evaluation and reporting at the end of the project. Effective documentation:
Anticipates needs and opportunities in order to capture key information as it presents itself and important events as they happen.
Is purposefully planned, for example to inform evaluation, support communications, tell the story, create a record, include different perspectives, and inform future activity.
Prioritizes what will be useful and manageable. Be mindful that documenting is more than capturing a moment, but also transcribing, editing, and organizing material to avoid “Now what do we do with 100 hours of video footage?”
Conventional evaluation has often been driven by funders and institutional leaders, and implemented by professional evaluators or outside experts. M/A partnerships are centered on the public realm, so it makes sense to be inclusive in planning and implementing evaluation so that it benefits communities as well as institutions. Some questions to ask include:
Who gets a say in defining what outcomes matter and what constitutes meaningful evidence of change?
Who is implementing the evaluation, including interpreting the data?
Does the evaluation team represent project constituents and context?
Identify who among the project team, stakeholders and participants has knowledge, expertise, and interest in what can be learned from evaluation, and involve them in planning. You can also:
Engage municipal evaluation resources. Some municipalities have research and evaluation experts on staff, or an entire evaluation division. Individual staff directly working in the partnership may have skills or training to bring to the table as well.
Tap creative resources. Artists instinctively evaluate— they ask questions and make observations as part of their creative process. They may bring evaluation experience from previous civic projects. Artists’ input is valuable in how to assess the ways that the arts aspects of the project supported civic interests. They can also provide expertise on how to make evaluation processes and tools more engaging.
Example
Toronto-based Jumblies Theater takes an arts-based approach to program evaluation by participants.
Involve evaluation professionals. An outside evaluator can provide expertise in a particular evaluation approach; cultural competency in working with specific populations; credibility; and setting up ongoing evaluation protocols. The partnership might benefit from working with an outside evaluator when:
The program or project is a pilot. Evaluation will inform the next stage.
The program or project is high stakes, such as a city-wide initiative.
You want to build internal capacity through coaching, skill building, and creating protocols.
A funder provides resources to support outside evaluation.
The municipality intends to seek funding based on the partnership work.
Include community partners, participants, and other stakeholders. A participatory evaluation honors the perspectives, voices, and decisions of all stakeholders, no matter where they are in the overall power structure. Participants can consult on or actively participate in evaluation processes including: defining the evaluation focus and procedures, data collection methods, interpreting evaluation data, and approving next steps. Often this process requires addressing different points of view.
Challenges and Strategies
Challenge
Partners, intermediaries, and other stakeholders may have differing ideas about what is important to evaluate.
Strategies
Make time to discuss expectations for evaluation, even if the demands of the work are pressing. Some ways to proceed:
An evaluator can bring structure and guidance to create focus and define priorities.
Identify key stakeholders who should have a voice in defining outcomes and what’s important to “measure.” Evaluation can provide different information to different stakeholders, so discuss which outcomes they most and least value, and integrate their perspectives into your evaluation methodology.
Define and share what success would look like from the perspectives of the artist, municipality, and third-party partner.
Negotiate evaluation priorities with stakeholders—what is most important to assess and learn?
Challenge
Outcomes seem hard to define or measure.
Strategies
Outcomes such as community resilience and social cohesion can seem too intangible or long-term in nature to measure. Partners can feel challenged to “prove” that the partnership project “causes” a particular outcome or that baseline data even exists against which to gauge change. Almost any project that aspires to contribute to community, civic, or social change should be able to measure change at some level. To define desired outcomes:
Allow for outcomes to emerge. Municipal agencies may be required to articulate outcomes up front, even before an artist is sought or secured. It is important to provide latitude, and to learn from the artist’s research phase what outcomes stakeholders see as meaningful.
Distinguish between a larger vision or long-term goals and the incremental changes that most projects realistically can achieve. Recalibrate your goals to consider more immediate outcomes, such as increased awareness of an issue, as preconditions for larger changes to occur in the future.
Get specific. Civic outcomes are often vaguely defined with broad terms like “transformation,” “community building,” or “equity,” that are not measurable. Articulating specific outcomes and indicators is key. The Continuum of Impact is a useful tool that helps define civic and social outcomes and indicators, and helps you collect evidence of the impact of arts and culture.
Tap the municipality’s expertise when you need to know what evaluation methods are used in particular fields (such as public health or transportation). The municipality can also lead you to other experts, such as an urban planning department in a local university.
Challenge
The prospect of evaluation feels daunting and/or beyond partners’ capacity or resources.
Strategies
Make sure partners are clear about the purpose and focus of an evaluation. Sometimes the problem is not about evaluation but rather project conception and planning.
Seek professional expertise to help define the scope and cost of evaluation. Think about this while developing budgets and writing grants. Build evaluation in as a line item.
Right-size the evaluation to the scale of the project or resources. Some evaluation plans lay out more variables than can possibly be measured, even if time and money were no object. Or they frame outcomes that can’t be realistically achieved in the project timeframe. Start small— define a modest and focused evaluation approach that is practicable and affordable.
Example
Although Art At Work’s initial phase involved three city departments, artist Marty Pottenger and evaluator Chris Dwyer narrowed the evaluation focus to just the Police Department, and two outcomes: improving morale within the department, and public perception in the community. With input from city leaders including the Police Chief, partners determined this was a high priority and would be a useful demonstration project. The artist and the evaluator then worked together to define a manageable work plan. Learn more in the Evaluation in Action section.
Reluctance to report “failure” or results that are less than hoped for.
Strategies
Measure the project or initiative in relationship to its starting point. Efforts to engage a community over issues of racial difference, for example, are more likely to bear fruit in a community that has already begun such a discussion than in one where it has been suppressed. The former might be considered a success and the latter a failure unless the distance travelled is considered.
Take a long view. Outcomes may not be evident until a period of time after project completion. Ongoing programs can collect data over time on the long-term and/or ripple effects of M/A projects. This counteracts a limited view of the success or failure of individual events.
Municipalities may need to embrace the idea of “failing forward.” Municipalities have little latitude for “failure,” while artists are continually experimenting, prototyping, and learning from less successful outcomes. There is always uncertainty and risk in trying something new. “Failing forward” is about leveraging and learning from mistakes, and acknowledging the complexity of the work and what it takes to achieve impact over time.
See this Evaluation in Action Profile for more on learning from challenges and taking adaptive actions.
Difficulty in knowing how to assess artistic excellence in civic contexts.
Strategies
Art and creative processes are not typically evaluated in municipal contexts. Creativity and aesthetics can be perceived as less important than civic or social outcomes. Because art embodies subjective qualities like emotional impact and creation of meaning, it can seem impossible to quantify. However, when the creative work is excellent it has greater potential to advance less tangible municipal goals such as “building community” or “enhancing civic pride.” So:
Provide opportunities for municipal staff to see and experience what art does for participants. They can then advocate for the value of the art from their own perspective.
Pilot and evaluate art strategies first before launching them in a larger public context.
Expand the criteria for “excellence” to include the unique value and capacities that artists bring to the table. See the Aesthetic Perspectives framework for help in defining “attributes of excellence.”
NYC’s Administration for Children’s Services was a municipal partner in NYC’s PAIR program. Staff had an “aha moment” after they attended a screening of young participants’ videos and heard them talk about their work. It helped them understand how participants’ self-expression led to self esteem. This shifted the partnership relationship, as staff became less enforcers of agency protocol to focus more on supporting new ways of meaningfully engaging youth.
