Evaluating for Casemaking

Profiles

Evaluating for Casemaking

Art At Work, Portland, Maine

The Project

Art At Work (AAW) was initiated by theater artist Marty Pottenger in 2007, who was embedded in city government in Portland, ME for nine years. Projects begin by working with key stakeholders to identify critical municipal challenges, develop an evaluation plan with specific goals and outcomes, and from there Pottenger or an artist she invites determine the art project that centers around engagement. AAW has put creativity to work delivering measurable outcomes to improve police department morale, deepen cross-cultural understanding among municipal workers, and raise public awareness and appreciation for the role of government. For more on AAW, see this Profile.

Profile

Art At Work

Portland, ME

Artists used printmaking, photography and poetry to engage employees in Health and Human Services and the Police Departments

Theory of Change

AAW leverages the creative intelligence and talents of the city’s own workforce as well as the communities it serves. Through art-making, AAW advances public understanding of what government workers contribute to society, fosters reflection that improves municipal operations, and strengthens awareness of municipal workers’ significance to the community.

Partners

Funding for evaluation

Animating Democracy (a program of Americans for the Arts) and its Arts & Civic Engagement Impact Initiative’s Evaluation Field Lab, which was in turn funded by the W.K. Kellogg Foundation.

Report

This Evaluation in Action profile draws from AAW’s Evaluation Plan, Kate Preston Keeny and Pam Korza’s chapter in Arts and Community Change: Exploring Cultural Development Policies, Practices, and Dilemmas, and other Animating Democracy sources.

Evaluation Focus

Evaluation of the early phase of Art At Work focused on making the case of the value of art as a strategy to improve municipal government. If Art At Work was to continue beyond the City’s initial three-year commitment and with a greater investment of public dollars, Pottenger would need to design programs that address department goals, and measure what matters to City leadership. Civic outcomes would have to be supported with evidence convincing to key stakeholders—municipal leadership, department heads, staff, and the public. The primary audience for the evaluation was municipal leaders.

Methods

The City did not have an internal evaluation mechanism for this program. Pottenger was able to worke with researcher and evaluator Christine Dwyer of RMC Research as part of the Evaluation Field Lab. By design, Dwyer’s role was to coach and mentor Pottenger, defining and co-designing ways that Pottenger and an intern could implement evaluation activities themselves.

“Having an in-depth evaluation design process and such a respected evaluator allowed partnering municipal government agencies and employees to see the project as a more serious, credible and useful activity.” Marty Pottenger

The evaluation process included these approaches:

  • Start by defining outcomes and evidence that will later, if achieved, substantiate the case to municipal leaders. Pottenger and Dwyer used an evaluation framework Dwyer had previously developed to frame outcomes and indicators with casemaking in mind. The framework guided the one-on-one conversations between Pottenger and Police Department leaders and stakeholders to get them to articulate the highest priority outcomes for the project; and to determine what indicators would be convincing to them.
  • See this worksheet to help you define indicators that matter to stakeholders and help you make the case for your project.
  • Narrow the focus and priorities. AAW was essentially a one-person operation. Given this limited capacity, Pottenger and Dwyer decided to focus the evaluation on the Police Poetry Project, rather than trying to evaluate three projects happening simultaneously in different departments. The evaluation focused on these two departmental priorities:
    • improved public perception of Portland police; and
    • improved morale within the Police Department.
  • Set achievable documentation strategies to collect evidence. Pottenger used low- or no-cost strategies and created simple systems for documenting and collecting qualitative evidence of change. For example:
    • She made written note of any impacts that were mentioned in meetings, conversations, and emails, and accumulated a body of qualitative data.
    • She facilitated "evaluation go-rounds" at the end of each activity and formal or informal meetings with municipal staff. She made sure that a note-taker was present for key meetings. Doing this constant evaluation increased staff investment in AAW’s success.
    • Planning photo and video documentation to best tell the story of AAW and collect evidence of impact. This, included strategically choosing activities and events to document; and preparing photographers/videographers in advance to get the best results (i.e. supplying a shot list.)

    • Other data collection methods included: questionnaires and follow-up interviews with Police Department participants, and brief questionnaires for attendees at public events.

Impact and Lessons Learned

  • Linking evaluation planning to program planning helped Pottenger to clarify the intent of the project and thereby focus program activities.
  • Defining outcomes that matter to the City kept Pottenger accountable to AAW’s overarching goal to improve municipal government.
  • The evaluation kept Pottenger focused on her theory of change — and that the process of making art increases participants’ ability to function as a team, understand other viewpoints, open lines of communication between city departments and the community; and envision positive outcomes.
  • Pottenger sharpened her own evaluative thinking and communications. She increasingly found it a natural part of her process to integrate evaluation findings in discussions with heads of participating departments, funders and participants, and to effectively frame the case for the program’s continuation.

Evaluating for Adaptive Action Toward Systems Change