We're still hard at work!

Welcome to our new site! We are currently previewing and testing. Please help us make the site better by reporting any profile problems using our online form or sending an email to jchin@artsusa.org.

Stories & Examples

On-the-ground examples of evaluation in action!

These case studies, evaluation reports, and profiles offer insights that may help you:

  • define civic/social outcomes and indicators
  • hone the purpose of or focus your evaluation
  • make choices regarding approach and methods
  • find creative ways to report findings.

Check out:
In-Depth Case Studies from Animating Democracy's Arts and Civic Engagement Impact Initiative.

Add your own:
Evaluation case studies, reports or stories. Share your work by contacting animatingdemocracy@artsusa.org


Chris Dwyer

This item relates to the Preliminary Menu of Possible Outcomes/Indicators/Measures from the Starksboro Art and Soul Project, as both are part of the Art and Soul Project. This item is a ten page community survey with thirty questions. The survey is an adaptation of a standard survey developed by the Orton Family Foundation to use in tracking changes in perceptions about land use planning in all their Heart and Soul community projects. The Preliminary Menu is a table relating outcomes, indicators, and data collection methods for the Art and Soul Project.) 

The the final portion of the survey relates specifically to the Art and Soul Project: it asks how much people know about the project, what their vision and goals are, and how involved they may wish to be. Visually pleasing but rather lengthy, this is a nice sample instrument. Complete information about the project in which the survey was used is found at: http://www.orton.org/projects/starksboro. The first questions are basic biographical/demographic items. The next four questions relate to land use perceptions and participants' willingness to get involved in the issue. The next three questions ask participants to check off boxes relating to their how much information they take in about community planning and how they communicate their concerns. Three questions are about participants' perceptions of how well the community and its leadership are doing with the issues at hand.

practical tool
4
Dudley Cocke, Linda Frye Burnham, Erica Kohl, Craig McGarvey

"Connecting Californians" reports on a research project completed in 2000, that explored story as a powerful means of building community. The project conducted a search in each of California's 58 counties to find projects that engaged residents in a public performance or story about local history and life. Maps were created to represent the various projects. It is a helpful model for collaborative planning and discussion regarding the arts, culture, and civil society. Although it does not offer frameworks or tools, it is a useful report sample and resource for arts and civic engagement practitioners.

Over the course of ten months, the research team interviewed more than 100 practitioners, educators and policy makers with experience in the arts, humanities and civic culture. They interviewed and convened repesentatives of projects at the intersection of the arts, the humanities, grassroots narrative and community organizing to explore the role that a cultural organizer plays in promoting and sustaining collaborations, facilitating relationships and encouraging local leaders to emerge. Participants included artists, humanists, organizers and residents and representatives of their civic and religious organizations. This report presents the findings of that inquiry and questions for the future. 

A section on program design discusses three core values — engagement, inclusion and inquiry. The report includes six comprehensive appendices. Appendix A: Three Local Projects presents brief case studies of public performances springing from the history, aesthetics and issues in three California communities. Appendix B: Evidence of Public Performance Based on Local Life offers a scan of the field.  Appendix C gives information on two focus groups. Artists, organizers, educators and funders that came together to discuss the role of story in building community. Appendix D reports on national and California interviews. Appendix E includes three research essays. Writer and critic Linda Frye Burnham interviewed practitioners and thinkers in the field grappling with issues at the intersection of community organizing, art and the humanities, producing three monographs. Appendix F is a literature review [Adapted from the Community Arts Network website.]

case study
0
Mark J. Stern, Susan Seifert

Grounded in a recent strategic plan, the Tucson Pima Arts Council is moving to advance civic engagement in the city and county through its programming, funding, and partnerships. As part of Animating Democracy’s Art & Civic Engagement Impact Initiative, and in addition to the qualitative focus reflected in the evaluation inquiry with Maribel Alvarez, TPAC wanted to know what concrete measures are reasonable to use to understand the civic engagement effects of its work as an agency. The objective of the collaborative inquiry with Mark Stern and Susan Seifert of the Social Impact of the Arts Project was to develop a plan for systematically collecting and analyzing data on civic engagement.

The report begins with a review of the policy context in which the collaboration was undertaken, including the findings of the TPAC strategic plan. This is followed by a discussion of the proposed strategy and specific recommendations for documenting civic engagement. Specifically, Stern and Seifert propose five strategies: improving organizational data gathering, telling stories, documenting artists and the informal cultural sector, identifying institutional networks, and using geographic information systems to integrate data for analysis. These recommendations provide a plan for documenting civic engagement and the arts. However, given the fiscal and social realities, the third section of the report outlines an implementation plan that would allow for staging these elements depending on the resources available.

book / article
0
Chris Dwyer

Art & Soul is a project of the Orton Family Foundation. The Orton Family Foundation, in partnership with the Town of Starksboro and the Vermont Land Trust hypothesize that, by getting in touch with deeper community values and connections to place, citizens will be able to improve upon traditional approaches to planning and make better decisions about the future of their communities. With the Art & Soul Civic Engagement Project they are testing whether the use of different forms of art will catalyze articulation of the unique assets of a community, in turn impacting community decision-making.  The project is being piloted in Starksboro, VT, with artist-in-residence, Matthew Perry. It is applying arts-based methods of community engagement to generate and inform conversations around values and community. Through storytelling, painting, dance, music, theater, media arts, photography, sculpture, poetry and prose and/or video, Starksboro is bringing together disparate groups of citizens to address the positive and negative aspects of the community. Townspeople will identify ways to reflect the articulated values and aspirations sparked by the story gathering and art making projects in concrete actions, policies and choices shaping the future of Starksboro and its land use. Stakeholders hold a range of expectations for the Starksboro project from fairly straightforward concrete accomplishments, e.g., an art product is produced, to community transformation, e.g. younger residents and newcomers begin to assume leadership positions. The evaluation plan attempts to capture that full range of desired expectations organized in three phases: process (short term); outcome (during the life of the project); and impact (long term, post project results).  

For each of the three phases, the evaluation plan lists the major questions to be answered in an evaluation related to desired outcomes; the indicators of desired behavioral or attitudinal change; potential data collection instruments and strategies that are appropriate for the indicators; notes about the target sample for a particular data collection strategy; timing of data collection; and, where relevant, any appropriate comparisons that might be made. The indicators can be used in a number of ways:  a beginning point for developing instrumentation; the framework for content analysis of documents and records; a frame for other types of documentation, e.g., the film documentary; and a file structure for maintaining anecdotal information about the project.

case study
0
Chris Dwyer, Marty Pottenger

The Art at Work is a national initiative to improve municipal government through strategic art projects between artists, city departments, unions, elected officials and the community.  Launched in 2007 in Portland, ME, as a three-year project, the initiative includes artmaking workshops led by artist Marty Pottenger with local artists (currently a printmaker, poets, and photographers) within the city’s Public Works, Health & Human Services, and Police Departments. Art At Work's working hypothesis is that it is useful for people to make art about their work and lives, and that doing so increases their chances to come up with better solutions to longstanding problems. One component, The Police Poetry Project, paired poets and photographers with members of the Portland Police Department to write poetry and take photographs and to address two key challenges--the relationship between police and the public and low department morale. The resulting work was assembled into a calendar that was distributed within the community and used as a focus for departmental and community dialogue.    

As part of Animating Democracy’s Art & Civic Engagement Impact Initiative, artist and initiative director, Marty Pottenger collaborated with evaluator Chris Dwyer of RMC Research. They applied an evaluation framework developed by Dwyer to systematically define outcomes and indicators that provide evidence of concern to targeted stakeholders and opinion leaders, as well as strategies for data collection and communication of results.  For the police project in particular, and the overall initiative, the evaluation plan lists the major questions to be answered in an evaluation related to desired outcomes; the indicators of desired behavioral or attitudinal change; potential data collection instruments and strategies that are appropriate for the indicators; notes about the target sample for a particular data collection strategy; timing of data collection; and, where relevant, any appropriate comparisons that might be made.  

Defined indicators could be used in a number of ways: a beginning point for developing instrumentation; the framework for content analysis of documents and records; a framing or focusing device for other types of documentation, e.g., the film documentary; and a file structure for maintaining anecdotal information about the project.  Finally, the framework’s application over time for particular initiative components and on the whole, can help substantiate the case for the role of the arts in civic systems and processes.  

For samples of evaluation instruments used to assess impact of the Police Poetry Project, visit the page on Art At Work Survey Instruments.

case study
0
Maria Rosario Jackson, John Malpede

Los Angeles Poverty Department (LAPD) is a Skid Row-based theater organization, founded and directed by artist John Malpede. LAPD has distinguished itself by its longstanding commitment to making change in L.A.’s Skid Row community, particularly regarding the homeless, through theater-based civic engagement work. Many have observed LAPD’s apparent potent effects on individuals and on social relations in Skid Row, and acknowledge its contributions to influencing structures, systems, and even policy.  

As part of Animating Democracy’s Arts & Civic Engagement Impact Initiative, LAPD and Urban Institute senior researcher, Maria Rosario Jackson engaged in field research that provides a foundation to recurrently identify, monitor, and assess the presence, density and richness of the cultural infrastructure of the Skid Row neighborhood. They undertook this research as a first step towards recommending data collection practices and tools that can help to bring into relief cultural assets in the Skid Row community, residents’ cultural needs and aspirations and the infrastructure or set of resources required to make cultural participation possible. Undergirding their collaborative inquiry are the ideas that cultural participation builds and strengthens social networks that potentially lead to increased social capital and collective efficacy; and that cultural participation is essential in helping to create a healthy environment and normative neighborhood.

Malpede and Jackson believe that with their proposed inventory of cultural vitality and recurrent information about various aspects of it, Skid Row community leaders can more effectively carry out the work of improving quality of life in the neighborhood. The framework would enable Skid Row cultural as well as community agencies that use arts and culture to see their work as part of a larger system and to create an asset-focused narrative for Skid Row that can help shift or expand the ways outsiders perceive the Skid Row community. This may enable people outside of the Skid Row neighborhood to better understand its cultural dimensions and help shift or expand the ways outsiders perceive the Skid Row community. This may, in turn, catalyze and inform new investments in Skid Row.  

book / article
0
Chris Dwyer

This tool serves as a model to align values, actions, and measures of progress for State Art Agencies. In table form, it lays out a generic base for locating concepts of participation within a framework of concepts of public value and motivating values of different groups. The table can serve as a basis for developing the types of outcomes and measures related to State Art Agencies' actions to broaden, deepen and diversify creators, stewards and spectators/participants. Dwyer adapted this tool as a worksheet for use by a broad range of organizations.  See Worksheet to Define Indicators with Casemaking in Mind.

practical tool
0
Suzanne Callahan

Artist Rha Goddess’s Hip Hop Mental Health Project (HHMHP) seeks to contribute to shifting the cultural paradigm of shame and alienation surrounding mental illness, and satisfy a need for a SAFE place to confront the issue and obtain vital information. Through the integration of performance and dialogue, the HHMHP works to impact public discourse and values among urban communities in a way that educates about the signs, symptoms, and spiraling course of mental illness, and to explore possible solutions to the contributing life stressors of societal stigma and the difficulties of an overtaxed mental health system. A key creative component of the project is LOW, Goddess’s one-woman performance that unflinchingly depicts the very human reality of mental illness in our culture. It fuses monologue, movement, and music to tell the story of Lowquesha, a vibrant young woman, and her all-too-common journey through the mental health system.

The Hip Hop Mental Health Project is committed to engaging young urban and low-income communities of color as they are the most detrimentally affected by the disparities in mental health diagnosis, treatment, and care. Dialogues that immediately follow performances aim to create a safe space for audiences to confront issues around mental health that might not happen in another context. Through performances such as LOW and its related dialogue, HHMHP seeks to:

  • educate about the signs and symptoms of mental illness and tools for recovery;
  • decrease the social stigma of mental illness, especially for those of lower incomes and of color;
  • explore possible solutions to some of the life stressors that influence mental health;
  • increase awareness of, and access to, mental health services and support; and
  • impact public discourse about mental health.

The Hip Hop Mental Health Project is an initiative of 1+1+1=ONE, a Brooklyn based nonprofit organization that utilizes the methodology of Arts Based Civic Transformation to empower individuals and communities to affect positive social change.

As part of Animating Democracy’s Arts & Civic Engagement Impact Initiative, the collaborative inquiry between Rha Goddess and evaluator Suzanne Callahan of Callahan Consulting for the Arts focused on the impact of LOW, and post-performance dialogue on audiences’ attitudes, beliefs and perceptions about mental health and illness. The assessment of impact is based on: 1) a formal evaluation conducted by City University of New York (CUNY) researchers using an IRB-approved study (Institutional Review Board); and 2) an alternative approach to audience evaluation developed in collaboration with Callahan and tested with one audience. The two studies complemented each other and allowed comparison of two research processes that asked similar questions in different ways.

For information about the CUNY study, visit the Hip Hop Mental Health Project web site.

book / article
0
Sue Wood

When Flint Youth Theatre began planning for a new play addressing the local and national problem of school violence, it had no idea that, in the process of developing the project, its own community would experience a devastating elementary school shooting. A year after the tragedy, the play ...My Soul to Take, written by artistic director and playwright William Ward, became a focal point for fresh attention on this persistent and painful issue. The play, stylistically atypical of most youth theater in its nonlinear, collage style and its treatment of the subject, captured a swirl of opinions surrounding the shooting. The play’s central metaphor, the Pied Piper, and the question “Can’t somebody do something?” (implored by children throughout the play), became a call to reinvigorate community dialogue and move toward action on this pressing issue.

Over several months, …My Soul to Take served as the backdrop for a diverse set of dialogue opportunities organized by Flint Youth Theatre and collaborating organizations concerned with education, neighborhood crime prevention, and community issues. These dialogue opportunities aimed to coalesce fragmented efforts to address school violence. More than 100 community members met in small study circle groups over several weeks to consider causes and effects of school violence, and options for action. Young people explored dimensions of the issue through participation in process drama workshops, facilitated by artist Gillian Eaton, and through curriculum-based efforts related to their experience of the play. Dialogue was also integral to the process of creating the play; the script was inspired by the words and views of actual participants in the process drama workshops. With local partners and educators, dialogue was planned and facilitated in conjunction with the play.

…My Soul to Take contributed to the following outcomes.  Participants in Flint Youth Theatre’s study circles dialogues came to a new understanding of the causes and effects of youth violence and defined actions they could take individually and collectively to stem school violence. Many of these actions were implemented. The project also coalesced the previously fragmented efforts of community organizations that worked in partnership with the theater.

Outcomes:  awareness, understanding, behavior, skill/practice, artistic
For more on this topic, visit the Social Impact Outcomes and Indicators section.

Evaluation sought to understand:

  • if and how arts-based dialogue might refocus public attention on the issue of youth violence
  • if greater understanding of causes and effects of youth violence was achieved
  • what actions were stimulated to prevent youth violence from happening
  • characteristics of an effective partnership approach to the issue as a way to improve previously fragmented individual efforts
  • the effectiveness of FYT’s imagistic aesthetic style in stimulating meaningful issue-based dialogue

Approach to evaluation:

FYT worked closely from start to finish with an evaluation coach made available through Animating Democracy. Evaluation activities were largely self-implemented.

A logic model was created for planning and evaluating the project. It became a constant and very useful reference point for shaping the project, helping FYT to focus and limit the project’s many possible programmatic directions. It also helped FYT to consider critical context (or conditions) such as the fragmented efforts by agencies in the community and statewide gun legislation and to identify civic objectives it was best positioned to effect, i.e. to pursue coalescing local agencies

Data collection methods and tools from FYT:

 

Other resources:

Theatre as Civic Dialogue, by Joan Lazarus
This article appreared in 2001 in TYA Today, a publication of Theatre for Young Audiences/USA. Lazarus is Associate Professor of Theatre at the University of Texas at Austin where she heads the Drama and Theatre for Youth Programs. It was intended as a critical analysis by an outside theater educator primarily to understand the effectiveness of FYT’s aesthetic style in stimulating meaningful issue-based dialogue, but also to describe and document the project to contribute to discourse in the field of theater for young people about these institutions’ civic role and potential.

 

book / article
0
Virginia Lacayo, Arvind Singhal

Edutainment - the mixture of education and entertainment ranging from video games to soap operas, harnesses the power of entertainment to promote social change. This publication focuses on unique communications-centered approaches to social change and introduces basic concepts, strategies, and theories supporting these strategies, describing challenges and lessons learned, especially related to strategic planning and evaluation.  It raises questions about how to measure the complexity of social change processes, as well as outcomes such as movement building, enabling environments, or empowerment as still needing to be addressed.

The authors examine the limitations of traditional cause-effect based evaluation that frames results as quantifiable and measurable for the purpose of evaluating long-range social change work. The authors suggest complexity science as an alternate assessment tool.  Complexity science approaches social change as a complex, unpredictable, uncontrollable and often contradictory long-term process. If evaluation must be used, the authors suggest that evaluation design, methods and analysis be made transparent, flexible, responsive, and incorporate diverse perspectives.

Edutainment uses the media to tell stories that encourage behavioral and societal changes. Roskilde University (Denmark) Professor Thomas Tufte has developed three “generations” of edutainment: 1) Use information to change behaviors and raise awareness 2) Emphasize society as the unit of change and incorporate public participation strategies 3) Understand that a lack of information is not the cause of social problems. Edutainment cannot overcome structural inequalities and histories of discrimination.

 “Pop Culture with a Purpose” explores the potential of edutainment using examples from three organizations. Soul Buddyz, a project of the Soul City Institute for Health and Development Communications (South Africa), uses a television show, radio program, and accompanying materials to promote children’s activism, gender equality, and respect for oneself and others in South Africa. Puntos de Encuentro (“Common Ground” or “Meeting Places”) in Nicaragua promotes feminism and women’s rights through a soap opera, talk show, and magazine.  Breakthrough promotes women’s issues in India and immigration issues in the United States living with publicity campaigns, music videos, and video games.

Awareness, knowledge, understanding
0