Exploring New Horizons: Building Evaluation Capacity From the Ground Up
Leaders of an environmental education group recognized the need to document their participants’ outcomes, after a long reliance on anecdotal evidence. There were many challenges: None of the staff possessed evaluation experience, some resisted the notion of “testing” participants, the program’s remote site had poor access to technology, and it would be impractical to evaluate all the program’s participants. Assistance was sought to develop the group’s first internal program evaluation system. The project involved designing and piloting a pre-/post-questionnaire along with feasible sampling, data collection and analytical methods.
“This project was a good first step for us. We learned a lot about the organizational capacity needed for evaluation. It helped staff better understand what evaluation is and why we need to do it. The preliminary survey findings will help reduce staff resistance to data collection.”—Tracey Weiss, Executive Director, Exploring New Horizons
WuYee Children’s Services: Rationalizing In-house Program Evaluation
Leaders of this agency’s family resource center sought assistance to develop more effective program evaluation tools and methods. With improved internal monitoring systems, leaders felt that staff and other internal stakeholders—as well as funders and other external stakeholders—would gain greater understanding of the program’s impact on its clients. Not only would better data help the program attract and maintain financial resources, it would save valuable staff resources by streamlining and/or eliminating some of the existing evaluation methods.
“I really appreciate all the dissecting work of our evaluation forms and processes. It’s been eye-opening to dive in and untangle things [and develop] something much more manageable. Thanks for helping us with this hard work and leaving behind an improved program.”—Lena Yu, Joy Lok Manager, WuYee Children’s Services
NIAD Art Center: Creating Internal Program Measurement Systems
This 30+-year-old center that supports artists with developmental disabilities, needed help articulating to outside stakeholders the effects of the center’s studio art program on the artists who participate. I was hired to work with the art teachers to identify ways the artists changed and grew. We developed a logic model and then a pilot observation tool the teachers would use to assess each artist’s art-making interests, abilities and social skills. We also developed a pilot interview protocol for a staffer to engage each artist in a conversation about their experiences at NIAD and how they felt they had benefited.
One year after data collection began, staff analyzed the data and determined that the studio art program appeared successful in moving the needle in participants’ commitment to art-making and their development of a distinctive artistic style–attributes tied to their ability to sell their work and generate income. The data also showed which artists benefitted the most from NIAD; this insight led to a significant program change intended to ensure deep impact.
Said Executive Director Dyer, “This topic [of program evaluation] is especially hard in arts organizations as they have focused on activities rather than the impact of programs. Add in the disability portion and it becomes even harder. Eleanor walked the staff through the process [of developing appropriate and meaningful participant outcomes] and brought consensus to this necessary step so everyone understood the importance in attracting funding as well as improving the program using solid data.”