The Role of Outcome Measurement, Part I

Earlier this month we outlined a side-by-side comparison of outcome measurement and program evaluation. In the next two blog articles, we unpack the outcome measurement process specifically.

Outcome measurement: Steps One Through Five

1. Assess your organization’s readiness.

Outcome measurement is the determination and evaluation of a program’s results, and their comparison with the intended or projected results. This definition requires a starting point.

2. Select the program and create a plan.

After understanding the state of your organization, decide which program you will measure. Choose a program that is undergoing a certain change or requires transformation. Outline a specific plan for outcome measurement, including a timeline with milestones by which it will be completed.

3. Describe the program’s mission and activities using a logic model.

Visually depict the program you choose with a logic model, such as this:

Copyright Isaac Castillo, 2014.

Complete the logic model with your outcome measurement team during a series of working sessions.

4. Identify the program’s intended results, or outcomes.

Program outcomes comprise the second half of the logic model. These goals are the “meat” of outcome measurement, and the focus of subsequent steps of the process.

5. Identify indicators of success for each outcome.

Bring your program’s outcomes to life by assigning each goal a metric, which ensures completion can be successfully measured.

Outcome measurement: The Story of Sue Ito

To illustrate this first half of the outcome measurement process, we are introducing Sue Ito, Director of Planning and Performance at the Jefferson County Library (JCL). Sue has been tasked by the County Board of Directors to report a set of agency outcomes on an annual basis.

To begin, Sue uses a “Start With the End in Mind” planning product, which shows that JCL’s Young Adult Education Program (YAEP) is ripe with opportunity, and much of the organization is eager to support its growth. Sue decides to conduct outcome measurement for YAEP and forms a cross-department team to assist her.

The team meets weekly and for the first month focuses its working sessions on creating a logic model to better visualize and understand the current program. Below is a snapshot of the YAEP inputs, outputs, and outcomes.

Inputs What we invest	Outputs What we produce	Short-term Outcomes Short-term impact	Mid-term Outcomes Mid-term impact	Long-term Outcomes Long-term impact
Volunteer staff   Paid staff   Computers   Career development materials   Library space	Career education and resources for disadvantaged youth   Technology training for program participants   Job skills training for program participants   Life skills coaching   Safe space for disadvantaged youth to meet weekly	New learning, such as:   IT knowledge gained   Job skills learned   Positive attitudes toward higher education developed	New actions, such as:   Participants apply to post-high school colleges or apprenticeship programs   YAEP participation grows   JCL receives increased community funding	New conditions, such as:   Rate of college-educated youth in community increases   Young adult crime rate decreases   Community reputation is elevated

 

Sue reviews the logic model with her team and realizes the outcomes are strong, but vague. How will the new learnings, actions, and conditions be proven? The team creates specific metrics, or indicators of success for each, captured below.

Short-term Outcomes Short-term impact	Indicators	Mid-term Outcomes Mid-term impact	Indicators	Long-term Outcomes Long-term impact	Indicators
IT knowledge gained	60 percent of participants show evidence of increased IT knowledge on pre- and post- program survey	Increase in college or apprenticeship program participation	80 percent of participants apply to higher education programs	Rate of college-educated youth in community increases	Number of college-educated youth in community increases by 20 percent
Job skills learned	100 percent of participants report acquisition of new job skills	Increase in YAEP participation	Program enrollment rate grows by 30 percent in the first year	Young adult crime rate decreases	Young adult crime rate decreases by 5 percent
Positive attitudes toward higher education developed	75 percent of participants cite intentions to pursue higher education opportunities	Increase in community funding for JCL	JCL receives a $25,000 grant from the local government to support growing YAEP	Community reputation elevated	Average real estate prices in Jefferson County grow by $10,000 year-over-year

What are your thoughts about the above outcomes and indicators? Which are easy wins, and which require a great deal of work to achieve? Please share your feedback in the comment section below.

In our next blog article, we’ll wrap up the outcome measurement process with our story of Sue and YAEP. Stay tuned!