|
So What is Program Evaluation?
First, we'll consider "what is a program?" Typically, organizations
work from their mission to identify several overall goals which must be reached to accomplish their mission. In nonprofits,
each of these goals often becomes a program. Nonprofit programs are organized methods to provide certain related services
to constituents, e.g., clients, customers, patients, etc. Programs must be evaluated to decide if the programs are indeed
useful to constituents. In a for-profit, a program is often a one-time effort to produce a new product or line of products.
So, still, what is program evaluation? Program evaluation is carefully
collecting information about a program or some aspect of a program in order to make necessary decisions about the program.
Program evaluation can include any or a variety of at least 35 different types of evaluation, such as for needs assessments,
accreditation, cost/benefit analysis, effectiveness, efficiency, formative, summative, goal-based, process, outcomes, etc.
The type of evaluation you undertake to improve your programs depends on what you want to learn about the program. Don't worry
about what type of evaluation you need or are doing -- worry about what you need to know to make the program decisions you
need to make, and worry about how you can accurately collect and understand that information.
Where Program Evaluation is Helpful
Program evaluation can:
1. Understand, verify or increase the impact of products or services
on customers or clients - These "outcomes" evaluations are increasingly required by nonprofit funders as verification that
the nonprofits are indeed helping their constituents. Too often, service providers (for-profit or nonprofit) rely on their
own instincts and passions to conclude what their customers or clients really need and whether the products or services are
providing what is needed. Over time, these organizations find themselves in a lot of guessing about what would be a good product
or service, and trial and error about how new products or services could be delivered.
2. Improve delivery mechanisms to be more efficient and less costly
- Over time, product or service delivery ends up to be an inefficient collection of activities that are less efficient and
more costly than need be. Evaluations can identify program strengths and weaknesses to improve the program.
3. Verify that you're doing what you think you're doing - Typically,
plans about how to deliver services, end up changing substantially as those plans are put into place. Evaluations can verify
if the program is really running as originally planned.
4. Facilitate management's really thinking about what their program
is all about, including its goals, how it meets it goals and how it will know if it has met its goals or not.
5. Produce data or verify results that can be used for public relations
and promoting services in the community.
6. Produce valid comparisons between programs to decide which should
be retained, e.g., in the face of pending budget cuts.
7. Fully examine and describe effective programs for duplication elsewhere.
How to Write Comprehensive Reports
A comprehensive evaluation report should include:
• An executive summary of findings
• A description of the program that was evaluated (including
funders, objective, and other key information)
• If applicable, information about the overall organization and
how the evaluated program fits into that organization’s mission
• The purpose of the evaluation, the evaluation process, the
methods used to carry out the process (tools, protocol, type of data-collection systems)
• Information (along with resumés or other biographical information)
on the individuals who were responsible for carrying out the work (including the lead person and other members of the evaluation
team)
• Information about the target area and statistical information
about the target participants, and the connection between those general statistics and the purpose of your evaluation
• A report and discussion of findings: What did the evaluation
reveal? How will the findings help facilitate program improvement? How can the information be used for program sustainability?
How to Communicate Results to Stakeholders
How do programs talk about or disseminate the results of their evaluation? The answer to this question
connects back to accountability and the purpose of evaluation efforts. If a program conducts an evaluation for program improvement,
results will most likely be communicated to administrators, staff, parents, and participants in various ways, many of them
informal. However, if a program is conducting a formal evaluation for funders, it must consider more formal ways to communicate
results, such as comprehensive reports or formal presentations.
Organizations have disseminated the results of their evaluations through:
• Presentation of results at staff meetings for management and staff within the organization
• Presentations at luncheons or seminars for external parties who are stakeholders (such as collaborative
partners)
• Presentations at regional meetings or national conferences in the fields of education, youth
development, family strengthening, or public policy
• Comprehensive reports for partners, community businesses, or funders who are looking for concrete
documentation of program impacts
• Executive summaries or full reports posted to program or organization websites
Sharing results does not have to mean sharing every bit of information that was collected. Deciding
which information is the most important to which audience is the key to communicating evaluation findings.
Pitfalls to Avoid
1. Don't balk at evaluation because it seems far too "scientific."
It's not. Usually the first 20% of effort will generate the first 80% of the plan, and this is far better than nothing.
2. There is no "perfect" evaluation design. Don't worry about
the plan being perfect. It's far more important to do something, than to wait until every last detail has been tested.
3. Work hard to include some interviews in your evaluation methods.
Questionnaires don't capture "the story," and the story is usually the most powerful depiction of the benefits of your services.
4. Don't interview just the successes. You'll learn a great deal
about the program by understanding its failures, dropouts, etc.
5. Don't throw away evaluation results once a report has been
generated. Results don't take up much room, and they can provide precious information later when trying to understand changes
in the program.
Doug Seubert
Non-Profit Development Specialist
PO Box 56
Marshfield, Wisconsin 54449
(715) 383-0897
|
 |
|
Evaluation Definitions
When it comes to the language of measuring results, a clear and consistent lexicon does
not exist. As stakeholders begin discussions about evaluation, it is likely that a variety of terms will come into play.
• RESULT:
A "result" is a bottom-line condition of wellbeing for children, families,
or communities. It is a broadly defined, fundamental condition that government and citizens consider essential. One such bottom-line
expectation of the community might be that all of its children should be born healthy. Results are umbrella statements that
capture the comprehensive set of needs that must be met to achieve success. By definition, achieving these basic conditions
of success requires concerted action by all sectors of the community. Some states and communities use the term "outcome"
instead of "result." The meaning is the same. However, the term "result" avoids potential confusion with unrelated debates
about outcomes-based education.
• INDICATORS: Indicators are measures, for which we have data, that gauge community-level progress toward agreed-on
results. Because results are broad statements of what communities want for their children, no single indicator is likely to
signal full attainment of any given result. Communities must decide what constellation of indicators add up to progress on
each result and then require a community-wide, cross-agency effort. Indicators that are most often used by community-based
organizations and schools include measurement of the following: decision making, civic responsibility or youth leadership
skills, academic achievement, and gainful employment.
• PERFORMANCE MEASURES:
Performance measures reflect the achievement of agencies and specific programs.
As such, they gauge progress at the agency level rather than at the community level. Appropriate performance measures are
closely related to an agency’s mission and purpose and are within its ability to control. They are narrow measures of
how well programs operate with their service populations, as part of a larger strategy to achieve results for the whole population.
Examples of performance measures include improvement in attendance for youth attending after school programs or high school
graduation rates for program participants.
Key Considerations
Consider the following key questions when designing a program
evaluation.
1. For what purposes is the evaluation being done, i.e., what do you want to be able to decide
as a result of the evaluation?
2. Who are the audiences for the information from the evaluation, e.g., customers, bankers, funders,
board, management, staff, customers, clients, etc.
3. What kinds of information are needed to make the decision you need to make and/or enlighten
your intended audiences, e.g., information to really understand the process of the product or program (its inputs, activities
and outputs), the customers or clients who experience the product or program, strengths and weaknesses of the product or program,
benefits to customers or clients (outcomes), how the product or program failed and why, etc.
4. From what sources should the information be collected, e.g., employees, customers, clients,
groups of customers or clients and employees together, program documentation, etc.
5. How can that information be collected in a reasonable fashion, e.g., questionnaires, interviews,
examining documentation, observing customers or employees, conducting focus groups among customers or employees, etc.
6. When is the information needed (so, by when must it be collected)?
7. What resources are available to collect the information?
Basic Principles for Small
Nonprofits to Remember Before Starting
Nonprofit personnel do not have to be experts in outcomes-based evaluation
in order to carry out a useful outcomes evaluation plan.
- In most major activities in life and work, there is a "20% of effort
that generates 80% of the results". This basic guide will give you the direction to accomplish that 20% needed to develop
an outcomes evaluation plan for your organization.
- Once you've carried out the guidelines in this basic guide, you can
probably let experience and funders help you with the rest of your outcomes evaluation planning, particularly as you implement
your evaluation plan during its first year.
- In life (particularly for us adults), problems exist often because
we’re making things far too complex, not because we're making things far too simple. Often, people who are new to evaluation
get "mindcramp", that is, they think too hard about evaluation. It's actually a fairly simple notion -- just don't think so
hard about it!
- Start small, start now and grow as you’re able.
- Ready, fire, aim!
Need more help? Advantage Consulting Services
specializes in collecting and interpreting data for evaluations and performance measurement.
|
|
|
|
|
|
|
|
