Make your own free website on Tripod.com
Advantage Consulting Services

Evaluations & Performance Measurement

Home
FREE Services
CONTRACTUAL Services
Grant Writing
Capital Campaigns & Fundraising
Board of Directors Development
Evaluations & Performance Measurement
Strategic Planning
Sustainability Planning
Marketing & Promotion
Starting a New Non-Profit Organization
Non-Profit News
Workshops and Trainings
Contact Us

logo

Advantage Consulting Services has moved!

Visit us at out new Website:

www.advantageconsultingservices.org

As accountability has become a feature of program planning, evaluation has moved from a "desired" priority to a "required" one.  Increasingly, funded programs specify an evaluation component, with a new vocabulary for proposals, contracts, and program reports.  Wading through all of the requirements and terminology can be an overwhelming task for non-profit directors and staff. 
 
Advantage Consulting Services demystifies the process and helps non-profit organizations conduct comprehensive performance measurement and program evaluations to determine the effectiveness of programs for the people they serve; to document that the program objectives were met; to share information about the services and programs (both internally and externally); and to help the program staff make changes that will improve the program.
 
If you have any questions about evaluations and performance measurement, or if you would like more information about additional resources or the services we provide, please contact us by sending an email stating your questions or your specific request for information.

So What is Program Evaluation?

First, we'll consider "what is a program?" Typically, organizations work from their mission to identify several overall goals which must be reached to accomplish their mission. In nonprofits, each of these goals often becomes a program. Nonprofit programs are organized methods to provide certain related services to constituents, e.g., clients, customers, patients, etc. Programs must be evaluated to decide if the programs are indeed useful to constituents. In a for-profit, a program is often a one-time effort to produce a new product or line of products.

So, still, what is program evaluation? Program evaluation is carefully collecting information about a program or some aspect of a program in order to make necessary decisions about the program. Program evaluation can include any or a variety of at least 35 different types of evaluation, such as for needs assessments, accreditation, cost/benefit analysis, effectiveness, efficiency, formative, summative, goal-based, process, outcomes, etc. The type of evaluation you undertake to improve your programs depends on what you want to learn about the program. Don't worry about what type of evaluation you need or are doing -- worry about what you need to know to make the program decisions you need to make, and worry about how you can accurately collect and understand that information.

Where Program Evaluation is Helpful

Program evaluation can:

1. Understand, verify or increase the impact of products or services on customers or clients - These "outcomes" evaluations are increasingly required by nonprofit funders as verification that the nonprofits are indeed helping their constituents. Too often, service providers (for-profit or nonprofit) rely on their own instincts and passions to conclude what their customers or clients really need and whether the products or services are providing what is needed. Over time, these organizations find themselves in a lot of guessing about what would be a good product or service, and trial and error about how new products or services could be delivered.

2. Improve delivery mechanisms to be more efficient and less costly - Over time, product or service delivery ends up to be an inefficient collection of activities that are less efficient and more costly than need be. Evaluations can identify program strengths and weaknesses to improve the program.

3. Verify that you're doing what you think you're doing - Typically, plans about how to deliver services, end up changing substantially as those plans are put into place. Evaluations can verify if the program is really running as originally planned.

4. Facilitate management's really thinking about what their program is all about, including its goals, how it meets it goals and how it will know if it has met its goals or not.

5. Produce data or verify results that can be used for public relations and promoting services in the community.

6. Produce valid comparisons between programs to decide which should be retained, e.g., in the face of pending budget cuts.

7. Fully examine and describe effective programs for duplication elsewhere.

How to Write Comprehensive Reports

A comprehensive evaluation report should include:

• An executive summary of findings

• A description of the program that was evaluated (including funders, objective, and other key information)

• If applicable, information about the overall organization and how the evaluated program fits into that organization’s mission

• The purpose of the evaluation, the evaluation process, the methods used to carry out the process (tools, protocol, type of data-collection systems)

• Information (along with resumés or other biographical information) on the individuals who were responsible for carrying out the work (including the lead person and other members of the evaluation team)

• Information about the target area and statistical information about the target participants, and the connection between those general statistics and the purpose of your evaluation

• A report and discussion of findings: What did the evaluation reveal? How will the findings help facilitate program improvement? How can the information be used for program sustainability?

How to Communicate Results to Stakeholders

How do programs talk about or disseminate the results of their evaluation? The answer to this question connects back to accountability and the purpose of evaluation efforts. If a program conducts an evaluation for program improvement, results will most likely be communicated to administrators, staff, parents, and participants in various ways, many of them informal. However, if a program is conducting a formal evaluation for funders, it must consider more formal ways to communicate results, such as comprehensive reports or formal presentations.

Organizations have disseminated the results of their evaluations through:

• Presentation of results at staff meetings for management and staff within the organization

• Presentations at luncheons or seminars for external parties who are stakeholders (such as collaborative partners)

• Presentations at regional meetings or national conferences in the fields of education, youth development, family strengthening, or public policy

• Comprehensive reports for partners, community businesses, or funders who are looking for concrete documentation of program impacts

• Executive summaries or full reports posted to program or organization websites

Sharing results does not have to mean sharing every bit of information that was collected. Deciding which information is the most important to which audience is the key to communicating evaluation findings.

Pitfalls to Avoid

1. Don't balk at evaluation because it seems far too "scientific." It's not. Usually the first 20% of effort will generate the first 80% of the plan, and this is far better than nothing.

2. There is no "perfect" evaluation design. Don't worry about the plan being perfect. It's far more important to do something, than to wait until every last detail has been tested.

3. Work hard to include some interviews in your evaluation methods. Questionnaires don't capture "the story," and the story is usually the most powerful depiction of the benefits of your services.

4. Don't interview just the successes. You'll learn a great deal about the program by understanding its failures, dropouts, etc.

5. Don't throw away evaluation results once a report has been generated. Results don't take up much room, and they can provide precious information later when trying to understand changes in the program.

 

 

logo.jpg 
 
Doug Seubert
Non-Profit Development Specialist
 
PO Box 56
Marshfield, Wisconsin 54449
 
(715) 383-0897
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Evaluation Definitions
 
When it comes to the language of measuring results, a clear and consistent lexicon does not exist. As stakeholders begin discussions about evaluation, it is likely that a variety of terms will come into play.
 
RESULT: A "result" is a bottom-line condition of wellbeing for children, families, or communities. It is a broadly defined, fundamental condition that government and citizens consider essential. One such bottom-line expectation of the community might be that all of its children should be born healthy. Results are umbrella statements that capture the comprehensive set of needs that must be met to achieve success. By definition, achieving these basic conditions of success requires concerted action by all sectors of the community.  Some states and communities use the term "outcome" instead of "result." The meaning is the same. However, the term "result" avoids potential confusion with unrelated debates about outcomes-based education.

INDICATORS: Indicators are measures, for which we have data, that gauge community-level progress toward agreed-on results. Because results are broad statements of what communities want for their children, no single indicator is likely to signal full attainment of any given result. Communities must decide what constellation of indicators add up to progress on each result and then require a community-wide, cross-agency effort. Indicators that are most often used by community-based organizations and schools include measurement of the following: decision making, civic responsibility or youth leadership skills, academic achievement, and gainful employment.

PERFORMANCE MEASURES: Performance measures reflect the achievement of agencies and specific programs. As such, they gauge progress at the agency level rather than at the community level. Appropriate performance measures are closely related to an agency’s mission and purpose and are within its ability to control. They are narrow measures of how well programs operate with their service populations, as part of a larger strategy to achieve results for the whole population. Examples of performance measures include improvement in attendance for youth attending after school programs or high school graduation rates for program participants.

evaluations.jpg

Key Considerations

Consider the following key questions when designing a program evaluation.

1. For what purposes is the evaluation being done, i.e., what do you want to be able to decide as a result of the evaluation?

2. Who are the audiences for the information from the evaluation, e.g., customers, bankers, funders, board, management, staff, customers, clients, etc.

3. What kinds of information are needed to make the decision you need to make and/or enlighten your intended audiences, e.g., information to really understand the process of the product or program (its inputs, activities and outputs), the customers or clients who experience the product or program, strengths and weaknesses of the product or program, benefits to customers or clients (outcomes), how the product or program failed and why, etc.

4. From what sources should the information be collected, e.g., employees, customers, clients, groups of customers or clients and employees together, program documentation, etc.

5. How can that information be collected in a reasonable fashion, e.g., questionnaires, interviews, examining documentation, observing customers or employees, conducting focus groups among customers or employees, etc.

6. When is the information needed (so, by when must it be collected)?

7. What resources are available to collect the information?
 
 
Basic Principles for Small Nonprofits to Remember Before Starting
Nonprofit personnel do not have to be experts in outcomes-based evaluation in order to carry out a useful outcomes evaluation plan.

  • In most major activities in life and work, there is a "20% of effort that generates 80% of the results". This basic guide will give you the direction to accomplish that 20% needed to develop an outcomes evaluation plan for your organization.
  • Once you've carried out the guidelines in this basic guide, you can probably let experience and funders help you with the rest of your outcomes evaluation planning, particularly as you implement your evaluation plan during its first year.
  • In life (particularly for us adults), problems exist often because we’re making things far too complex, not because we're making things far too simple. Often, people who are new to evaluation get "mindcramp", that is, they think too hard about evaluation. It's actually a fairly simple notion -- just don't think so hard about it!
  • Start small, start now and grow as you’re able.
  • Ready, fire, aim!

 

 

 

Need more help? Advantage Consulting Services specializes in collecting and interpreting data for evaluations and performance measurement.