Program evaluation is a crucial tool for assessing the effectiveness and maintaining ethical integrity in public health programs. A frequent assessment provides essential data that will help improve your program, build trust, and engage your community. Public health programs are complex and ever-evolving. Yet, the following six-step evaluation process can help you organize your assessment and get meaningful and actionable conclusions.
These are the six steps that can improve your public health programs:
Step 1. Engage the Stakeholders
When evaluating a public health program, you want to pay particular attention to three groups of stakeholders; those affected, those who administer or carry out public health work, and those who will use the results of your evaluation. You will need to engage each of these groups. Start with asking questions about what they want or need from both the program and the evaluation process. You should schedule different check-in points with each group of stakeholders. Frequent meetings will allow you to listen to their feedback on open-ended questions. You will learn about what matters to them and what they want to see going forward.
Do not be afraid to get more specific, asking questions such as:
- What steps are needed to increase the credibility of or effective advocacy with this program and its evaluation?
- Who will administer which parts of this program, and what do they feel they need to be successful?
- What are the required permissions, funding, or authorization decisions during this program? Who has the final say in those decisions? Is everyone involved onboard and clear on the necessary actions?
Step 2. Describe the Program
Create a single document with a comprehensive description of the program you are evaluating. Include elements like goals, targeted populations, evidence of need, and specific, measurable impacts. Your meetings with stakeholders should shape this description.
Then, list every activity this program has or will undertake—yes, every single one. Tie these activities—the labor of staff, volunteers, and other partners—to specific goals, and then check whether these goals align with your programmatic goals and whether the activities are meeting them.
While we are on the subject of the program description, remember that the way you talk about the evaluation matters. How you describe the evaluation clarifies your thinking about it. Your description influences who gets engaged and what their level of engagement is.
Prepare materials for your logic models by considering the following:
- What were the outcomes of the selected activities?
- How did the activities change over time?
- How did the results change over time?
Step 3. Focus Evaluation Design
Now it is time to design your actual evaluation. Once you have all the input, data, and framing documents compiled, you can construct your framework. The purpose that prompted the assessment will likely drive the bulk of what questions you ask in the evaluation document. But that should not be the entire evaluation.
You might also consider issues such as:
- Who will use the results of this evaluation, and what questions can I ask to make these results useful to them?
- What phase is the project in, and how should that information impact the program?
- Are the goals I am evaluating subjectively measurable? If not, how can I adjust the evaluation to consider quantifiable objectives?
- Does my evaluation adequately address all the concerns raised by stakeholders? If not, how can I make the process more inclusive?
Step 4. Gather Credible Evidence
While many evaluation formats collect data about a project, it is vital to collect enough data to provide context. For example, let’s assume you are evaluating a project for effectiveness, and it appears that the program is failing. Your data should allow you to differentiate whether that perceived failure is because the program itself is not working or because this new initiative did not have enough time to meet its goals.
In many situations, this will mean developing detailed protocols for collecting data that account for variables, repeating data collection to ensure consistency, and even considering a mixed-methods approach. Once again, relying on stakeholders here is crucial. They can distill which data they need and even assist in the collection process.
Make sure your data is reliable, valid, and informative. Ask strategic questions such as:
- Are the sources and methods credible and suitable for my purpose?
- Does the evaluation team have the expertise to collect and organize credible data?
- How do I maintain confidentiality and avoid disrupting the program during the collection process?
Step 5. Justify Conclusions
Once your data is collected, you have to figure out how to represent it. This process should include verifying data for accuracy, collecting additional data as necessary, and finding ways to present data to your various stakeholders with the appropriate context. If you used multiple ways of data collection, you need to compare the data across the different approaches. Cross-examine the information for consistency as well as any anomalies or potential disputing data that might emerge. If your data contradicts what you expected to find, you will want to look for sufficient context in your reports. The context will allow you to discuss the issue with stakeholders and figure out how to address any problems.
When interpreting your findings, remember to:
- Check for alternative explanations of your results
- Consider possible bias
- Compare your findings with similar projects
- Contrast different data collection methods
Step 6. Ensure, Use, and Share Lessons
While you might want to share data at the end of your evaluation process, you should consider dialoguing with stakeholders throughout the evaluation process. Doing so allows their feedback to shape your final reports, helping you improve the way you present data and ensuring it will be as usable as possible.
Talking early and often with stakeholders also establishes a pattern of continued conversation, lesson-sharing, and general use of the evaluation data. The last thing you want out of a public health program evaluation is a thick report that gathers dust in a closet. Make sure you are frequently engaging with all parties involved. You also need to ensure that you are creating statements tailored to the specific needs identified by stakeholders. The conversation around your data should focus on continued use and actionable changes based on the evaluation.
Before you share your results, you need to:
- Identify ways to reduce wrong interpretation of findings
- Ensure that your delivery is clear, timely, and in a format that can be easily used and distributed
- Strive to promote the program, its long-term goals, and its public image.
To help you simplify your program evaluation process, you can download our free tool that includes directions, checklists, and worksheets. This guide combines data from the CDC data and our expertise. The program evaluation tool is efficient, methodologically sound, and practical.
Do you need more assistance? Let our public help experts help you improve your efforts. Connect with your specialized partner today. Let’s Connect.
REFERENCES & RESOURCES
- Anon, (2019). Framework for Program Evaluation – CDC. [online] Available at: https://www.cdc.gov/eval/framework/index.htm.
- www.cdc.gov. (2018). CDC – Program Evaluation Steps. [online] Available at: https://www.cdc.gov/eval/steps/index.htm.
By: Erik McLaughlin MD, MPH and Aikaterini Papadopoulou, B.Arch
