Introduction - Planning Process - Built Environment - Services - Sustainability - Evaluation - Policies & Procedures - Resources

Evaluation

What is a program evaluation?

An evaluation is a systematic assessment of an activity, project, strategy or policy. It analyzes both expected and unexpected results. It usually examines processes, contextual factors, outcomes, objectives and short and medium term impact-as well as sustainability. It should provide credible evidence based information on the impact of the program and recommendations for how to improve the program (1).

Why evaluate the maternity waiting home?

• Learn ways to improve the service delivery and function of the maternity waiting home (MWH)

• Demonstrate who you served and what you did

  • How many women stayed at the MWH?

  • How long did they stay? 

  • What services and referrals were provided?

• Assess whether or not progress was made towards the goals of the MWH program. For example, were referrals for facility based care increased? 

  • Did women receive needed services? 

  • Were women from remote areas included in services? 

  • Did more women complete antepartum visits? 

  • Did more women deliver at health facilities? 

  • Did more women receive postpartum care? 

  • Did more infants receive follow up visits and immunizations?

• Higher level (more complex) measures focus on impact and the ability to make a difference in maternal and infant health. 

  • Were maternal and infant health outcomes in your area better than they were before the MWH existed? Or better than in a similar community that does not have access to a MWH?

• Pathway to measure: Evaluation Data → Outcomes → Impact

  • May be influenced by other variables (quality of health system)

  • Measurement modalities (pre/post surveys, randomized control trial of maternity waiting homes versus quasi-experimental, matching)

  • Qualitative approaches/Mixed methods

How do you plan for a program evaluation?

Data sources for program evaluation vary based on the desired outcomes. They may include collecting data from logbooks/health records, surveys, and/or stakeholder interviews. Regardless of the data collection method, it is important to have goals/objectives that are driven by the purpose of the program. Below are two examples of creating purpose-driven goals/objectives.

1. SMART Goals or Objectives (2) 

The SMART acronym is a tool designed to help organizations set goals or objectives that an organization is hoping to achieve or evaluate. The SMART acronym translates to goals or objectives that are:

Specific – outline the concept that is being measured (Who and What?)

Measurable – include a measure to capture the concept and to monitor progress and know when the objective has been achieved (How much?)

Achievable - the goal or objective is achievable; can actually be measured within the desired timeframe and with available resources (Why?) 

Realistic - accurately address the scope of the problem and the steps to achieve stated goals/outcomes (How?)

Time-bound – agree the date by which the outcome must be achieved or the time frame for measurement (When?)

1. Logic Models

Logic models are another approach to both program design and evaluation. A logic model is a graphic representation of what a program intends to do and how the activities of the program would lead to the results. Creating logic models with groups of stakeholders is a way to bring multiple perspectives into the design and evaluation of an intervention. Working together a group can use a logic model to come to a shared understanding of the current situation, the changes the program would address, how the activities planned would improve the situation, what resources are needed, how change would be measured, and what external factors could influence the results (3). The key components of a logic model are:

Situation: This includes your problem statement (What problem does your program address?) and goals (What is the overall purpose of the program?)

Rationale and Assumptions: What are the underlying assumptions and important external factors that should be considered?

Inputs: What resources are available? Financial? Staff? Equipment? 

Activities: What will you do? When, where and for/with whom?

Outputs: What are the tangible products of your work?

Outcomes: What changes do you expect to occur as a result of your work? Short term? Medium term? Long term? Think about connections between the outcomes