MPH 507 Week 6 Evaluation Design Paper Example

Evaluation Design: Was Screen at Home Delivered as Planned, Did Overdue Women Get Screened, and Was the Program the Reason?

Student Name

Chamberlain University

MPH 507: Public Health Program Planning and Evaluation

Instructor Name

Month Day, Year

Evaluation Design: Was Screen at Home Delivered as Planned, Did Overdue Women Get Screened, and Was the Program the Reason?

Three Questions, Kept Apart

This paper designs the evaluation of Screen at Home, the self-collection outreach described in weeks 3 to 5. Three different questions need three different answers.

The process question is whether the program was delivered as planned: whether community health workers reached the women on the list, offered kits, followed up unreturned kits and called women with positive results on time.

The outcome question is whether the women changed: whether overdue women got screened, whether women with positive results completed follow-up and whether women learned the screening interval.

The attribution question is whether the program caused the change, rather than some outside change in the same months, a new state reminder mailing, say, or shifts in insurance.

An evaluation that answers only the first, with counts of kits given and satisfaction ratings, and then concludes that the program worked would be measuring delivery and claiming effect.

Framework

Glasgow et al. (1999) proposed judging a program on five dimensions at once, from the share of intended people it reaches to whether it lasts, and their RE-AIM model organizes this design. RE-AIM fits a program delivered in an ordinary health center by existing staff, because it treats delivery in the real world as part of the result rather than as background.

Process Evaluation, Taken Seriously

Process measures are tied to the program's process objectives. Reach is the share of the 1,180 women on the list with at least one completed contact and the share offered a kit, broken down by district, age group and insurance, to show who the program is missing. Implementation is the share of unreturned kits followed by a reminder call and second visit on schedule, the share of positive results called within three working days, laboratory turnaround and the number of invalid samples. Adoption is the share of community health workers' scheduled hours actually spent on the program, since competing programs could crowd it out.

Process findings will be reviewed every quarter, not only at the end, so that a delivery problem can be fixed while there is time.

Outcome Evaluation, Tied to the Objectives

Outcomes are those already defined in the objectives: the share of the 1,180 women with a valid screening result by June 30, 2028; the panel up-to-date rate; the share of women with a positive kit who complete recommended follow-up within 90 days; and the share of kit recipients who can state the screening interval. Each uses the measure specified in the objectives, so that the evaluation tests the program against the promises made for it.

Design, Justified From the Question

The attribution question needs a comparison: what would have happened without the program. A randomized design, assigning women to receive the program or usual care, is not acceptable to the health center, which wants to serve every overdue woman. A stepped-wedge design fits the program's real constraints. The three districts will start at different times, four months apart, in an order set by lot before the program begins. Until a district starts, its overdue women receive usual care, which is a reminder letter. Comparing screening among women in districts that have started with women in districts not yet started, month by month, gives a fair comparison, because the order was chosen by chance and every district is eventually served.

The design was chosen because of the question and the setting, not because it was the most recent design covered in class. A simple before-and-after comparison of the panel rate would be cheaper but could not separate the program from other changes over two years. A comparison with a neighboring county was considered and rejected, because no neighboring county has comparable records.

Table 1

Evaluation Questions, Measures, Design and Timing

Note. Note. District start order set by lot before launch.

The Comparison, Handled Honestly

The stepped-wedge design has weaknesses that must be stated. With only three districts, chance differences between districts can be large, so the analysis will adjust for age, insurance and prior screening history. Screening is counted within six months of first contact, the window used in the U.S. mailed-kit trial, so results can be set beside that trial's 26% and 17% (Winer et al., 2019). Usual care may change during the program, for example if the state adds new reminders; the evaluator will log any such change with its date so it can be lined up against the district start dates. Women in later districts may hear about the kits from relatives in earlier ones and ask for them, which would make usual care look better than it is and understate the program's effect. The follow-up outcome has no concurrent comparison, because every positive woman will be navigated, so it will be compared only with the 2025 referral log, and that comparison will be described as descriptive, not causal.

Limits Stated in Advance

Four limits are stated before the evaluation begins, so that results are not reinterpreted afterward. The evaluation can measure screening and follow-up but not cancer outcomes, because cases are too few. It covers health center patients and the women met at food distributions, not all county women. It cannot separate the effect of self-collection from the effect of in-person delivery, because the program combines them. And costs will be tracked but not compared with a full economic model, though published analyses suggest mailed kits can be cost-effective in U.S. settings (Meenan et al., 2023).

Who Uses the Findings, and When

Quarterly process findings go to the program manager and the health center's quality committee, which can change delivery, such as adding evening visits, while the program runs. The final outcome and attribution findings go to the health center's board in September 2028, when it must decide whether to keep the program after the grant ends, and to the state cancer program, which funds similar work elsewhere. Planning for these users now shapes what is reported: the board needs cost per woman screened and a clear answer on attribution, while the quality committee needs reach by district and group. An evaluation designed without its users in mind tends to produce a report no one acts on.

References

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322-1327. https://doi.org/10.2105/AJPH.89.9.1322

Meenan, R. T., Troja, C., Buist, D. S. M., Tiro, J. A., Lin, J., Anderson, M. L., Gao, H., Green, B. B., & Winer, R. L. (2023). Economic evaluation of mailed home-based human papillomavirus self-sampling kits for cervical cancer screening. JAMA Network Open, 6(3), e234052. https://doi.org/10.1001/jamanetworkopen.2023.4052

Winer, R. L., Lin, J., Tiro, J. A., Miglioretti, D. L., Beatty, T., Gao, H., Kimbel, K., Thayer, C., & Buist, D. S. M. (2019). Effect of mailed human papillomavirus test kits vs usual care reminders on cervical cancer screening uptake, precancer detection, and treatment: A randomized clinical trial. JAMA Network Open, 2(11), e1914729. https://doi.org/10.1001/jamanetworkopen.2019.14729