NR 539 Week 7 Outcome Evaluation Plan Example

Outcome Evaluation Plan: Judging Whether Up and Steady Reduced Repeat Falls When There Is No Untreated Copy of the County Next Door, Using a Historical Cohort, an Interrupted Time Series and Honest Limits

Student Name

College of Nursing, Chamberlain University

NR 539: Population Health II: Planning, Implementation, and Evaluation

Instructor Name

Month Day, Year

Outcome Evaluation Plan: Judging Whether Up and Steady Reduced Repeat Falls When There Is No Untreated Copy of the County Next Door, Using a Historical Cohort, an Interrupted Time Series and Honest Limits

The Problem

Up and Steady offers every older adult in the county who has a lift-assist call a home visit, a fall risk screen, home modifications and a referral to strength and balance classes. Because the program is offered to everyone eligible, there is no untreated group of similar people in the county to compare with, and no neighboring county has comparable lift-assist data. A randomized trial was never possible within the grant. That absence is the central problem of the outcome evaluation, and this plan does not pretend otherwise. It uses designs that address the most likely alternative explanations for any change, states which explanations it cannot rule out, and limits its conclusions accordingly.

Outcomes

The plan evaluates three outcomes, matching the program's logic model. The primary outcome is repeat lift-assist calls: the share of people visited who have another lift-assist call within 12 months. The secondary outcomes are the monthly rate of lift-assist calls among all county residents aged 65 or older and the change in balance, measured by a timed standing and walking test, among class participants over 24 weeks. Fall-related hospitalizations will be reported but not treated as an outcome the program can be expected to change in two years.

The Main Threat: Regression to the Mean

The most important alternative explanation is regression to the mean. People enter the program because they have just had a lift-assist call, often after a bad week or month. Many would have fewer calls in the following year even without the program, simply because a cluster of falls is often followed by a calmer period. A simple comparison of each person's calls before and after the visit would therefore show improvement whether or not the program worked.

To address this, the primary outcome will be compared with a historical cohort: everyone aged 65 or older who had a first lift-assist call in the two years before the program began, followed for 12 months after that call using the same ambulance records. The historical cohort entered the same way, by having a call, so it will show the same regression. If people visited by the program have fewer repeat calls than the historical cohort, regression cannot explain the difference. The cohort was fixed when the baseline data were pulled in May, before the first visit, so it cannot be chosen after the results are known.

A Second Design: Interrupted Time Series

For the county-wide rate of lift-assist calls, the plan uses an interrupted time series. Monthly rates for the 36 months before the program will be compared with the 24 months during it, testing whether the level or trend changed when the program began. Interrupted time series designs are well suited to evaluating population-level interventions when randomization is not possible, because they use the population's own history as the comparison and can account for existing trends and seasonal patterns (Lopez Bernal et al., 2017). Winter will be modeled explicitly, since icy months bring more falls, and so will the change of season in late autumn. The 2022 change in how the ambulance service recorded lift assists falls within the pre-program period, so the series will start after it.

A Check on Other Explanations

Something other than the program could change lift-assist calls during the evaluation period: a change in ambulance staffing, a new assisted living facility, or a change in how crews code calls. To detect such changes, which would otherwise be mistaken for program effects, the plan will track lift-assist calls among residents aged 55 to 64, who are not eligible for the program but are served by the same crews and recorded in the same system. If calls fall among older adults but not among this younger group, a change in the service itself is a less likely explanation. If calls fall in both, the program cannot claim the change.

Balance in Class Participants

Every class participant's balance will be measured, by an instructor trained to give the test the same way each time, at the first session and at 24 weeks. Without a comparison group, improvement could reflect practice with the test itself or the natural recovery of people who joined after a fall. The plan will report the change but interpret it cautiously, noting that pooled randomized trials show exercise lowering fall rates in older people outside institutions (Sherrington et al., 2019), which makes improvement plausible but does not prove that ours was caused by the classes.

What the Evaluation Can and Cannot Conclude

If repeat calls among people visited are clearly lower than in the historical cohort, the county-wide series shows a change at the program's start, and the younger comparison group shows no similar change, the evaluation will conclude that the program probably reduced repeat falls requiring help. It will not be able to say by how much with precision, or which component, visits, modifications or classes, produced the effect. It will also be limited by who accepted the program: the process evaluation found lower acceptance among men living alone, so the results may not apply to them.

The plan also commits in advance to how results will be reported if they are disappointing. If the historical comparison shows no difference, the report will say so plainly and use the process evaluation to consider whether the program was delivered with enough reach and intensity to have an effect, rather than searching the data for a subgroup in which it seems to work. The plan's reporting will follow the RE-AIM approach of describing reach alongside effectiveness (Glasgow et al., 1999), so that a reader knows how many eligible people the results describe.

Timeline and Responsibility

The analysis will be simple enough to repeat each year if the program continues, since commissioners may want the same evidence in future budget cycles. The program lead will conduct the analysis with support from the state health department's epidemiologist. The historical cohort and pre-program series are already pulled. Interim results at 12 months will be presented to the steering group with a clear statement that they are preliminary, and the full analysis will be completed at 24 months, in time for the funder's decision about continuation.

References

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322-1327. https://doi.org/10.2105/AJPH.89.9.1322

Lopez Bernal, J., Cummins, S., & Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: A tutorial. International Journal of Epidemiology, 46(1), 348-355. https://doi.org/10.1093/ije/dyw098

Sherrington, C., Fairhall, N. J., Wallbank, G. K., Tiedemann, A., Michaleff, Z. A., Howard, K., Clemson, L., Hopewell, S., & Lamb, S. E. (2019). Exercise for preventing falls in older people living in the community. Cochrane Database of Systematic Reviews, 2019(1), Article CD012424. https://doi.org/10.1002/14651858.CD012424.pub2