NR 539 Week 6 Process Evaluation Write-Up Example

Process Evaluation Write-Up: Whether Up and Steady Ran as Planned in Its First Six Months, Measured by Reach, Dose, Fidelity and the People It Missed, Before Anyone Asks Whether It Worked

Student Name

College of Nursing, Chamberlain University

NR 539: Population Health II: Planning, Implementation, and Evaluation

Instructor Name

Month Day, Year

Process Evaluation Write-Up: Whether Up and Steady Ran as Planned in Its First Six Months, Measured by Reach, Dose, Fidelity and the People It Missed, Before Anyone Asks Whether It Worked

Purpose

Up and Steady began home visits on July 1 and strength and balance classes on September 8. This write-up covers July through December. It does not ask whether the program reduced falls; six months is too early, and any outcome evaluation would be meaningless unless we first know whether the program was delivered as intended. Before the question of whether it worked comes the question of whether it ran. Saunders et al. (2005) set out the questions a process evaluation should answer, among them how many of the intended people were reached, how much of the program was delivered and received, and whether it was delivered as designed. This write-up answers it for each of the program's main components, using the ambulance service's records, visit documentation, handyman program logs, clinic acknowledgments and class attendance sheets.

Reach: Who Was Offered and Who Accepted

In six months, the ambulance service recorded 164 lift-assist calls for residents aged 65 or older, involving 118 different people. The community paramedic attempted to contact all 118. Eighty-one accepted a visit, 69%. Twenty-six declined, and eleven could not be reached after three attempts.

Acceptance was not even across groups. Among people living with a spouse or family, 78% accepted. Among men living alone, 52% accepted, and among men living alone on farms, it was 9 of 19, or 47%. Those who declined most often said they were fine, did not want a fuss or were too busy. Several added that they did not want anyone to think they could no longer manage at home, a worry that surfaced in the listening session as well. The paramedic's notes suggest that people who had been told about the visit by the crew at the time of the call were more likely to accept than those who first heard of it on the telephone days later. Men living alone on farms were also the group the needs assessment's listening session had reached least well. The program is reaching the people most like those who helped design it, and least reaching some of those at highest risk.

Timeliness: Visits Within Seven Days

The target was a visit within seven days of the lift-assist call. For the 81 people visited, the median time from call to visit was five days, and 72% of visits occurred within seven days, below the 80% target. Most late visits were in August and December. In August, the paramedic took a week of leave with no cover, and in December, calls rose with icy weather while two ambulance crews were short-staffed, pulling the paramedic back to emergency shifts.

Dose: What Was Delivered at Each Visit

Of 81 visits, 79 included a complete fall risk screen; two were stopped early because the person was unwell. All 81 included a home hazard check. The paramedic identified hazards needing modification in 66 homes, most often a bathroom with no grab bar, a rug that slid underfoot or a dark path between bedroom and toilet. The handyman program completed modifications in 54 of the 66 homes by the end of December, with a median of 16 days from referral, slightly longer than the 14-day target. Eight people declined modifications and four were still waiting.

Medication lists and fall risk summaries were sent to primary care for all 79 people screened. Clinics acknowledged 41, or 52%. Two clinics acknowledged almost all summaries; three acknowledged few. The program cannot yet tell whether unacknowledged summaries were read. It can tell that the two clinics that acknowledged almost everything also changed medications for several patients after receiving summaries, most often reducing a sedating drug taken at night, while no such changes were recorded from the other three clinics. That is the clearest sign so far that the summaries matter when they are read, and the clearest reason to find out why some are not.

Fidelity: Were Visits Done as Designed

The program lead reviewed documentation for a random sample of 20 visits against the visit protocol's eight required elements. Seventeen visits documented all eight. Three omitted the conversation about what to do after a fall, which was added to the protocol after the intervention selection memo and was not included in the paramedic's original training. The paramedic has since been briefed, and the element has been added to the documentation template so that it cannot be skipped.

Classes

Classes began at the senior center and two churches in September, using Tai Ji Quan: Moving for Better Balance, a program that reduced falls among older adults in a randomized trial (Li et al., 2005). Forty-one people enrolled: 22 referred by the paramedic and 19 who heard about the classes through the senior center or churches. By the end of December, after 16 of 24 weeks, 29 of the 41, or 71%, had attended at least three-quarters of sessions. Attendance was lowest at the church site farthest from the county seat, where three participants depended on rides from one volunteer driver.

What the Process Evaluation Changes

The findings lead to four changes. First, the ambulance service has agreed to name a backup paramedic for leave and for weeks when the community paramedic is pulled to emergency shifts. Second, for people who decline, the paramedic will offer a shorter visit or a telephone conversation, and crews will be asked to mention the program in person at the time of the call, since acceptance was higher when crews had done so. Third, the program lead will visit the three clinics with low acknowledgment to ask what would make the summaries useful. Fourth, the aging agency will recruit a second volunteer driver for the farthest church site.

The RE-AIM framework distinguishes reach from effectiveness and implementation (Glasgow et al., 1999). This evaluation shows reasonable reach overall but weak reach in one high-risk group, good fidelity with one gap now closed, and timeliness below target for reasons that can be fixed. The outcome evaluation will need to take these findings into account: if repeat falls do not fall among men living alone, it may be because too few of them received the program, not because the program does not work.

References

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322-1327. https://doi.org/10.2105/AJPH.89.9.1322

Li, F., Harmer, P., Fisher, K. J., McAuley, E., Chaumeton, N., Eckstrom, E., & Wilson, N. L. (2005). Tai Chi and fall reductions in older adults: A randomized controlled trial. The Journals of Gerontology Series A: Biological Sciences and Medical Sciences, 60(2), 187-194. https://doi.org/10.1093/gerona/60.2.187

Saunders, R. P., Evans, M. H., & Joshi, P. (2005). Developing a process-evaluation plan for assessing health promotion program implementation: A how-to guide. Health Promotion Practice, 6(2), 134-147. https://doi.org/10.1177/1524839904273387