NR 587AI Week 3 Training Data Critique Example

Training Data Critique: The Deterioration Index's Development Hospitals Set Beside Our Four Medical-Surgical Units, Line by Line, With a Direction for Every Difference

Student Name

College of Nursing, Chamberlain University

NR 587AI: Advanced Nursing Leadership in Artificial Intelligence-Integrated Healthcare Environments

Instructor Name

Month Day, Year

Training Data Critique: The Deterioration Index's Development Hospitals Set Beside Our Four Medical-Surgical Units, Line by Line, With a Direction for Every Difference

Provenance Before Judgment

According to the vendor's written answer to our evaluation questions, received on 12 January 2027, the deterioration index was developed on about 130,000 adult admissions to three academic medical centers between 2012 and 2016, and validated on a further 40,000 admissions from the same centers in 2017. The outcome it predicts is a composite of rapid response call, transfer to intensive care or death within 12 hours. The vendor states that it has since been validated at two community hospitals, both over 350 beds, but has not shared the figures.

Our Case Mix, in the Same Terms

The development figures come from the vendor's answer; ours come from our analyst's report of 10 December 2026.

Operational Differences Move Outputs Too

Demographic differences are the easy ones to see. The differences most likely to move the score are operational.

Batch charting. On our night shift, nurses often take vital signs at 00:00 and 04:00 but enter them together near 06:00 when the workload eases. The score updates only when data are entered, so during the night it may be running on values several hours old.

Laboratory timing. Our morning laboratory draw is at 04:00, with results around 05:30. The development hospitals drew at varied times. Our scores therefore jump at 05:30 for many patients at once.

Transfers. A quarter of our patients arrive from two smaller hospitals, often with their first laboratory results drawn elsewhere and not in our record. Their early scores rest on fewer inputs.

Response capacity. With fewer ICU beds and no dedicated night rapid response team, our threshold for calling for help may differ, and the outcome the model predicts, which includes rapid response calls, depends partly on how readily a hospital calls them.

Giving Every Difference a Direction

Older patients. Many older patients have chronically abnormal vital signs and laboratory values. Direction: the score is likely to run high, producing more flags that turn out to be nothing. Over-call.

Active cancer. Our oncology unit has many patients with low blood counts and chronic tachycardia. Direction: over-call, concentrated on one unit.

Batch charting at night. Scores lag the patient. Direction: under-call during the night, with a late surge at 06:00.

Morning laboratory surge. Direction: a burst of flags at 05:30, just before shift change, when nurses have least time.

Transfers. Fewer inputs early in the stay. Direction: uncertain, probably under-call in the first hours.

Fewer rapid response calls in our hospital. Direction: because the predicted outcome includes calls, a hospital that calls less may see the model's predictions appear less accurate even when patients deteriorate at the same rate.

What the Vendor's Single Figure Hides

The vendor reports one summary accuracy figure for its development data. A model can rank patients well overall and still perform poorly at the threshold a hospital uses or in the subgroups that matter. When a widely used deterioration index was studied in patients with COVID-19, its overall ranking was reasonable but its performance at particular cut points was weaker (Singh et al., 2021). Our check below therefore reports results at the threshold we use and for the subgroups the comparison above identifies, rather than one overall figure.

Ranking the Differences

The two most likely to change what nurses experience are age and cancer (over-call, every day) and batch charting (under-call at night, when staffing is thinnest). Laboratory timing matters for workload. Transfers and response capacity matter but are harder to size.

Finlayson et al. (2021) describe how models can fail when the data they meet in practice differ from the data they learned from, including through changes in workflow rather than in patients. Our batch charting is an example: the patients may be similar, but the way their data arrive is not.

What Would Make It Usable Here

The critique does not conclude that the tool is unusable. It suggests three conditions. First, a unit-specific threshold for the oncology unit, set using our own data. Second, a change in night charting practice, or a clear statement to nurses that night scores may be out of date. Third, local validation before any decision, described below.

Obermeyer et al. (2019) found that a care-management algorithm under-ranked the needs of Black patients, because the quantity it learned to predict was spending, which tracks access as well as illness. Our population is mostly white and the deterioration index predicts clinical events rather than cost, so that specific bias is less likely here, but the case shows why the outcome a model predicts must be checked against what the hospital wants to know.

Ending on Something Checkable

Who Will Read the Results

The results go to the nursing and medical quality committee with this critique, so that the directions predicted here can be checked against the data before the risk assessment.

The Check

Our analyst will run the model's archived scores for all 2025 admissions on the four units against the outcomes recorded in our record, and report by 15 February 2027: flags per day at a threshold of 60, the share of flags followed by the outcome within 12 hours, the share of outcomes preceded by a flag at least 2 hours earlier, and each of these split by unit, by shift and by age over or under 80. If the oncology unit's positive predictive value is less than half that of the other units, that confirms the over-call direction. If the share of night outcomes preceded by a flag is clearly lower than the share on days, that confirms the under-call direction.

Limits

The vendor's description is brief and has not been independently checked. Our case mix figures come from one year. The directions above are predictions to be tested, not findings.

References

Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021). The clinician and dataset shift in artificial intelligence. New England Journal of Medicine, 385(3), 283-286. https://doi.org/10.1056/NEJMc2104626

Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. https://doi.org/10.1126/science.aax2342

Singh, K., Valley, T. S., Tang, S., Li, B. Y., Kamran, F., Sjoding, M. W., Wiens, J., Otles, E., Donnelly, J. P., Wei, M. Y., McBride, J. P., Cao, J., Penoza, C., Ayanian, J. Z., & Nallamothu, B. K. (2021). Evaluating a widely implemented proprietary deterioration index model among hospitalized patients with COVID-19. Annals of the American Thoracic Society, 18(7), 1129-1137. https://doi.org/10.1513/AnnalsATS.202006-698OC