NR 588AI Week 1 Delegating to Software Post Example

Week 1 Discussion: Initial Post

Delegating a Choice to Software: Who Gets a Transitional Care Call After Discharge, and Three Commitments to Obtain Before the Readmission Model Goes Live

Initial Post

Our four-hospital system discharges about 36,000 adults a year, and the transitional care team can call and follow about one in five of them. Until now, discharge nurses chose who got that follow-up using a list of diagnoses and age. If I wanted to know why a patient was or was not chosen, I could ask the nurse, and get reasons I could argue with.

Next spring, a vendor's readmission risk model will make that choice: the 20% of patients it scores highest each day go on the transitional care list. When I ask why a patient was not chosen, the answer will be a score. That is the old problem of delegation, handing a decision to someone else, with one difference: the delegate cannot explain itself in terms I can question at the bedside.

What replaces the conversation? I think three commitments, each easier to obtain before the contract is signed than after.

First, a condition in the purchase agreement that the vendor reports, every quarter, how well the model performs for specific patient groups in our population, and tells us in advance before changing the model. Who grants it: the system's vice president of supply chain, who signs the contract, on the advice of legal counsel.

Second, an independent person who reviews performance and can pause the tool, someone outside the care management department that is buying it. Who grants it: the system's top nursing and medical executives together, since the tool crosses both their areas. Char et al. (2018) point out that those who buy a clinical model may have interests that differ from patients', which is why the reviewer should sit outside the buying department.

Third, a written right for any nurse to add a patient to the list without asking permission. Who grants it: the vice president of care management, who owns the transitional care program.

Not every automated tool needs all three. A tool that formats discharge instructions does not decide who gets care. This one does, and the people it leaves off the list may never know they were considered.

That is my worry. A national study of a care management algorithm found it ranked Black patients as less sick than they were, since it learned from spending, and spending on them had been lower for equal illness (Obermeyer et al., 2019). Our model predicts readmission, not cost, but it uses prior visits as inputs, and patients with less access to care have fewer visits on record.

So the question I raised, what replaces asking for reasons, has a partial answer: a regular report of who the model misses, someone independent to read it and nurses who are free to correct it. I do not think that fully replaces the conversation, but it is what a leader can secure in advance.

Reply to Rachel

Rachel, your post weighed the benefits and risks of artificial intelligence in general. Is there one tool your organization is about to adopt? If so, what is one condition you would insist on before it goes live?

Reply to Victor

Victor, you wrote that your unit should accept the new fall risk tool because the vendor is reputable. What would you want to know six months from now, and who would have to agree today for you to find out?

References

Char, D. S., Shah, N. H., & Magnus, D. (2018). Implementing machine learning in health care: Addressing ethical challenges. New England Journal of Medicine, 378(11), 981-983. https://doi.org/10.1056/NEJMp1714229

Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. https://doi.org/10.1126/science.aax2342