If you’re a frequent reader of Health Tech Insider (or better yet, a subscriber to our free weekly email newsletter), you know that we’re cheerleaders for artificial intelligence (AI) in healthcare because it has the potential to save lives and lower costs significantly. But that does not mean that all AI is good enough. Have you see the above video yet? It’s a simple automated soap dispenser, but it has a serious flaw in its software’s logic. It fails to recognize hands that have darker skin pigmentation.
For dispensing soap, this flaw is possibly amusing and at worst annoying. But for systems that make healthcare decisions, this type of mistake is not acceptable.
A study published in Science reveals racial bias in algorithms used to determine which patients get high-risk healthcare management. Researchers from UC Berkeley, University of Chicago, and Partners HealthCare in Boston looked at the records of about 50,000 patients enrolled in a certain hospital and found that the risk scores for black patients were significantly lower than the scores for white patients. This resulted in the black patients being less likely to receive the advanced treatment.
Looking at the patients’ medical histories, however, they found that 47% of the black patients should have been enrolled in the advanced program, instead of just the 18% who were selected based on the AI algorithm. Digging into it further, the researchers discovered that the algorithm weighted the risk score based on how much had been spent on a patient’s healthcare in the past. As it turns out, healthcare costs for black patients are much lower on average than for white patients with similar health histories.
The good news is that this bias can be addressed, and the software company behind the algorithm made adjustments to their system to rely more on other variables, such as avoidable costs or number of chronic conditions.
We still see AI as being a potential force for positive change in our healthcare systems, but as with any other technology, it must be developed carefully and tested thoroughly in order to make certain that it is producing the intended outcomes.
