Doctors in our health system are increasingly relying on making decisions guided by AI systems. In 2018, the first-ever autonomous Artificial Intelligence (AI) diagnostic system has been introduced to the American health system. For many reasons we are amazed to see the promising advancements of AI: It relieves the overloaded healthsystem by supporting surgery, management and the monitoring of patients, as well as prescribing treatments. Imagining AI to increase healthcare’s efficiency seems too good to be true and raises concerns as to whether our trust in AI exceeds its capabilities. AI seems to be a powerful tool to boost our health system, but it also has controversial drawbacks. Are we actually ready to place trust in AI when it comes to our highest good: health?
Trust, per definition, is the belief that something is reliable or good. This belief might include a fair amount of uncertainty. If you are convinced that someone or something is reliable there is no need to have trust. Trust establishes stable expectations that make risk considerations less relevant. Surely, AI can outperform humans in many domains. But can we really allow ourselves to neglect a rational assessment of its risks?
A fundamental problem with AI is that its reasoning is based on information that might be biased. In the relatively short history of AI, we repeatedly heard of discriminatory cases. AI has classified black people as gorillas and detected Asians to be blinking: These are biases we need to be aware of. One fatal example in medicine is skin cancer detection using AI. Decades of clinical research focused primarily on middle aged white men. This specific data is fed into AI when training to detect dangerous molecules in the skin. The underrepresentation of patients of colour leads to lower sensitivity in detecting cancer in dark skin. This increases the potential risk to misdiagnose certain groups of people.
This is further problematised by the fact that most AI tools deployed in healthcare are “black boxes”. While we know the data that AI is fed with (medical information) and the results it produces (diagnostic decisions), we are lacking understanding of the processes in between. This makes it difficult for patients and doctors to place trust in AI. If we cannot understand the reasoning of AI, we cannot detect and reduce the systematic biases. This is important to keep in mind, since mistakes in the field of healthcare can have catastrophic consequences. Luckily, we can look into the future with hope. Explainability is an active field of research, trying to design AI that is more transparent to our understanding.
Another important question is the following: If a person of colour dies from cancer because the software is not trained to detect cancer in dark skin, who is responsible? Is it the provider of the AI selling a faulty product, the practitioner using the wrong tool, the government for allowing its use, or even the AI itself? When doctors need to make decisions, they have to carry the consequences themselves. However, embracing trust in AI easily fosters this diffusion of responsibility. We have to be aware that AI is merely a tool and cannot hold moral responsibility. Rather than placing our trust in AI, we should trust all of the entities mentioned above in evaluating the risks of AI.
We are convinced that AI needs to be implemented with care, not trust. When it comes to our health, we want the best help we can get. We have to be wary of biases that can be incorporated by training data, making sure we are able to detect them, and establishing relations of trust with all parties that are added to the client practitioner relationship. Thankfully, all of these issues are of central attention to researchers around the world. There are still a number of issues to be tackled before we can benefit from AI’s strengths in medicine without concerns. For now, it is crucial to step back from the hype around AI to ask: Is it trustworthy enough?