The following is a guest article by Anand Shroff, President at Verantos.
Artificial intelligence (AI) holds great promise to dramatically improve clinical outcomes for patients, but in a perverse twist, AI algorithms that are used to improve drug development and support healthcare don’t just highlight social inequities — they may ultimately exacerbate them. A lingering shortcoming of AI use in the life sciences and healthcare sectors is that both the underlying patient data and the rules that are established to develop and train the algorithm often have built-in biases.
AI algorithms are first developed and trained using real-world data (RWD) from typical sources like patient records and are then used to recognize patterns in broader data sets. AI algorithms are trained to find patterns in massive amounts of data, so when bias exists in the underlying RWD itself, or in the rules and assumptions used to develop the AI algorithms, then any patterns that are later identified by the AI algorithm in larger data sets will perpetuate that bias in the findings.
Similarly, AI may be inadvertently applied in inappropriate contexts, or used to evaluate RWD sets that were not necessarily the target of the underlying algorithm development, and this too can lead to results that perpetuate underlying biases in the system. Such bias reflects existing inequities found within the real world — biases related to socioeconomic status, race, ethnicity, religion, gender, disability or sexual orientation.
Unfortunately, people of privilege — those of a certain socio-economic status, race, ethnic background, religion, gender, and even sexual orientation — tend to receive better care and thus have better outcomes. In recent years, as financial incentives have begun to align with society’s collective moral compass, hospitals and health systems have sought to actively remedy systemic disparities in the quality of care.
Technology is behind much of the progress that’s been made to date. The adoption of electronic health records (EHRs) has made it easier to not only identify inequities in care, but to give stakeholders access to rich sources of RWD that can be used to develop AI algorithms. At the same time, the 21st Century Cures Act has provided a framework to enable broader use of real-world evidence (RWE) to advance many drug-development and healthcare objectives. Today, with concerted effort in how we develop and use AI, we have an opportunity to deliver more equitable care and enable more equitable outcomes for patients, regardless of socio-economic status, language, gender, net worth, or skin color.
How AI bias happens
Bias is not intentional. In fact, in many cases, AI researchers and the developers of the underlying NLP and ML algorithms may not be aware that the outcome they are trying to predict is itself the result of structural racism, sexism, or other forms of discrimination. Algorithmic systems are developed by humans who are influenced by their own pre-existing attitudes and biases, which can be based on cultural, social, or institutional experiences.
Overall, two major sources of bias can impact the validity of AI algorithms:
Algorithmic bias creates modeled results that are systematically erroneous due to faulty assumptions in the underlying rules and training of the algorithms. In healthcare, such sources of bias may unfairly privilege one particular group of patients over another.
In a landmark study published in Science Magazine, researchers concluded that one widely used commercial identification and stratification algorithm systematically deprioritized Black patients because the algorithm used healthcare costs as a proxy for severity, and it recognized incurred lower healthcare costs because Black patients were less likely to be treated. By the time these patients were identified for more intensive care, they were 26% more likely to have a chronic illness than white patients.
Selection bias leads to similar disparity. Selection bias occurs when the data used to train the algorithm are biased; i.e., focusing on a particular type of patient or condition, or being unnecessarily restrictive or limited.
A recent study published in Nature found that a deep learning model predicted about 90% of acute kidney injuries that subsequently required dialysis. However, only about 6% of patients in the dataset were female, and “model performance was lower for this demographic,” which translates to “does not work for women.”
Reducing and removing bias
There must be a concerted effort to eliminate bias in AI algorithms. To do this, the algorithms must be trained on data that is broadly representative and is free of bias. Similarly, the inclusion criteria established for randomized controlled trials should establish demographic and socioeconomic requirements that result in a truly representative and diverse cohort of patients. For algorithms that are trained on RWD, it is also important that such data be de-identified so that selection bias is not inadvertently introduced. Using de-identified data actively protects against biased selection when the AI algorithms are being trained.
As stakeholders work to reduce bias in AI, they should also be aware that the most well-curated data sets used to train AI algorithms typically come from prestigious academic medical centers. However, patients who have access to care at such facilities may not be representative of the population as a whole. A greater effort must be made when developing AI algorithms to use RWD sourced from community health systems, as this will provide access to high-quality EHR data sets from patients more representative of the population at large.
Another way to reduce bias in AI is to insist on these three characteristics for the data sets that are used to train the algorithm:
- The accuracy of the data must be both sufficient and documented
- The data must come from credible sources, with assurances that the processes used to capture the data were consistent and repeatable
- The data must be representative of the population — not selective to include or exclude certain patient sub-groups
There is no one-size-fits-all solution to eliminate bias in AI. However, with careful effort, it can be minimized.
About Anand Shroff
Anand Shroff is a healthcare and life sciences technology founder, executive, and investor. Anand currently serves as President of Verantos, an advanced real-world evidence company that has built an AI-powered platform to generate research-grade evidence for life sciences organizations.
