Back in 1821 when Charles Babbage introduced the world to his Difference Engine, one of the world’s first mechanical computers, he taught us that bad input = bad output. This is a lesson we are still learning today in healthcare. As we leap into the world of artificial intelligence (AI), machine learning (ML), and large language models, we would do well to remember this lesson before relying too heavily on the output of these fantastical technologies.
At the recent HIMSS23 Conference in Chicago, Charlie Harp, CEO of Clinical Architecture – a company that provides solutions for healthcare data quality, interoperability, and clinical documentation – delivered a spotlight session that highlighted the work of Babbage and his important lesson.
Charlie Harp giving us a history lesson – channeling Charles Babbage and his “difference engine” – even back in the mid-19th century bad input = bad output #interop #HIMSS23 @ClinicalArch pic.twitter.com/Wtdr9zhJvj
— Colin Hung (@Colin_Hung) April 19, 2023
Relying Output from Bad Input is Bad
In his presentation Harp recited this hilarious quote from Babbage from the early 1800’s (you have to read it and imagine a posh British accent):
“On two occasions I have been asked [by members of Parliament], ‘Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?’ I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.”
Using dry English wit, Babbage effectively created the concept of “Garbage In. Garbage Out.” This concept is as true today as it was in Babbage’s time and we are in danger of not learning the lesson.
In recent months the world has become enamored with new AI tools like ChatGPT which can perform amazing feats of writing while responding to user created prompts. Over the past few years, AI tools have been helping to improve radiology workflows, direct patients to the most appropriate level of care, optimize clinician schedules, and automated thousands of administrative tasks. Yet, rarely do we ask the question – what was the data that was used to train these AI tools? Is that data representative of the world these AI tools now operate in? Are we confident the data was of good quality?
Harp challenged the audience to think about this during his HIMSS23 presentation.
In order to truly leverage new algorithms, apply ML or even to get better reporting, the data on a patient needs to be as accurate as possible and as high quality as possible or we risk drawing bad conclusions – via Charlie Harp #HIMSS23 @ClinicalArch #HITsm pic.twitter.com/SXWYttx2O1
— Colin Hung (@Colin_Hung) April 19, 2023
More Focus on Data is Happening
It was very apropos that Harp presented his own data analysis as part of his presentation. He analyzed the occurrence of the term “Garbage in. Garbage out.” In PubMed and plotted the results over time.
Love the analysis that Charlie Harp did on PubMed – looking at the occurence of “Garbage-in, Garbage-out” For a long time, it was quiet but as we starting using more data the occurrence started to rise. @ClinicalArch #HIMSS23 pic.twitter.com/bkeYuFMlzV
— Colin Hung (@Colin_Hung) April 19, 2023
From 1972 to 1998 there is barely a mention of the term. From 1999 to 2022, however, Harp found a steady rise in the use of the term in publications. Interestingly, Harp also plotted the major Health IT milestones on his chart like – MIPPA, MIPS, MACRA, and CURES. You could say that the concern around data seems to be growing as the need for quality data rises through regulations.
This analysis aligns with a message that Healthcare IT Today has discussed with Harp on recent occasions – that quality data is vital to healthcare.
Quality of Patient Data
According to Harp, the biggest determinant of the level of quality of health data is patient data.
Charlie Harp shares that’s patient data is the largest contributor to the overall quality of your data. #HIMSS23 @ClinicalArch pic.twitter.com/k3pKSi8ODs
— Healthcare IT Today (@hcittoday) April 19, 2023
In a recent data quality survey conducted by Clinical Architecture, Harp’s team found two interesting results. First was that healthcare organizations felt that their SDOH, Allergies, and Procedure data was the poorest quality vs Demographic data, which was ranked highest quality.
What is the quality of the specific domain of patient data?
Demographic data ranked as high quality, SDOH ranked the poorest quality. #HIMSS23 @ClinicalArch pic.twitter.com/HRB8TFW1eS
— Healthcare IT Today (@hcittoday) April 19, 2023
The second interesting result was that overall, organizations are not very confident in the quality of patient data they have collected. Worse, the survey found that organizations have very little trust in data that originates from outside their organization.
The survey found that we do not feel that the quality of our patient data is where it should be and we don’t trust the data that comes from others (it is the only data worse than ours).#HIMSS23 @ClinicalArch pic.twitter.com/KQdUKG8ZXe
— Healthcare IT Today (@hcittoday) April 19, 2023
And therein lies the paradox. If we ourselves are not confident in the quality of the health data we have collected, then how confident should we be in tools that are based on or trained on that same data? After all, where do the companies that are making the AI algorithms get the datasets they use for training? Makes you wonder.
“Poor patient data quality impacts our ability to be successful as an industry”#HIMSS23 @ClinicalArch pic.twitter.com/Tws6uOE9hV
— Healthcare IT Today (@hcittoday) April 19, 2023
For the full survey results check out: https://clinicalarchitecture.com/data-quality-survey/
Improving Data Quality
Harp ended his presentation on a positive note by quoting Aristotle, which for accuracy, he used the actual translated quote: “As it is not one swallow or fine day that makes a spring, so it is not one day or a short time that makes a man blessed and happy.” In other words, our desired endpoint does not happen in an instant or with one event. It is achieved over time.
If we want quality health data (and we should, according to Harp) then we need to invest the time and resources to make it so. We have the technology to solve our data challenges, now we need to be willing to commit ourselves to the journey of data quality.
If we don’t then Babbage will have been correct all those years ago.
Learn more about Clinical Architecture at: https://clinicalarchitecture.com/
Clinical Architecture is a sponsor of Healthcare Scene.