New York, USA. – Google’s foray into the realm of artificial intelligence (AI) has reached a significant milestone as its medical chatbot successfully passed a challenging US medical licensing exam. However, the chatbot’s responses still fall short when compared to those provided by human doctors, according to a peer-reviewed study published on Wednesday.
The debut of ChatGPT, developed by OpenAI, a company backed by Google’s competitor Microsoft, ignited a competition among tech giants in the rapidly expanding field of AI. While the future implications, both positive and negative, of AI have been widely discussed, the healthcare sector has already witnessed substantial progress, with algorithms capable of interpreting specific medical scans on par with human experts, mentions the Science Alert website.
Google introduced its AI tool, Med-PaLM, for answering medical inquiries in a preprint study last December. Unlike ChatGPT, Med-PaLM has not been made available to the public.
The US tech giant claims that Med-PaLM is the first large language model—an AI technique trained on extensive amounts of human-generated text—to achieve a passing grade on the US Medical Licensing Examination (USMLE).
A passing score on this exam, which is taken by medical students and physicians-in-training in the United States, is approximately 60 percent.
In February, a study revealed that ChatGPT had achieved passing or near-passing results on the USMLE.
In a peer-reviewed study recently published in the journal Nature, Google researchers disclosed that Med-PaLM achieved a score of 67.6 percent on USMLE-style multiple-choice questions.
“While Med-PaLM performs admirably, it still falls short of clinicians,” the study concluded.
To address and minimize “hallucinations,” a term referring to instances when AI models provide erroneous information, Google developed a novel evaluation benchmark. Karan Singhal, a Google researcher and the lead author of the study, stated that the team employed this benchmark to test a more advanced version of their model, yielding “super exciting” outcomes.
AI Chatbot Developed by Google Achieves Success on Medical Exam#ArtificialIntelligence #Google https://t.co/Pc353QS5dQ
— ReturnByte – Daily Tech Dose (@returnbyte_tech) July 13, 2023
According to a preprint study released in May, which has yet to undergo peer review, Med-PaLM 2 has attained an impressive score of 86.5 percent on the USMLE exam, surpassing the performance of the previous iteration by nearly 20 percent.
Acknowledging the limitations of AI-powered medical chatbots, James Davenport, a computer scientist at the University of Bath in the UK, remarked that there exists an obvious challenge that needs to be addressed. Davenport emphasized that there is a stark distinction between answering “medical questions and actual medicine,” which encompasses the diagnosis and treatment of genuine health conditions.
Anthony Cohn, an AI expert at Leeds University in the UK, suggested that hallucinations are likely to persist as a challenge for large language models due to their statistical nature. Consequently, these models should always be perceived as aids rather than definitive decision-makers, Cohn argued.
Singhal proposed that in the future, Med-PaLM could serve as a supportive tool for doctors, offering alternative perspectives that may not have been considered otherwise.
Recent reports from The Wall Street Journal indicate that Med-PaLM 2 has undergone testing at the renowned Mayo Clinic research hospital in the US since April. However, Singhal refrained from disclosing specific partnerships and clarified that any testing conducted would not involve clinical settings or patient interactions that could potentially harm patients. Instead, the focus would be on automating relatively simple administrative tasks with minimal risk.
The development of Google’s AI chatbot passing the USMLE exam marks a notable achievement, signaling the progress made in the application of AI to healthcare. While there is still room for improvement, the potential for AI models like Med-PaLM to enhance medical practices, when used in conjunction with human expertise, is promising.
