Google's AI health chatbot Med-PaLM cracks US medical licensing exam, says study

Artificial Intelligence, which previously could only scare the humanity through works of fiction can now answer real-life questions, write codes and even seduce people. In what can be dubbed as next achievement (or travesty, depending on how you see it), an AI chatbot has passed a tough medical examination in the US. The chatbot is a health chatbot developed by Google. 

Med-PaLM, the google AI chatbot, has cracked the medical licensing exam but a peer-reviewed study says its answers still fall short of those from human doctors. 

The launch of ChatGPT, a generative AI developed by Microsoft-backed OpenAI, made the topic of artificial intelligence extremely popular. Much has been said about benefits and possible dangers of AI but health is one area where the tech has shown good progress. Media reports suggest that algorithms have been able to read certain medical scans as well as humans.

Google unveiled Med-PaLM in a preprint study in December. The AI chatbot has not yet been released to public.

Now, in a peer-reviewed study published in the journal Nature, researchers from Google said Med-PaLM scored 67.6 per cent on US Medical Licensing Examination (USMLE). The minimum passing percentage required to clear this examination is 60 per cent.

"Med-PaLM performs encouragingly, but remains inferior to clinicians," the study said.  

Google has said that it has developed a new evaluation benchmark to identify and cut down 'halucinations' -- the name given for when AI models offer false information.

Karan Singhal, a Google researcher and lead author of the new study, told AFP that the team has used the benchmark to test a newer version of their model with "super exciting" results. 

Med-PaLM 2 has reached 86.5 per cent on the USMLE exam, topping the previous version by nearly 20 per cent, according to a preprint study released in May that has not been peer-reviewed. 

James Davenport, a computer scientists at the UK's University of Bath said that "there is an elephant in the room" when it comes to AI-powered medical chatbots. He was quoted by AFP.

Davenport further said that there was big difference between answering "medical questions and actual medicine,"

Anthony Cohn, an AI expert at the UK's Leeds University, said that hallucinations would likely always be a problem for such large language models, because of their statistical nature. 

Therefore these models "should always be regarded as assistants rather than the final decision makers," Cohn said. 

Singhal said that in the future Med-PaLM could be used to support doctors to offer up alternatives that may not have been considered otherwise.