Hugging Face’s Open Medical-LLM Assess AI Models in Medical Tasks 

Hugging Face has launched Open Medical-LLM, a new tool to test and evaluate AI models’ performance in medical tasks. 

Developed in partnership with Open Life Science AI and the University of Edinburgh, Open Medical-LLM was developed as a standard test to assess the performance of generative AI models regarding medical tasks.  

The AI startup’s tool combines existing medical tests, such as MedQA and PubMedQA to see how efficient these models are in performing many medical tasks, including summarizing patient records, as well as answering health related questions. 

This test also includes multiple choice and open-ended questions covering medical knowledge, anatomy, pharmacology, genetics, and clinical practice. 

The AI startup believes that this new test will help identify strengths and weaknesses by using AI approaches, which will lead to more advancements and enhancements regarding patient care. 

While such tools are considered a breakthrough for medicine, medical experts highlight the importance of controlling the reliance on such tests, pointing out that the gap between the test environment and real clinical practice can be significant.  

Clementine Fourrier, a research scientist at Hugging Face agrees, notes that although leaderboards can guide model selection, real-world testing remains essential to understand the model’s limits and relevance. 

For instance, Google’s experience with an AI screening tool for diabetic retinopathy in Thailand highlights the challenges. Despite high theoretical accuracy, the tool performed inconsistently in real-world testing, indicating the difficulty in translating lab performance to clinical settings. 

Open Medical-LLM provides valuable insights, yet it could not replace real-world testing.  

The FDA has not approved any of the generative AI medical devices, due to the complexity of assessing their performance and results when used in practical healthcare. 

Inside Telecom provides you with an extensive list of content covering all aspects of the Tech industry. Keep an eye on our Medtech section to stay informed and updated with our daily articles.