Health experts warn popular AI chatbots, like OpenAI’s GPT and Replika, risk putting a patient’s mental health in jeopardy by giving medical advice, from toxic substitutions to suicidal guidance, due to chatbots’ flawed algorithms and clinical oversight lack, which could impact mental health services.
Recent incidents include a 60-year-old man hospitalized after a bot suggested replacing salt with toxic wastewater chemicals, sodium bromide. Other cases exposed ChatGPT giving teens detrimental advice around drug and alcohol consumption and sometimes suggesting suicide, according to a Center for Countering Digital Hate study.
“Unfortunately, none of these products were built for that purpose,” said senior director of Health Care Innovation at the American Psychological Association, Vaile Wright.
“The products that are on the market, in some ways, are really antithetical to therapy because they are built in a way that their coding is basically addictive,” Wright added.
AI Competing with the Best Mental Health Treatment Centers
Doctors warn that while chatbots may offer quick answers, those comprehensive mental health services responses can dangerously impact mental health services in an inaccurate way.
“That being said, the quick answer may not be accurate,” said chief medical information officer at the American Medical Association, Dr. Margaret Lozovatsky. She recalled asking a bot how to treat a urinary tract infection, only to receive the response: “Drink urine.”
Chatbots lack the ability to access a patient’s medical history and intensive mental health treatment which lead them to apply clinical judgment, meaning their answers are often generic. Experts note that AI models can “hallucinate” — producing incorrect or even harmful recommendations — and risk giving users false confidence.
Dr. Tiffany Munzer of the University of Michigan added that chatbots also mimic users’ emotions, which can backfire, saying that “if you’re in a sadder state, the chatbot might reflect more of that emotion.”
“The emotional tone tends to match and it agrees with the user. It can make it harder to provide advice that is contrary to what the user wants to hear,” said Munzer
Privacy is another concern regarding mental health treatment plans.
Asking GPT-like chatbots sensitive questions on different levels of mental health treatment can expose personal health data online, with little oversight on how it is stored or used.
“A chatbot does not have clinical judgement,” Lozovatsky emphasized, “You lose the relationship you have with a physician.”
AI Delivering in Home Mental Health Services
Despite the risks, experts acknowledge that AI mental health startups could one day help fill gaps in care, particularly for people who struggle to access traditional health services. Any chatbot could become the new clinical mental health counseling master. With telehealth already mainstream, AI-driven tools could extend support into the home — offering guidance between doctor visits or supplementing therapy in areas where providers are scarce.
“I do think you’ll see a future where you do have mental health chatbots that are rooted in the science, that are rigorously tested, they’re co-created for the purposes, so therefore, they’re regulated,” Wright said. “But that’s just not what we have currently.”
Advocates argue that, if properly designed and overseen, AI relationship therapist could eventually serve as an accessible first point of contact for mental health needs, helping patients track symptoms, practice coping strategies, and connect with professionals when necessary.
However, until those safeguards of AI in psychotherapy are in place, experts stress that in-home AI services should be seen only as complementary — not a replacement — for trained medical providers so that they won’t impact mental health services and lead to a misdiagnosis.
Inside Telecom provides you with an extensive list of content covering all aspects of the tech industry. Keep an eye on our Intelligent Tech sections to stay informed and up-to-date with our daily articles.