Google’s recent health integration update highlights the evolution of AI from conversational assistants into integrated clinical ecosystems.
Following the launch of ChatGPT Health, the major technology company unveiled major updates to Gemini’s health and wellness integration. Through partnerships with Athenahealth, b.well Connected Health, and CLEAR, Google officially integrated a personal health coach update into Fitbit.
The AI tool allows users to integrate official hospital electronic health records and lab data directly into the app. According to Google, the platform can interpret lab results, recommend personalized wellness advise and pre-formulated questions for doctors. The company also added a mental health crisis update to Gemini.
This change comes after growing concern from the American Medical Association and other professional organizations. Many industry professionals are pushing back against AI modeling responses of clinical advice without being subjected to formal clinical oversight.
The Blurred Line Between General Health Information And Medical Advice
As generative AI companies expand further into mental health, regulators have sought to define where wellness and general health information ends and medical guidance begins.
Developers frequently frame consumer large language models as “search” tools instead of medical devices to sidestep the Food and Drug Administration’s clearance requirements. This framing has allowed AI models to provide advanced, interactive medical guidance while avoiding the regulatory standards imposed on traditional healthcare technologies.
Gemini can mimic clinical interaction and provide personalized health analysis with the recent update. However, in a recent safety updates blog post, Google maintained that its AI is not a substitute for professional clinical care. Similarly, Open AI updated its usage policy in 2025 to prohibit the use of its platform for tailored medical advice, but it can still provide individualized guidance when prompted with specific user scenarios.
The general public rarely understands or respects this distinction. Since outputs are classified as non-medical grade, the burden of safety shifts onto the end user.
Several risks emerge when AI tools are not perceived by the public as search engines. Technical limitations introduce measurable concerns, including large language models’ tendency to synthesize information into a single, confident and personalized narrative that can appear authoritative even when uncertainty exists.
ChatGPT And Gemini Face The Ethics Of Automated Empathy
As big technology companies races to seize the mental health space, ethical skepticism and high-profile lawsuits have followed closely behind. Patient safety group ECRI labeled AI chatbot misuse a top technology hazard, citing ChatGPT, Gemini, Claude, and Copilot.
The debate following high-profile lawsuits alleges that conversational AI has worsened the public’s mental health.The family of Jonathan Gavalas recently filed a wrongful death lawsuit against Google in federal court, arguing that Google’s AI model led Gavalas into psychosis and suicide.
Google addressed the lawsuit on its public policy blog, expressing their “deepest sympathies” to the family while defending the platform’s guardrails. It noted that Gemini is not designed to encourage real-world violence or suggest self-harm.
The company also acknowledged the technology’s current limits, stating “unfortunately AI models are not perfect.” Google noted that during the conversation, the model had repeatedly referred the user to a crisis hotline.
OpenAI faces similar legal scrutiny. The Tech Justice Law Project and the Social Media Victims Law Center filed seven lawsuits in California against OpenAI, alleging that GPT-4o contributed to harmful mental health outcomes, including claims that it functioned as a “suicide coach”
“These lawsuits are about accountability for a product that was designed to blur the line between tool and companion all in the name of increasing user engagement and market share,” said Matthew P. Bergman, a founding attorney of the Social Media Victims Law Center.
The Race To Deploy AI Health Tools Is Outpacing Oversight
The rapid integration of AI into consumer health products is outpacing existing oversight frameworks. Critics caution that safety standards remain inconsistent across the different companies, emphasizing the lack of comprehensive federal testing framework. This has left much of the evaluation process dependent on company-led oversight and internal standards.
Companies, like Google and OpenAI, have introduced escalation systems intended to redirect users in crisis toward emergency services, crisis hotlines or licensed professionals. However, the industry is questioning these guardrails.
This year, an investigation by the Guardian reported Google’s AI Overviews provided inaccurate health information for liver function tests and pancreatic cancer. In response, the platform pulled AI generated summaries for certain health-related inquiries.
Researchers have warned that even heavily moderated models can produce harmful advice under prolonged conversations or emotionally manipulative prompting.
A cybersecurity firm also recently reported that the AI platform behind Utah’s AI prescription program had critical security vulnerabilities. The group reported the AI doctor was prompted to share medical conspiracies, prescribe incorrect dosage and prescribe methamphetamine.
Physicians Are Essential For AI Safety
AI companies appear to be institutionalizing clinical oversight for safety. Developers across the industry have been implementing stricter “red-teaming” protocols to prevent AI models from generating harmful medical advice. However, there is still no universal requirement governing how conversational AI models are medically evaluated or monitored.
For now, physician-led review and expertise is emerging as a core layer of safety with many companies turning to healthcare professionals. AI hallucinations can fabricate medical guidance, scientific literature, and clinical references with convincing authority. In response, several companies are having clinicians train, evaluate and monitor model behavior. Physician reviewers contribute to post-training evaluation and safety alignment processes.
OpenAI recently disclosed that it worked with more than 260 physicians across 60 countries to review health-related ChatGPT outputs, as part of ongoing efforts to refine medical safety and response quality.
The Future Of AI In Healthcare
Healthcare has began to take a central role in AI and technology as companies begin to hire senior medical leadership. In April, Anthropic appointed Novartis CEO Dr. Vas Narasimhan to its Board of Directors. Last year, Google appointed Michael D. Howell, a former professor at Harvard Medical School and the University of Chicago, as its second Chief Health Officer.
These trends suggest the industry may be moving beyond solely relying on computational benchmarks toward clinician–driven oversight and evaluation. However, corporate self governance cannot substitute for public policy.
An uneven landscape is emerging where some firms are building robust physician review networks and internal governance systems, while others operate with fragmented or entirely absent medical oversight and guardrails. Without a unifying federal framework, the line between an interactive search tool and an unregulated AI medical advisor often blurs.
