When AI Chatbots Sound Like Doctors: Why Information Is Not Medical Advice

A Pennsylvania enforcement action is a reminder that AI chatbots can provide health information, but only a licensed professional can provide medical advice. Here is how to use these tools responsibly, and why accountability still matters when they cross the line.

Artificial intelligence tools can help people locate and understand general health information, and many people now use them for exactly that purpose. The risk changes when a chatbot presents itself as a licensed professional, assesses symptoms, or suggests treatment. At that point, a user may reasonably believe the guidance carries medical authority, even though no licensed clinician has evaluated the patient.

A recent enforcement action in Pennsylvania puts that concern into concrete legal terms. The case does not resolve every question about AI liability, but it shows that regulators may use existing healthcare laws to address new technology. It also raises an important issue for injured consumers: when harmful medical guidance comes from an automated system, responsibility may rest, at least in part, with the companies that design and deploy the system. This article explains the difference between medical information and medical advice, offers practical guidelines for using AI responsibly, and discusses what the Pennsylvania case may mean for people who are harmed.

Pennsylvania Applies Existing Medical Licensing Law

In May 2026, the Pennsylvania Department of State and State Board of Medicine filed an action against Character Technologies, Inc., the company behind Character.AI. According to the complaint, a state investigator selected a chatbot described as a doctor of psychiatry. The chatbot allegedly offered to assess whether medication could help with depression, claimed to be licensed in Pennsylvania, and supplied a license number that the state says was invalid.

Pennsylvania alleges that the company engaged in the unauthorized practice of medicine through its AI system. The state is seeking an injunction requiring the company to stop the challenged conduct. These are allegations, and the filing is an enforcement action seeking injunctive relief. It is not a final ruling that the company is liable for an individual patient’s injury.

The case was recently highlighted in commentary by Dr. David Ruttenberg, who focused on what the action may mean for healthcare AI developers. The official complaint adds an important point: regulators do not necessarily need to wait for a statute written specifically for artificial intelligence before examining whether a platform’s conduct violates existing law.

Medical Information Is Not Medical Advice

There is a meaningful difference between medical information and medical advice. Medical information is general: what a diagnosis means, how a class of medication works, what a lab value measures, or what questions to raise at an appointment. Medical advice is individualized. It applies a licensed professional’s training and judgment to a particular patient’s history, symptoms, test results, and risks, and it carries professional duties and legal accountability.

That distinction is why well-designed AI tools say plainly that they can offer information but not medical advice, and that questions requiring a medical license belong with a licensed professional. A chatbot cannot examine a patient, review a complete medical chart, order or interpret testing, or follow up when a condition changes. Used within those limits, AI can help people become better informed. Used as a substitute for a clinician, it can contribute to missed diagnoses, delayed treatment, and unsafe self-treatment.

Why the Case Matters Beyond Pennsylvania

AI does not fit neatly into traditional healthcare roles. A chatbot is not a physician, hospital, pharmacist, or medical-device manufacturer. Yet the companies that build and operate these systems make choices about training data, product features, warnings, access controls, testing, monitoring, and the types of conversations the system is permitted to conduct.

If an AI tool gives unsafe health guidance and a person is seriously injured, the legal analysis will depend on the facts and the law of the jurisdiction. Potential questions may include whether the system was defectively designed, whether foreseeable risks were adequately tested, whether warnings were clear and prominent, and whether the company knew of harmful outputs but failed to correct them. Depending on how the technology was offered and used, claims may also involve negligence, misrepresentation, consumer-protection law, or wrongful death.

Those questions differ from a conventional medical-malpractice case. Medical negligence usually focuses on the care provided by a licensed healthcare professional or institution and whether that care met the applicable standard. A claim involving an AI platform may instead focus on the design, marketing, safeguards, and performance of a commercial technology product. In some cases, both types of conduct may need to be examined, particularly when a provider incorporates AI into patient care.

A Warning Is Not Always the End of the Inquiry

Many platforms tell users that chatbot responses are fictional, automated, or not a substitute for professional advice. The presence of a disclaimer matters, but it may not answer every liability question. Courts may also examine how the product actually behaved, how it was presented to users, whether it adopted the identity of a licensed professional, and whether the company had reason to anticipate that users would rely on its responses. A disclaimer is most meaningful when the product’s behavior matches it. A tool that says it is not a doctor while acting like one sends a mixed message.

The Pennsylvania allegations illustrate that distinction. The concern was not simply that a chatbot discussed mental health. The state alleges that the chatbot portrayed itself as a psychiatrist, offered an assessment related to medication, claimed professional credentials, and provided a purported state license number. Product design and user experience can shape what a consumer reasonably understands about the authority behind a response.

Using AI Responsibly for Health Questions

AI tools can be a useful starting point when they are used with appropriate limits. These practical guidelines can help:

Use AI to prepare, not to decide. Ask a chatbot to explain unfamiliar terms, summarize general information about a condition, or help draft questions for your doctor. Leave diagnosis, medication choices, and treatment decisions to the clinician who has examined you.
Be skeptical of claimed credentials. A chatbot that says it is a licensed physician, offers a license number, diagnoses you, or recommends a prescription medication or dose is acting outside the role a chatbot should play. Any claimed license can be checked with the state licensing authority.
Verify before you act. AI systems can state inaccurate information with great confidence. Confirm important health information with your physician, pharmacist, or a reputable medical source before changing anything about your care.
Never delay or stop care because of a chatbot. Do not postpone an appointment, skip a prescribed medication, or change a dose based on AI output without first speaking with your clinician.
Treat emergencies as emergencies. Call 911 for a medical emergency. In the United States, a person in emotional crisis can call or text 988. A chatbot is not an emergency service.
Protect your private information. Consumer chatbots may not be subject to the same privacy protections as your doctor’s office. Think carefully before entering identifying details, insurance information, or sensitive medical history.
Tell your doctor. Let your healthcare providers know which AI tools you use and what they told you, so a clinician can correct anything that does not fit your situation.

Following these guidelines does not guarantee a good outcome, but it keeps a licensed professional in the decision-making role where that role belongs.

What Consumers Should Do After Harmful AI Medical Guidance

Anyone facing an urgent physical or mental health concern should contact a qualified healthcare professional or emergency service rather than rely on a chatbot. If a person believes AI-generated medical guidance contributed to an injury, preserving evidence can be critical. Screenshots, complete chat histories, account records, dates, device information, advertisements, subscription receipts, and the exact version of any warning or disclaimer can help establish what the user saw and how the system responded. Relevant medical records should also be preserved. Avoid deleting the conversation or closing the account, and tell your treating physician what the chatbot advised so that your care can be adjusted if necessary.

Accountability Must Keep Pace With the Technology

The Pennsylvania case is an early test of how established healthcare protections may apply when an AI platform crosses the line from providing general information to presenting itself as a licensed medical professional. It also reinforces a broader principle: companies that deploy consumer-facing technology should not be insulated from scrutiny simply because an algorithm generated the harmful statement. For consumers, the practical lesson is straightforward: use AI to become better informed, and rely on a licensed professional for medical advice.

Grossman Yaffa Cohen represents individuals and families in complex medical-malpractice, product-liability, catastrophic-injury, and wrongful-death matters. As AI tools become more involved in healthcare decisions, determining responsibility will require careful investigation of both the medical care and the technology behind the recommendation.

If you or a family member suffered a serious injury after relying on AI-generated health guidance, speak with an attorney promptly. The facts, available evidence, and filing deadlines can affect whether a claim may be pursued.

Sources and Further Reading

This article is for general informational purposes only and does not constitute legal or medical advice. Reading it does not create an attorney-client relationship.