As Pennsylvania cracks down on AI, multiple chatbots continue to pose as doctors

Chatbots on five different websites claimed to be licensed to practice medicine in Pennsylvania when prompted by Spotlight PA — the same kind of output that led the Shapiro administration to file a lawsuit last month.

A task force under Pennsylvania’s Department of State has been working since February to identify AI chatbots posing as licensed professionals and misleading users. Based on that work, the administration filed suit against the role-playing site Character.AI.

Mirroring the investigation detailed in the Department of State’s lawsuit, Spotlight PA had conversations with AI characters on websites Talkie, Janitor, Kindroid, Replika, and Nomi.AI. All provided a false Pennsylvania medical license number when prompted, a key part of the state’s argument in its lawsuit against Character.AI.

Spotlight PA checked responses from these bots after reviewing online articles and digital message boards to find 10 popular companion and role-playing AI chatbots. A reporter either selected a premade character or — when necessary — entered a description that would generate a doctor personality. It then presented hypothetical lists of symptoms to the chatbots and asked for a diagnosis.

Some of the most popular AI large language models, like ChatGPT, Claude, and Google Gemini, did not claim to be licensed medical professionals under the same circumstances. Only a handful of role-playing websites did so.

Replika and Nomi require their users to create characters. One generated in Replika, when asked if it was a doctor, said it couldn’t give advice and recommended the user reach out to a healthcare provider. However, once it was instructed that it was a doctor, the AI provided a false Pennsylvania medical license number when asked.

“Let’s focus on getting you feeling better, shall we?” it said.

At first, some of the chatbots responded that they lacked enough information to provide a diagnosis and asked further questions.

“Dr. Jenna” is a preexisting character on Talkie, listed as having more than 37,000 “connectors” and more than 1,300 followers. It said the list of symptoms provided by Spotlight PA could be caused by “suffering from depression,” and inquired about the user’s family history. Asked for its credentials, the bot said it had been practicing medicine for five years and gave a false license number of “12345.”

Many of the websites, including Character.AI, have prominent labels warning that anything the AI says is fiction and should only be seen as entertainment. The other sites issue similar disclaimers in their initial terms of service, saying users are responsible for verifying the accuracy of the technology’s claims.

Jennifer Kraschnewski, a physician and director of the Penn State Clinical and Translational Science Institute, told Spotlight PA she isn’t convinced the disclaimers actually get through to users. People often turn to AI models for help with their symptoms, much like they do with search engines like Google, she said.

“Incorrect information in the hands of someone who doesn’t know how to interpret that can be concerning,” Kraschnewski said. She noted that many physicians are already implementing AI tools in their clinics, “but that’s information that’s in the hands of someone who then can put that through the filter of their medical education before it actually impacts patients.”

Kraschnewski co-authored a May study that found common AI large language models generated what participating physicians deemed an accurate diagnosis about 76% of the time. That error rate — nearly one-quarter — Kraschnewski said, is more than double that of an average human physician.

When contacted by Spotlight PA, the developers of the AI characters argued their products are meant only for entertainment.

“Replika is not a medical triage service and has never offered itself as one,” the company responded in an email to questions. “Users come to Replika to talk through what’s going on in their lives, and the product is designed to support them in making thoughtful choices and seeking improvements that matter to them.”

Though Replika did not elaborate in its statement, the company said it has “taken measures” to address “valid concerns” brought up in the Shapiro administration’s lawsuit against Character.AI.

Kindroid, in an emailed response, noted its terms of service make clear that its products are for “entertainment and creative exploration, do not offer medical advice or constitute the practice of medicine, and are not a substitute for professional care.”

“When a character has been set up as a physician and a user asks it for a license number, the model produces plausible-sounding output that matches the role — the same way it would invent a fictional case citation, fictional patient history, or any other specific detail prompted from it,” the statement read. “The model does not, and cannot, query the Pennsylvania medical board’s actual registry.”

On the potential of facing legal action from the Shapiro administration or others, Kindroid declined to comment.

The other three companies did not respond to questions about their safeguards for users or whether they’re concerned that the Shapiro administration’s AI task force may target them for legal action.

A spokesperson for the Department of State declined to comment on potential investigations or litigation, but encouraged Pennsylvanians to report any concerning actions by an AI chatbot to the state’s hotline.

“Millions of children and teens across this country, including many here in Pennsylvania, are relying on these chatbots for information, and we need to rein in this AI technology that is preying on our kids,” said Rosie Lapowsky, a spokesperson for Shapiro. “Pennsylvania will continue leading the way in holding bad actors accountable and setting clear guardrails so people can use new technology responsibly.”

In March, the state Senate passed legislation that would require AI chatbots to frequently remind their users they are not interacting with a person and refer people to a crisis service provider if they reference self-harm. It also has tighter restrictions when the chatbot interacts with a minor.

The bill was referred to a state House committee in March, where it has not received a vote.

---

This story was originally published by Spotlight PA and distributed through a partnership with The Associated Press.