As the public increasingly turns to artificial intelligence for health information and emotional support, the line between what constitutes a digital tool and a substitute for professional health care has the potential to become dangerously blurred. According to an article published on Microsoft AI’s platform in March of this year, Microsoft receives over fifty million health-related questions daily across its AI consumer products, which include Bing and Microsoft Copilot.[1] Another popular AI platform, ChatGPT, receives queries from more than forty million Americans daily related to healthcare.[2] On ChatGPT, one out of four of the platform’s eight hundred million users submits a question about healthcare weekly. [3]
A new bill, NC S624, aims to regulate the use of AI in healthcare in North Carolina.[4] The bill was introduced in March of 2025, and as of March 26, 2026, is now “in committee”[5] and being referred to the Committee of Rules and Operations of the Senate.[6] This means the bill is being debated by committee members, who will then decide if the bill should proceed to a full legislative vote.[7]
Senate Bill 624 has a broad scope. It would require any AI chatbot that “deals substantially with health information” to obtain a license from the North Carolina Department of Justice before it may operate in the state.[8] To obtain that license, developers must submit documentation of their platform’s technical architecture, data security measures, and privacy practices.[9] The bill also establishes a statutory “duty of loyalty,” mandating that chatbots prioritize users’ best interests.[10] Under this proposed “duty of loyalty,” AI healthcare platforms would include measures that limit patients’ emotional dependence on those platforms. [11]
One of the primary initiatives of Senate Bill 624 is a new mandate that would require healthcare providers to inform patients when those providers use AI to determine whether a particular service or treatment is necessary.[12] This transparency provision could be designed to make patients aware of the role that artificial intelligence plays in their medical care, in an effort to safeguard informed patient consent.
In a recent FOX Carolina news story, one of the bill’s proponents, State Senator Jay Luneau stated,
“Let’s pump the brakes just a little bit. Let’s slow down a little bit and let’s look at these things. It’s one thing if you use AI to determine if something is covered under a car warranty. It’s another thing to determine if you need a heart procedure done or not. So, hopefully, we’re gonna apply this in the right way.”[13]
These remarks reflect a growing bipartisan recognition that North Carolina should take legislative steps to regulate the use of AI technology to deliver medical care to patients.
Even with these safeguards, AI use for healthcare in North Carolina still lacks many of the significant safeguards that apply to in-person medical practitioners. For example, a recent New York Times article about AI use and mental healthcare included remarks from Sam Altman, the founder of ChatGPT, describing how many individuals divulge personal information to chatbots, in a relationship that mimics the client-therapist relationship.[14] Many states have laws that regulate when a therapist must report sensitive information, such as threats to third parties, that clients may make to them (referred to as “the duty to warn”).[15] AI chatbots currently hold no equivalent duty.[16] Patients who may be hesitant to disclose plans of self-harm or harm to others to a therapist out of fear of triggering mandatory reporting may feel comfortable sharing such plans with a chatbot under the belief that these plans will remain private.[17]
Under North Carolina law, mental health professionals who have reason to believe a patient poses a threat to themselves or others do not have a duty to act to protect their patients or identifiable third parties.[18] (North Carolina is one of four states that lack such a duty to warn.[19]) However, the North Carolina Court of Appeals has recognized that a common-law duty to act may arise in certain limited circumstances based on general negligence principles.[20] While the new bill includes safeguards to attempt to protect patients from becoming overly reliant on AI for health information, it still leaves open the question of what should happen when patients do heavily rely on AI for healthcare, or how a platform should respond when patients input sensitive personal information or information about plans to harm third parties.
Senate Bill 624 represents North Carolina state legislators’ effort to increase accountability of AI’s role in healthcare in the state. The bill’s proposed duty of loyalty provision may help patients to understand when an AI platform is utilized to make decisions in their treatment. However, the bill stops short of fully addressing the potential dangers of AI use in healthcare. Additional legislation will be needed in the future to continue to direct how the public can interact with AI to produce the safest possible outcomes.
[1] Pavel Tolmachev et al., Health Check: How People Use Copilot for Health, Microsoft AI (Mar. 10, 2026), https://microsoft.ai/news/health-check-how-people-use-copilot-for-health/.
[2] Naomi Diaz, 40 Million Americans Use ChatGPT for Healthcare: Report, Becker’s Hosp. Rev. (Jan. 5, 2026), https://www.beckershospitalreview.com/healthcare-information-technology/ai/40m-americans-turn-to-chatgpt-for-healthcare-report/.
[3] Id.
[4] See generally New Bill Targets AI Health Insurance Coverage, FOX Carolina (Mar. 17, 2026), https://www.foxcarolina.com/video/2026/03/17/new-bill-targets-ai-health-insurance-coverage/.
[5] Bill Detail 1876091, Bill Track 50, https://www.billtrack50.com/billdetail/1876091.
[6] AI Chatbots – Licensing/Safety/Privacy, UNC Sch. of Gov’t, https://lrs.sog.unc.edu/bill/ai-chatbots-licensingsafetyprivacy.
[7] Standing Committees, N.C. Gen. Assemb., https://sites.ncleg.gov/lad/standing-committees/.
[8] S. 624, 2025–2026 Gen. Assemb., Reg. Sess. (N.C. 2025) (proposed) (AI Chatbots – Licensing/Safety/Privacy).
[9] Id.
[10] Id.
[11] Id.
[12] Id.
[13] FOX Carolina, New Bill Targets AI Health Insurance Coverage, at 0:23 (FOX Carolina, Mar. 17, 2026), https://www.foxcarolina.com/video/2026/03/17/new-bill-targets-ai-health-insurance-coverage/.
[14] Kashmir Hill, When Chatbots Are Used to Plan Violence, Is There a Duty to Warn?, N.Y. Times (Feb. 26, 2026), https://www.nytimes.com/2026/02/26/technology/chatbots-duty-warn-police.html.
[15] Id.
[16] Id.
[17] See generally id.
[18] Scott R. Polick et al., Clarifying the ‘Duty to Warn’ in North Carolina, Medpage Today (Sept. 18, 2023), https://www.medpagetoday.com/opinion/second-opinions/106381.
[19] Id.
[20] See Gregory v. Kilbride, 136 N.C. App. 482, 525 S.E.2d 487 (2000) (holding that while North Carolina does not recognize a psychiatrist’s duty to warn third persons, an independent duty to protect third persons from harm may arise under general negligence principles when a psychiatrist exercises control over a patient).
