California passes first US law requiring safety measures for AI chatbots

(Patria) - California Governor Gavin Newsom signed the first law regulating artificial intelligence (AI) chatbots, defying White House pressure to leave such technology uncontrolled, AFP reported.

“We have seen some truly horrific and tragic examples of young people being harmed by unregulated technology, and we are not going to stand by while companies continue without necessary guardrails and accountability,” Newsom said yesterday after signing the bill.

This significant law requires chatbot operators to implement “critical” safety measures regarding interactions with artificial intelligence-based chatbots and provides individuals with the ability to sue if a failure to do so leads to tragedies, according to State Senator Steve Padilla, a Democrat who sponsored the bill.

The law comes after the discovery of suicides of teenagers who used chatbots before taking their own lives.

“The tech industry is motivated to attract the attention of young people and keep it at the expense of their real-world relationships,” Padilla said before the bill passed the state Senate.

Padilla referred to recent teen suicides, including that of fourteen-year-old son of mother Megan Garcia from Florida. Megan Garcia’s son, Sewell, fell in love with a “Game of Thrones”-inspired chatbot on the Character.AI platform, which allows users, many of whom are young, to interact with beloved characters as friends or lovers.

When Sewell struggled with suicidal thoughts, the chatbot encouraged him to “come home”.

Seconds later, Sewell shot himself with his father’s gun, according to a lawsuit Garcia filed against Character.AI.

“Today, California has ensured that a companion chatbot will not be able to talk to a child or vulnerable person about suicide, nor will a chatbot be able to help a person plan their own suicide,” Garcia said about the new law.

“Finally, there is a law that requires companies to protect their users who express suicidal ideas to chatbots.”

There are no national regulations in the United States aimed at curbing the risks of artificial intelligence, and the White House is trying to prevent individual states from establishing their own rules.