On October 5th, the New York City Council held a full committee hearing on artificial intelligence (AI) security. Representatives from four companies – OpenAI, Anthropic, Google, and Meta – took an oath and faced questioning from council members.
The council members demanded that the companies explain how they assess the risk of AI running out of control, who would be held responsible if the system causes harm, and whether they are willing to commit to not releasing AI models that have not passed safety tests.
All four companies emphasized the importance of AI security, but they did not provide direct or positive answers to specific questions regarding risk assessments, legal responsibilities, and third-party security verifications.
Morgan Dwyer, the head of policy development and operations at OpenAI, testified on behalf of the company. Council Chair Julie Menin asked her to estimate the likelihood of catastrophic consequences caused by AI. Dwyer stated that she couldn’t give a specific percentage, but emphasized that any level of risk, whether 1%, 10%, or 20%, is unacceptable. She mentioned that OpenAI should not train models that the company cannot prove can be kept under human control.
When further pressed by the chair about whether OpenAI would take legal responsibility if its AI system went out of control and led to significant financial losses, data breaches, injuries, or deaths, Dwyer did not provide a direct answer.
Regarding the question of whether models not passing internal security tests or independent third-party verifications would not be released, Dwyer mentioned that OpenAI would not release models deemed unsafe by the company. While they had delayed product releases due to safety concerns in the past, she did not commit to a blanket statement of not releasing models that fail any test.
Anthropic sent Logan Graham, the head of its advanced AI security testing team, to testify. He explained Anthropic’s approach to assessing AI model security, including cybersecurity and risks of “losing control,” but did not give specific probabilities of catastrophic AI consequences.
Graham also mentioned that in the past, Anthropic had discovered strong software vulnerabilities in models during testing and decided not to release them publicly, instead restricting their usage. He stated that the company would determine whether a model could be released based on security assessment results.
Alice Friend, Google’s head of AI and emerging technology policy, mentioned that there is currently no rigorous scientific method to provide a reliable percentage for the likelihood of catastrophic AI events in the future.
Regarding legal responsibilities, Friend’s response was more direct. She stated that Google believes existing legal frameworks apply to AI as well and affirmed that if an action was illegal without AI, it would remain illegal with AI. She added that Google will continue to cooperate with the government to establish appropriate safety regulations for AI development.
Shane Cahill, Meta’s head of AI policy and legislation, expressed reluctance to provide an imprecise risk number without sufficient basis. When asked about whether Meta would take legal responsibility if its AI system caused significant harm, Cahill stated that he was not a legal representative of Meta and therefore could not speculate on specific legal responsibilities.
None of the four companies gave identical responses regarding the “third-party verification” proposed by the city council.
The council proposed a measure requiring some AI models to undergo independent third-party verification and include the ability to manually shut down the system before being used in New York City. None of the corporate representatives committed to not releasing models that had not passed internal or third-party security tests.
The hearing discussed a series of AI regulatory measures under consideration by the city council, including third-party security verification, manual emergency shutdown mechanisms, the rights of AI harm victims, and whistleblower incentives to encourage employees to report security issues within companies.
The council stated that they aim to pass legislation to increase safety measures and corporate responsibilities as AI rapidly develops. Reportedly, this is the first city council in the U.S. to hold such a hearing.