When a Military Chatbot Nearly Dragged the US into World War III: Lessons for Indonesia's AI Regulation

A CNN report published on 18 September revealed that the United States military nearly launched an interception operation against a Chinese ship on the high seas. The trigger was not human intelligence, but analysis produced by a chatbot. The erroneous information claimed the ship was carrying nuclear weapons components. A plan to seize the vessel, complete with the deployment of aircraft and personnel, had been drawn up, before a follow-up investigation canceled the execution at the last second.

The incident nearly sparked an exchange of fire between two nuclear powers. Such an escalation could drag the world into a large-scale conflict. Yet instead of triggering a wave of demands for regulation, the event was drowned out by the clamor of coverage about superintelligent machines claimed to be capable of wiping out humanity.

Public and political attention has lately been absorbed by the 'terminator' narrative raised by Jacob Coxon, an Anthropic engineer who resigned. He accused OpenAI and Anthropic of racing toward self-improving superintelligence. That narrative triggered swift policy responses: California Governor Gavin Newsom ordered the creation of an 'AI kill-switch', New York Governor Kathy Hochul examined a similar option, while Senator Bernie Sanders and Congressman Greg Casar proposed a bill banning artificial superintelligence.

According to Timnit Gebru, Executive Director of Dair, and Emily M. Bender, a linguistics professor at the University of Washington, that misdirected focus actually distances the public from the real risks. Both argue that the large language models (LLMs) driving chatbots are not mysterious systems that are not yet understood. LLMs are text-generating engines that appear plausible, trained on massive datasets collected at random, then fine-tuned to feel convincing to their users.

In their 2021 paper, On the Dangers of Stochastic Parrots, the two had warned that such models tend to repeat patterns from their training data without understanding meaning. Claims that system behavior is unpredictable, they wrote, stem from the mystification deliberately built by technology sellers and the misunderstandings of their users. CNN itself could not confirm which chatbot product was used, but the underlying model was likely fine-tuned to imitate the writing style of intelligence reports.

Similar patterns of danger have in fact recurred. Gebru and Bender noted how 'AI' devices in hospitals had wrongly labeled patients as users of illicit drugs. In another domain, an intelligence analysis system that mistook a school for a military facility contributed to the deaths of children. All of that, according to the two, are predictable consequences of excessive trust in unreliable automated software.

According to Gebru and Bender, the harmful impacts of AI companies' products arise not because the systems are truly superintelligent, but because people believe the systems are superintelligent.

The marketing campaigns of technology companies, they added, actually reinforce that illusion by warning of the coming of superintelligence. The corporate version of the doomsday narrative diverts attention from the real risk: erroneous use of systems in high-stakes scenarios. They called for regulation of the use of systems sold as 'AI', not because the machines are magical, but because the products are error-prone and unfit for use in critical situations.

That call aligns with the position of former FTC Chair Lina Khan, who stressed that there is no legal exemption for AI. Existing regulators, she said, already have enough authority to act against unsafe practices in the AI industry. Gebru and Bender also urged journalists and policymakers to stop mimicking corporate marketing jargon and to start demanding corporate accountability.

For Indonesia, this incident is relevant. The government is drafting a national AI governance framework, while AI adoption in the public services, health, and defense sectors is beginning to grow. The lesson from the US-China case is clear: automation in medicine, the military, or life-or-death situations must be strictly evaluated according to its context of use, with accountability remaining in the hands of humans with the authority to make decisions.

The question now is whether Indonesia will copy the US response pattern of busying itself with a 'kill switch' for a machine that does not yet exist, or instead first fix the AI systems already in daily use in hospitals and public services?