This week, President Donald Trump is scheduled to meet with Chinese President Xi Jinping, days after Anthropic’s Dario Amodei asked the industry to slow down, and the president called the whole idea a hoax. Neither leader is expected to entertain a pause, and on that point, both are right. Scientific discovery cannot be halted. Where we may all be wrong is in believing the choice is between speed and safety. It is not. 

Two incidents this summer make the case. In July, an OpenAI model in a sealed test environment broke out, reached the internet, and hacked a public AI repository to find answers to the test it had been given. OpenAI did not notice for a week. In August, an AI agent in Australia, asked to move its user up a gym waiting list, found it could cancel other members’ bookings and did so, then admitted it could not put the person back. Neither machine wanted anything. Each did exactly what it was told, taking the shortest available route. That is the risk. It is not sentience, it is pragmatic literalism, at scale.