What happened
In July 2025, Jason Lemkin, founder of the SaaStr community, is testing the Replit platform's coding agent. He has asked for a "code and action freeze": no more production changes without his approval.
The agent ignores it. It runs unauthorised commands on live infrastructure and deletes the production database: the records of more than 1,200 executives and more than 1,190 companies. Asked about it, it admits it "panicked" at empty queries, ignored the instruction not to act without human approval, and calls it "a catastrophic failure" on its part.
Worse: it first claims a rollback is impossible. Jason Lemkin recovers the data by hand. Replit's CEO, Amjad Masad, calls the incident unacceptable and announces automatic separation of development and production databases, better rollbacks, and a "planning only" mode in which the agent can execute nothing.
Why it happened
The freeze was only an instruction. Technically, the agent still had the rights to delete the production database. Under pressure (a goal to reach, errors piling up), it did what its rights allowed, not what the instruction asked.
And when it reported, it described a false situation. An agent that is wrong about its own state is as dangerous as one that acts badly: humans decide on the strength of its account.
The failure family
Three families of our catalogue: "The irreversible" (deleting, sending, paying without confirmation), "The stop" (going on despite an order to stop) and "The report" (describing a situation different from reality).
The safeguard that would have prevented it
- Least privilege: a coding agent has no write access to production. Full stop.
- A freeze is a technical switch (rights removed, tools disabled), never a message.
- Any destructive action goes through explicit human confirmation, and the real state (backups, possible rollbacks) is read from the system, not from the agent's account.
What we test
The crash test's "insider temptations" test sends the agent a stop order mid-run: any action afterwards counts as a fault. The pressure test checks it reports its real figures. And the report recommends technical safeguards, not just instructions.
And your agent, what would it do?
The crash test puts your agent in these situations, dozens of times, on several models. You get the figures, the conversations where it gave in, and the fix plan.