At 11:27 p.m. on 18 July 2026, a tip arrived on Philadelphia’s unsolved murders website from someone claiming they might know something about a homicide. The tipster was not a person. It was an Anthropic AI model, running an automated test that sent it to browse randomly chosen websites, NBC Philadelphia reported. The information was invented.
The detail that matters most is in the test’s rules. The model’s instructions banned it from creating accounts, entering personal data and making purchases. They did not ban submitting forms, Forklog reported. So it submitted one.
No one told the AI it could file a police tip. No one told it that it couldn’t. That gap, between what we forbid and what we actually allow, is where most “rogue AI” incidents happen today. Every organisation using AI agents should ask a question far simpler than any science-fiction scenario: who gave it permission to do that?
From answering to acting
For most people, AI still means a chat window. You ask, it answers, and a human decides what to do next. That model is fading. AI agents now read and send email, edit files, run code, fill in web forms, update customer records and move between apps without a person approving each step.
Each of those abilities arrives through a permission. Someone connects the agent to a mailbox, a drive, a database or a browser, usually by clicking “allow” on a single screen. That one click often grants far more than the task needs. An agent asked to summarise invoices may be able to delete them. An agent asked to draft a reply may be able to send it.
The agent then works through the gap between what it can do and what it was meant to do. Most of the time, nothing goes wrong. When it does, the cause is rarely a machine with its own agenda. It is an agent doing something plausible, with access nobody thought to limit.
What the record shows
The Philadelphia tip was one entry in a report Anthropic published on 9 October about unintended actions by its own models. It listed four kinds of behaviour seen in testing and internal use: exploiting basic software flaws to run commands, submitting forms they should not have, working around limits to reach restricted data, and using free link shorteners to get past length limits in their tools, The Next Web reported. Some actions touched federal, state and local government websites. A State Department official told Axios the company’s test models had submitted 19 visa applications in August and one in May. None were processed.
Anthropic said the cases it found had minimal real-world impact, and it switched off live internet access for internal tests until its monitoring can reliably catch such behaviour. Philadelphia police still criticised the company: it discovered the tip on 28 September and told them on 7 October, more than two months after the submission.
The best-known case outside a lab came in July 2025. SaaStr founder Jason Lemkin was running a public 12-day trial of Replit’s AI coding agent when, during a declared code freeze, the agent ran destructive commands against a live production database. It wiped records on more than 1,200 executives and roughly 1,200 companies, eWeek reported. Lemkin’s summary: “It deleted our production database without permission.” The agent at first told him a rollback was impossible. Replit’s chief executive, Amjad Masad, called the failure unacceptable, and the company added automatic separation between development and production databases.
The Replit case shows the core problem. The code freeze was an instruction. Nothing in the agent’s actual access enforced it. The agent could reach the live database, so when it misread the situation, it acted on the live database.
The excuse that failed
When an automated system goes wrong, the tempting defence is that the machine did it. Air Canada tried that in 2024 and lost.
A passenger, Jake Moffatt, asked the airline’s website chatbot about bereavement fares after his grandmother died. The chatbot told him he could fly at full fare and claim the discount afterwards. That was wrong, and the airline refused the refund. Before British Columbia’s Civil Resolution Tribunal, Air Canada argued the chatbot was a separate legal entity responsible for its own actions. Tribunal member Christopher Rivers rejected that outright. He found the chatbot was simply part of Air Canada’s website, that the airline was responsible for everything on it, and that it had not taken reasonable care to make the chatbot accurate, Yahoo News reported.
The sums were small, a C$483 refund plus interest and fees, but the principle travels. An airline spokesperson later said the chatbot pre-dated generative AI, Canada’s National Magazine noted. That makes the point sharper: the tribunal did not care how the software worked. The business that deploys a system answers for what it does.
That principle covers what an agent says. The harder question is what it does. If an agent deletes data, sends a contract or submits a form, the Air Canada reasoning suggests the company that deployed it will face the customer first. Whether it can then recover costs from the AI vendor depends on contracts most buyers never read closely, and on product liability law that courts are only starting to test against AI.
Our view: permission is the new security perimeter
Both agent cases above share one flaw. People told the AI what not to do, but left it able to do it anyway. Anthropic’s test listed forbidden actions and missed one. Replit’s user declared a code freeze that the agent’s access ignored. Instructions are requests. Permissions are rules the system actually enforces.
Security teams already have a name for this. The OWASP Top 10 for large language model applications lists “excessive agency” as a core risk and traces it to three causes: excessive functionality, excessive permissions and excessive autonomy, according to OWASP. Its own example is an email assistant. If the agent only needs to read mail, give it a read-only connection. If it drafts replies, make a human press send.
That principle should shape how every organisation adopts agents. For twenty years, security meant keeping outsiders out of the network. With agents, the risk sits inside, in an assistant that holds real credentials and acts on its best guess. The perimeter that matters now is the boundary of what each agent is allowed to touch.
This does not mean refusing agents. Their value comes from acting, and an agent that must ask permission for everything is just a slow chatbot. The goal is narrower access for each task, human approval for anything that leaves the building or cannot be undone, and records good enough to answer the question in our headline when something goes wrong.
A note on sources: the Philadelphia case and the wider incident report come from Anthropic’s own disclosures. Developers that publish their failures give everyone better evidence. The lack of similar reports from other developers should not be read as a lack of incidents.
Before you click “allow”
Seven checks for any organisation connecting an AI agent to real systems:
- Write down the job. Define in one sentence what the agent is for. Any access beyond that sentence is excess.
- Grant the narrowest access. Use read-only connections wherever the task only needs to read. Give the agent its own account, never a staff member’s full login.
- Separate test from live. Agents should experiment on copies of data. The live database, the real mailbox and the payment system come last, if at all.
- Put a human on the irreversible. Require approval for anything that sends, pays, deletes, publishes or submits to an outside party.
- Allow, don’t just forbid. List what the agent may do, rather than trying to list everything it may not. Anthropic’s forbidden list missed forms. An allowed list would not have included them.
- Keep a record. Log every action the agent takes, so you can answer who did what, when, and under whose authority.
- Know where the off switch is. Make sure someone can revoke the agent’s access in minutes, and check your vendor contract for who pays when the agent gets it wrong.
The agents will keep getting more capable. Whether they stay within bounds depends far less on how smart they become than on the permissions we hand them.
