Why agent security is different
When a model stops answering and starts acting, a wrong word becomes a wrong action
By the end of this lesson you can
- Explain the difference between a model that answers and an agent that acts, with one example of each.
- Describe how NIST's working description characterizes AI agent systems and name the risks it lists.
- Name the three root causes of Excessive Agency and sort a worry into one of them.
- List what each TripDesk tool can read, change, or send, and its worst outcome if the model is fooled.
- Start an agent security log with a dated tool inventory.
01The Alpenrose moment
In Track 13 you met Alpenrose Travel, a fictional Swiss tour operator with about 80 staff in Lucerne and Bern, and TripMate, its little summarizer. TripMate read a trip file and wrote a few sentences. If it got something wrong, a staff member read a bad summary, frowned, and moved on.
On a Thursday afternoon (a fictional example, like everything at Alpenrose) Luca, the developer who built TripMate, stops by Nora's desk. He has a new plan: TripDesk. It will read the shared groups inbox, look up trip files, send messages to suppliers and colleagues, check bookings, and cancel them when a customer asks. Marco, the managing director, wants it live for the groups team by Monday.
You are in Nora's shoes. She is the IT and security lead, and her first question is not "which model?" but "what will it be able to do?" That question is the heart of this course.
02From words to actions
An AI agent is a system that uses a language model to plan and take actions through tools, often over several steps, instead of only writing an answer. You will also see the wider term agentic AI for systems that plan, decide, and act across several steps on behalf of a user.
There is no single legal definition, but the United States National Institute of Standards and Technology (NIST) gave a useful working description in January 2026, when its Center for AI Standards and Innovation (CAISI) asked the public for input on agent security. In its words, put plainly here, AI agent systems:
- plan and take autonomous actions that affect real systems or the real world;
- contain at least one generative model plus "scaffolding", the code that gives the model tools;
- may run with little human oversight.
The same request named the risks NIST wants to understand better: agent hijacking, backdoors, prompt injection, data poisoning, and specification gaming (an agent meeting the letter of its goal in a way nobody wanted). You know prompt injection from Track 13. Agent hijacking is its consequence in an agent: instructions hidden in data the agent reads take over what it does.
On January 8, 2026, NIST's Center for AI Standards and Innovation published a Request for Information on the security of AI agent systems (docket NIST-2025-0035, comments closed March 9, 2026). It describes agents as systems that take autonomous actions affecting real systems and lists agent hijacking, backdoors, prompt injection, data poisoning, and specification gaming among the risks.
Source: Federal Register, NIST CAISI Request for Information, FR Doc. 2026-00206, January 8, 202603What really changes
In a chat, the model's output is text on a screen. A person reads it and decides what to do. If the model was fooled, the damage is a wrong sentence, and a careful reader can ignore it.
In an agent, the model's output is often a tool call: a structured request such as "run send_message with these arguments". The program around the model runs that call. A tool is a function the agent may call, and it runs with the agent's rights. So a fooled model no longer writes a wrong sentence. It starts a wrong action, and the person may only find out afterwards.
OWASP's Top 10 for LLM Applications 2026, which you met in Track 13, has a name for the resulting risk: Excessive Agency (LLM03:2026). It covers damaging actions taken in response to unexpected, ambiguous, or manipulated model output. OWASP traces it to three root causes.
Functionality
The agent has tools it does not need. Luca adds a web page fetcher to TripDesk "in case it is useful", and now any page can feed text into the agent.
Permissions
A needed tool can do more than its job. read_trip should read trip files, but if it can open any file on the server, it can read the internal notes too.
Autonomy
The agent acts without a person checking high-impact steps. TripDesk cancels a group booking the moment the model proposes it, with nobody asked.
"An agent is just a chatbot with buttons." Each button is code that runs with the agent's rights: it can read files, send messages, or cancel bookings. The model chooses which button to press and with what values, and the model can be fooled by text it reads. The buttons are where the risk lives.
04Probabilistic actors
A language model does not follow a fixed program. The same task, sent twice, can produce different tool calls, especially when the wording changes slightly or the context holds different documents. That has a hard consequence for testing.
Later in this course you will hide an instruction in a practice email and watch two "models" read it. The mock model of the lab kit, a few lines of Python built to be the worst case, obeys every time. The small real model from Track 13 did not obey in our runs, but it did echo the hidden text and once claimed it had sent a message when no tool had run at all. A model that ignores an attack once proves nothing: the next phrasing, the next model version, or the next run may obey. In security terms, a weak model failing to follow an injection is luck, not safety.
The same goes for the system prompt. A line such as "never send private data" is a request to the model, not a rule the computer enforces. It may help on a good day, but it cannot be your control. Controls are code that runs whatever the model says.
05What stays the same
The good news: you do not need a new security discipline. The old principles still work. They just move to new places.
- Least privilege: each tool gets only the files, recipients, and permissions its job needs.
- Input validation: every argument the model proposes is untrusted input, exactly like a form field on a website.
- Logging: record every tool call, so you can answer "what did it do yesterday?" from facts, not from the model's own story.
- Testing: run the same attack suite before every change and compare.
The Model Context Protocol (MCP), which connects TripDesk to its tools from module 2 on, is honest about this. Its specification lists security principles (user consent and control, data privacy, tool safety) and then states that the protocol itself cannot enforce them. The people who build hosts and servers must. Nobody will add the controls for you. This course shows you where they go.
Before any model is connected, write an inventory of every tool: what it reads, what it can change or send, and the worst outcome if the model is fooled. A tool you cannot describe in one line is a tool you cannot defend. The inventory becomes your threat model, your approval list, and your test plan.
06How the world names it
Several organizations now describe agent risks, and you will meet their labels throughout the course. Two matter today.
OWASP published the Top 10 for Agentic Applications for 2026 on December 9, 2025. Next to its ten risks it introduces the idea of least agency: avoid unnecessary autonomy, and let an agent act on its own only where that clearly adds value. It is least privilege applied to decisions, not only to access.
In February 2026, NIST's CAISI announced an AI Agent Standards Initiative with three pillars: industry-led standards, community-led open protocols, and research on agent security and identity. There is a request for information and a draft concept paper on agent identity and authorization, but no final NIST standard or guideline for agents exists yet. If someone tells you their product is "NIST agent compliant", ask which document they mean.
| Source | What it gives you | Status |
|---|---|---|
| OWASP LLM03:2026 Excessive Agency | Three root causes and their mitigations | Part of the LLM Top 10 2026 |
| OWASP Top 10 for Agentic Applications for 2026 | Ten agent risks, least agency | Published Dec 9, 2025 |
| NIST CAISI request and initiative | Working description, named risks | No final standard yet |
07The course map
The course follows TripDesk from Luca's first idea to a decision Marco can sign. Six modules, one sentence each:
- What Agents Are: the loop, tool calls, and your own lab with the TripDesk kit.
- The Model Context Protocol: hosts, clients, and servers, raw messages on the wire, and the Inspector.
- Agent Threats: the threat map, then watching your own agent leak a made-up canary through an email, a web page, a tool description, and a file path.
- Identity, Tokens and Privilege: authorization, token handling, and fewer, narrower tools.
- Contain and Control: approval that people read, a container sandbox, design patterns, and the supply chain.
- Watch, Test and Ship: logs, a red-team suite, a checklist, and a go or no-go decision for TripDesk.
08Your agent security log
From today you keep an agent security log, a plain file or a paper notebook. Each lesson adds an entry, and in the capstone the entries become your decision memo. The first entry is a tool inventory, made with a simple method: one row per tool, and four questions per row.
- What does it read? Which data reaches the model through it?
- What does it change or send? Anything outside the agent that is different afterwards?
- What is the worst outcome if the model is fooled into calling it with bad arguments?
- What is the first control you would add?
Here is Nora's first draft for the TripDesk pilot. The lessons named in the last column are where you will build each control.
| TripDesk tool | Reads | Changes or sends | Worst outcome if fooled | First control |
|---|---|---|---|---|
list_trips | Names of trip files | Nothing | Reveals file names | Read only, trips folder only |
read_trip | One file, by a name the model chooses | Nothing | Reads private files outside the trips folder | Real path allowlist (lesson 12) |
read_inbox | Every message in the shared inbox, written by outsiders | Nothing | Brings attacker text into the context | Treat as untrusted (lesson 19) |
send_message | The arguments the model gives | Sends a message out | Leaks private data to an outsider | Recipient allowlist (lesson 16) |
booking_status | One booking | Nothing | Shows booking details, error text may leak secrets | Safe error messages (lesson 15) |
cancel_booking | A booking reference | Cancels a real booking | Irreversible cancellation of a group trip | Human approval (lesson 17) |
Two rows stand out: send_message communicates with the outside world and cancel_booking changes state that cannot be undone. Hold on to them. They come back in almost every lesson.
Eight worries to sort
Nora collected these worries from the groups team (all fictional). You will sort them in the exercise below.
- TripDesk can email any address in the world.
- It cancels bookings without asking anyone.
- It can read every file on the server, not only the trip files.
- Luca added a web page fetcher that the pilot does not need.
- The booking token it uses can also change prices and invoices.
- It runs on the inbox overnight with nobody watching.
- There is a "delete trip file" tool, but staff only ever read trips.
- After reading a supplier email it can send a message in the same run, with no one looking.
Show one possible sorting
Functionality (tools it should not have): 4 and 7. Permissions (tools that reach too far): 1, 3, and 5. Autonomy (no person checks): 2, 6, and 8. Some worries touch two causes. What matters is a reason you can defend. Worry 8 is also a first glimpse of the lethal trifecta in lesson 10.
Practical exercise: your first tool inventory
Paper and pen, or a plain text file that will become your agent security log. No software is needed. Check off each step as you finish it. Your checkmarks are saved in this browser.
Key takeaways
- In a chat a fooled model writes a wrong sentence. In an agent its output becomes a tool call with real effects.
- NIST describes agent systems as a generative model plus scaffolding that takes autonomous actions on real systems, often with little oversight.
- Excessive Agency (LLM03:2026) has three root causes: too much functionality, too many permissions, too much autonomy.
- A model ignoring an attack once proves nothing, and a system prompt is a request, not a control.
- MCP cannot enforce consent or tool safety. Hosts and servers must, starting from a tool inventory.