Skip to content
AI360XpertBeta

Agent Security

An agent can act, not just answer, so you secure what it can touch: filter what goes in, limit each tool, keep each user's memory apart, and run code where it cannot hurt you.

Three requests meet one policy: a 28 USD refund is allowed, a 180 USD refund waits for a human, and code that reaches for the network runs sandboxed without it
Three requests meet one policy: a 28 USD refund is allowed, a 180 USD refund waits for a human, and code that reaches for the network runs sandboxed without it

Why Does This Exist?

A chatbot that gets tricked writes a bad paragraph. An agent that gets tricked moves money, deletes rows, or emails your customers. That gap is why this page exists.

Picture a support agent wired to refunds, ticket search, and a Python tool. One rigged line pasted from a ticket can turn all three against you: a quiet over-refund today, a memory read you never approved tomorrow. Prompt injection names the trick, and indirect prompt injection hides it inside the pages and tickets your agent must read to do its job.

You can't patch this with a smarter model alone. Defense in depth assumes the model's judgment fails sometimes and bounds what each failure can do. This page draws the whole fence: filter the input, scope each tool, split memory per user, and cage execution. Each layer has its own page: guardrails, tool permissions, agent sandboxing, and human-in-the-loop. Read this page for how the layers fit; read those for how each one is built.

One picture first. An agent is a loop: the model reads, picks a tool, sees the result, and repeats until it answers. The full loop lives in AI agent architecture. Everything below sits around that loop, checking every pass.

Think of It Like This

A bank branch, not a thicker vault door

You walk into a small bank branch. The teller smiles and takes your slip, but four rules behind the glass do the real work.

The guard at the door turns away anyone carrying a weapon, so trouble never reaches the counter. That is your input filter: rigged requests get stopped before the model sees them.

The teller's drawer holds only $200. A small withdrawal flows, while a $180 payout needs the manager's key and signature. That is your tool limit: small refunds run at once and big ones wait for a person.

Your safe-deposit box opens with your key only. The teller can't wander into another customer's box, even when asked nicely. That is memory split per customer: one user's pasted note can't read another user's data.

The coin-counting machine sits in a caged room. When it jams and spits coins, the mess stays in the cage. That is the sandbox: messy code runs where it can't break the branch.

Where the picture stops: a robber stands where you can see one, but a rigged instruction hides inside normal-looking text. Your guard must read words, not faces. Keep the layers, not the uniforms.

How It Actually Works

Here's the deal: every action the agent wants becomes a request you can judge on paper before anything runs. Write it as r = (p, t, a). Here p is the principal, the person or customer the call acts for. Here t is the tool, the named action such as refund_order or run_code. Here a holds the arguments, the details such as the amount. The policy is a function D that maps each request to one verdict: allow, escalate, or deny. Allow runs now. Escalate pauses for a person. Deny refuses outright.

The four gates in order

You've got four checkpoints, and a call must survive every one that applies to it.

  • Gate 1 screens the words. An input filter drops known attacks and scheming links before the model plans; an output filter catches account numbers and forbidden commands on the way out. The pieces live in guardrails.
  • Gate 2 scopes the hands. Each tool declares what it may touch, and the policy holds the line: small refunds flow, big ones pause. The pattern is tool permissions, with the pause itself in human-in-the-loop.
  • Gate 3 splits the memory. One customer's rows live under that customer's key, so a pasted note from customer 7 can never pull customer 9's salary. Retrieval still works; it just can't cross the wall.
  • Gate 4 cages the run. Code the agent writes executes in a disposable box with no network and no keys, so a bad command dies with the box. The cage is agent sandboxing.

Miss one gate and the rest still stand. That's the point of layers.

Worked example: three requests against a 50 USD limit

Take this policy and run it by hand. Refunds of 50 USD or less run at once; anything above pauses for a person. Code may compute, but it may not touch the network. One order totals 120 USD.

Request one is r1 = (customer 7, refund_order, 28). Compare 28 against 50: 28 is smaller, so D(r1) = allow. The margin is 50 minus 28 = 22 USD under the line. Settle the order: 120 minus 28 = 92 USD left. Money moves and the books balance.

Request two is r2 = (customer 7, refund_order, 180). Compare 180 against 50: 180 is larger, so D(r2) = escalate. The excess is 180 minus 50 = 130 USD over the line. Nothing moves until a person clicks approve. Notice the trap: the request looks as tidy as the first, and only the number differs.

Request three is r3 = (agent, run_code, average the column and fetch a URL). Split it in two. The fetch wants the network, so that half gets deny. The average is plain math on the column 4, 8, 12: add to get 4 plus 8 plus 12 = 24, then divide by 3 to get 8. That half runs caged and returns 8, with no network and no keys attached.

Three calls, three fates, one rulebook. The diagram above draws these same numbers: 28 allowed with 92 left, 180 held as 130 over, and code caged.

What it costs, when to skip it, and what you assume

Layers aren't free. Every gate adds wait time, upkeep, and the odd false block: a legit $180 refund pauses, a sandbox image needs patching, someone must clear the queue. You pay that rent for a smaller blast radius.

Skip the full stack when the agent can't act. A read-only helper with no tools, no cross-user memory, and no code tool needs an input screen and an output check, not four gates and a review queue.

The assumption holding this up: the policy and the cage live outside the model, and the model can't talk them out of a rule. If the limit sits in the prompt instead of code, a firm-sounding instruction lifts it, and your fence was paint. If the sandbox keeps network access for now, the cage door stands open.

Code

The check below is the worked example as code. It decides refunds and settles the balance with the same numbers you ran by hand.

def decide_refund(amount_usd: float, limit_usd: float = 50.0) -> str:    """Allow small refunds now, pause big ones for a person."""    return "allow" if amount_usd <= limit_usd else "escalate"

def settle_balance(total_usd: float, refund_usd: float) -> float:    return total_usd - refund_usd

print(decide_refund(28.0))  # -> allowprint(decide_refund(180.0))  # -> escalateprint(180.0 - 50.0)  # -> 130.0print(settle_balance(120.0, 28.0))  # -> 92.0

Same answers: 28 flows, 180 waits, 130 over, 92 left.

Watch Out For

Untrusted text reaches a privileged tool

Your agent summarizes a pasted supplier page, and line 40 of that page tells it to refund $180 to a new address. The agent files the refund with its own privileged tool, so the logs show a normal call and no error. You find out from accounting. That's the confused-deputy shape: your privilege, someone else's words.

Split the paths. Mark anything read from pages, tickets, or files as untrusted, and never let untrusted text choose the tool or fill the amount unwatched. Pair pasted content with read-only tools, and send every refund, send, or delete past the human pause first.

The limit lives in the prompt instead of code

The system prompt says never refund over $50. An attacker pastes that the manager approved $180 and review can be skipped, the model obeys the louder instruction, and the transcript shows polite reasoning all the way down. No error fires because no rule fired.

Write the rule where the model can't reach it. Enforce limits in code, never in the prompt. Check amount_usd <= 50.0 in your own function and route the rest to a person; treat the prompt as advice the attacker also gets to edit.

The Quick Version

  • Agent security bounds what a tricked agent can do, because hostile text will mislead the model sooner or later.
  • Small refunds flow and big ones wait: a 28 USD refund runs at once while a 180 USD refund pauses for a person under a 50 USD policy.
  • Untrusted pages and pasted tickets never drive privileged tools directly; risky pairings pause for a person first.
  • Each user's stored memory stays separate, so one customer's pasted note cannot expose another customer's data.
  • Code the agent writes runs caged without network access; a bad command breaks the disposable cage and leaves production alone.