Skip to content
AI360Xpert
Gen AI

Guardrails

Guardrails are external security layers that sit between the user and the LLM, screening inputs for attacks and filtering outputs for harmful or incorrect content.

An input guardrail blocks a prompt injection before it reaches the main LLM, while an output guardrail blocks the LLM from sending restricted information back to the user.
An input guardrail blocks a prompt injection before it reaches the main LLM, while an output guardrail blocks the LLM from sending restricted information back to the user.

Why Does This Exist?

Large Language Models are probabilistic. No matter how much fine-tuning or alignment training you do, it is mathematically impossible to guarantee that a model will never be jailbroken, or never hallucinate a toxic response. Relying entirely on the model's internal safety is like securing a bank by just asking the tellers to be really good at spotting robbers.

To build production-ready applications, you need an external layer of defense. This is a guardrail. Guardrails are separate software components—often smaller, specialized models or rigid rule-based scripts—that intercept traffic going into and coming out of your primary LLM. If the input looks like a prompt injection, the guardrail rejects it before the LLM ever sees it. If the output contains a hallucinated fact or a leaked secret, the guardrail blocks it before it reaches the user.

Think of It Like This

A bouncer and a customs officer at an event

Imagine a high-profile guest speaker (the LLM) at a public event.

You wouldn't let the audience hand questions directly to the speaker on stage. Instead, a moderator (the input guardrail) reviews every question card first. If a card contains a bomb threat or a prank, the moderator throws it away.

Similarly, before the speaker's answers are broadcast on live television, a producer with a delay button (the output guardrail) listens in. If the speaker accidentally starts revealing classified state secrets, the producer cuts the feed.

The speaker is highly capable, but the security of the event relies on the independent layers checking what goes in and what comes out.

How It Actually Works

Input Guardrails

Input guardrails screen the user's prompt before passing it to the main generation model. Their job is to detect:

  1. Prompt Injections & Jailbreaks: Checking if the user is using known "Do Anything Now" (DAN) prompts or attempting to override the system instructions.
  2. Toxicity & Hate Speech: Blocking offensive language before computing a response.
  3. Topic Enforcement: If your app is a customer service bot for a shoe store, the guardrail detects if the user is asking about politics and blocks the request immediately.

These guardrails are typically implemented using fast, cheap classification models (like a small BERT model trained specifically to detect injections) or regular expressions.

Output Guardrails

Output guardrails screen the text generated by the LLM before sending it to the user. Their job is to detect:

  1. Hallucinations (Fact-Checking): In a RAG pipeline, an output guardrail can compare the LLM's answer against the retrieved documents to ensure the model didn't invent facts.
  2. PII and Secrets Leakage: Scanning the output for Social Security Numbers, phone numbers, or internal API keys.
  3. Tone and Brand Safety: Ensuring the model isn't being rude, aggressive, or using language that violates corporate brand guidelines.

Implementation Frameworks

Building these filters from scratch is tedious. Frameworks like NeMo Guardrails (by NVIDIA) or Guardrails AI allow developers to define these checks declaratively. You write a configuration file saying "block off-topic questions" and "ensure the output is valid JSON," and the framework orchestrates the sequence of checks automatically.

Show Me the Code

# A conceptual implementation of input and output guardrailsdef generate_safe_response(user_input: str) -> str:    # 1. Input Guardrail: Check for injection    if injection_classifier.predict(user_input) == "MALICIOUS":        return "Request blocked: Policy violation detected."            # 2. Main LLM Generation    prompt = f"System: You are a helpful bot.\nUser: {user_input}"    raw_response = main_llm.generate(prompt)        # 3. Output Guardrail: Check for PII leakage    if pii_scanner.contains_secrets(raw_response):        return "Response blocked: Contained sensitive information."            return raw_response

This pattern—often called the "sandwich" pattern—ensures the expensive and unpredictable main LLM is wrapped by cheaper, predictable security boundaries.

Watch Out For

Latency and cost bloat

Guardrails add processing steps. If you use an LLM (like GPT-4) as your guardrail to evaluate the input, and then run it again to evaluate the output, you have tripled your latency and API costs. Production guardrails usually rely on much smaller, faster models (like specific classifiers or local heuristics) to keep the user experience snappy.

Overly restrictive filtering

If your input guardrail is too aggressive, it will block legitimate user queries (false positives). For example, a medical AI might be blocked by a toxicity guardrail because the user typed a clinical term that the guardrail flagged as inappropriate. Tuning the threshold of your guardrails is an ongoing operational challenge.

The Quick Version

  • Guardrails are external software layers that validate inputs to and outputs from an LLM.
  • Input guardrails block prompt injections, jailbreaks, and off-topic requests before they reach the main model.
  • Output guardrails block hallucinations, PII leakage, and toxic responses before they reach the user.
  • They are necessary because internal model alignment can never guarantee 100% safety.
  • Frameworks like NeMo Guardrails simplify orchestrating these checks.
  • Developers must balance security against the added latency and cost of running multiple checks per request.

Related concepts