Mitigate Insecure Output Handling in Your LLM Application

Tier: Core · Workflow area: LLM attack surface & defense

The model's output is untrusted the moment it can be influenced by untrusted input — and if your app pipes that output into a browser, a shell, a SQL string, or an HTTP client without treating it as tainted, you have handed an attacker XSS, SSRF, SQL injection, or remote code execution through the model. This playbook treats every LLM response as attacker-controlled data and hardens the sinks it flows into, in your own app.

When to reach for this

• •
  Your app renders model output as HTML/markdown, or feeds it to a code interpreter, database, shell, or downstream API.
• •
  An LLM agent generates commands, queries, or URLs that another component executes.
• •
  You are reviewing a new AI feature and need to enumerate where model text reaches a dangerous sink.

The workflow

Threat, briefly. The model, steered by a prompt injection, emits <img src=x onerror="fetch('/admin/delete')"> and your frontend renders it as trusted HTML — stored XSS. Or it emits a SQL fragment you string-concatenate, or a URL your server fetches (SSRF), or shell text your agent runs (RCE). The model is just the delivery vehicle; the bug is the unescaped sink.

1. 1.

   Map every sink first. Grep the codebase for where model output leaves the LLM boundary: template rendering, innerHTML, dangerouslySetInnerHTML, eval, subprocess, DB query builders, HTTP clients, file writes. You cannot defend sinks you haven't listed.

2. 2.

   Encode at the boundary, per sink type. The fix is the same discipline as any untrusted data: context-appropriate output encoding.

   Copy

   import html
   # HTML sink: escape, never render model text as raw markup
   safe = html.escape(model_output)
   
   # SQL sink: parameterize, never concatenate
   cur.execute("SELECT * FROM notes WHERE body = %s", (model_output,))
   
   # Shell sink: no shell=True, pass an arg vector, allow-list the command
   subprocess.run(["ripgrep", "--", model_output], shell=False)
   

3. 3.

   Render markdown safely. If you must render model markdown, run it through a sanitizer that strips scripts, event handlers, and dangerous URI schemes, and disable raw HTML passthrough.

   Copy

   import bleach
   clean = bleach.clean(
       rendered_html,
       tags=["p", "ul", "ol", "li", "code", "pre", "strong", "em", "a"],
       attributes={"a": ["href"]},
       protocols=["http", "https"],   # drop javascript:, data:
       strip=True,
   )
   

4. 4.

   Constrain the shape of output. Where the output feeds a machine sink, don't accept free text — force a schema so anything outside it is rejected before it reaches the sink.

   Copy

   from pydantic import BaseModel
   
   class DbFilter(BaseModel):
       status: str   # validated against an allow-list downstream
       limit: int
   # response = client.messages.parse(..., output_format=DbFilter)
   

5. 5.

   Test the sink against your local app. On a local, deliberately vulnerable copy, inject a payload that steers the model to emit an XSS or SSRF string, then confirm the encoded/sanitized build neutralizes it.

   Copy

   # local target only: prove the sink is closed
   curl -s localhost:8080/summarize \
     -d '{"note":"Summarize: <img src=x onerror=alert(1)>"}' | grep -o '<img'
   # defended build: no <img passes through; grep returns nothing
   

Verify before you trust it

The defense holds when the exploit fires on the vulnerable build and dies on the patched one. For XSS: the payload string appears escaped (<img) in the DOM and no script executes — check with a headless browser assertion, not by eye. For SSRF: the model-supplied URL is rejected by an egress allow-list and your server never makes the outbound request (confirm in the network log). For RCE: the command runs through an arg vector against an allow-listed binary, so shell metacharacters are inert. Keep one regression test per sink so a refactor that reintroduces innerHTML fails CI.

Pitfalls

• Sanitizing input instead of encoding output. You cannot predict every payload on the way in; encode at the sink where the context is known.
• One global escaper. HTML escaping does nothing for a SQL or shell sink. Each sink needs its own context-correct handling.
• Allowing javascript: and data: URIs in "safe" markdown links — a classic bypass. Restrict protocols explicitly.
• Trusting structured output to be safe. A valid JSON string field can still carry an XSS payload; schema validation is not output encoding.
• Agent-generated code executed unsandboxed. If the model writes code you run, that is an RCE sink by design — sandbox it (see the excessive-agency playbook).

Tools

• LLM Guard-class output scannersdetect and neutralize dangerous output patterns (markup, secrets, malformed data) inline before the sink.
• Standard sanitizers (bleach, DOMPurify-class) — vendor-neutral HTML/markdown sanitization at the render boundary.
• SAST tools (Semgrep/CodeQL-class) — flag where model output reaches innerHTML, eval, subprocess, or unparameterized queries across the codebase.

Client-safe variant

For client engagements, the sink review is the same but the code is theirs — get scope approval before pointing any SAST tool or assistant at a client repo. If public tools are barred, run the sink grep and the sanitizer locally, and describe the vulnerable pattern abstractly (no client hostnames, schema names, or URLs) to any approved assistant helping you reason about fixes. The mitigations — output encoding, parameterization, egress allow-lists, sandboxing — are framework- and vendor-independent and work fully offline, so the client's data never needs to leave their environment to build the defense.