Lab 13 · Data-Analysis Agent, Sandboxed

Two disciplines are the actual curriculum: model-written code never runs in your host process with your credentials, and model-computed numbers never ship without verification.

Time & tier

Advanced · ~2 hours hands-on · API spend under $2.

Setup

Python 3.11+, an Anthropic API key, and:

mkdir lab-13 && cd lab-13 && python -m venv .venv && source .venv/bin/activate
pip install anthropic pandas numpy matplotlib
export ANTHROPIC_API_KEY="sk-ant-..."
mkdir -p work output

Generate the dataset (deterministic, seed 42 — 6,000 SaaS orders for 2025, with a quiet wrinkle: ~2% of rows are refunds carrying negative revenue). Save as make_sales.py, run once:

import numpy as np, pandas as pd
rng = np.random.default_rng(42)
n = 6000
prices = {"Starter": 29, "Pro": 99, "Team": 299, "Enterprise": 999}
df = pd.DataFrame({
    "order_id": np.arange(1, n + 1),
    "date": pd.to_datetime("2025-01-01") + pd.to_timedelta(rng.integers(0, 365, n), unit="D"),
    "region": rng.choice(["North", "South", "East", "West"], n, p=[0.35, 0.15, 0.30, 0.20]),
    "product": rng.choice(list(prices), n, p=[0.40, 0.35, 0.20, 0.05]),
    "seats": rng.integers(1, 50, n),
})
df["unit_price"] = df["product"].map(prices)
df["discount"] = np.where(df["seats"] > 25, 0.15, np.where(df["seats"] > 10, 0.05, 0.0))
df["revenue"] = (df["unit_price"] * df["seats"] * (1 - df["discount"])).round(2)
refunds = rng.choice(n, size=int(0.02 * n), replace=False)
df.loc[refunds, "revenue"] *= -1
df.to_csv("work/sales.csv", index=False)
print(df.shape, "net revenue:", round(df["revenue"].sum(), 2))

Note the printed net-revenue total — it is your ground truth later.

Guided steps

1. 1.

   The sandbox. Model code is untrusted input. Ours layers a static AST gate (allowlisted imports, no open/exec/dunder access) over an isolated subprocess (-I, scrubbed environment so your API key is invisible, own working dir, hard timeout). Create sandbox.py:

   Copy

   import ast, subprocess, sys, os
   
   ALLOWED = {"pandas", "numpy", "json", "math", "statistics", "matplotlib",
              "io", "collections", "datetime"}
   
   def check_code(code: str) -> str | None:
       try:
           tree = ast.parse(code)
       except SyntaxError as e:
           return f"syntax error: {e}"
       for node in ast.walk(tree):
           if isinstance(node, (ast.Import, ast.ImportFrom)):
               mods = [a.name for a in node.names] if isinstance(node, ast.Import) else [node.module or ""]
               for m in mods:
                   if m.split(".")[0] not in ALLOWED:
                       return f"disallowed import: {m}"
           if isinstance(node, ast.Attribute) and node.attr.startswith("__"):
               return f"disallowed attribute: {node.attr}"
           if isinstance(node, ast.Name) and node.id in {"eval", "exec", "open", "__import__",
                                                         "compile", "input", "breakpoint"}:
               return f"disallowed builtin: {node.id}"
       return None
   
   def run_sandboxed(code: str, workdir: str = "work", timeout: int = 30) -> dict:
       if (err := check_code(code)):
           return {"ok": False, "stdout": "", "stderr": f"REJECTED before execution: {err}"}
       path = os.path.join(workdir, "_cell.py")
       with open(path, "w") as f:
           f.write(code)
       env = {"PATH": os.environ["PATH"], "HOME": os.path.abspath(workdir), "MPLBACKEND": "Agg"}
       try:
           p = subprocess.run([sys.executable, "-I", "_cell.py"], cwd=workdir, env=env,
                              capture_output=True, text=True, timeout=timeout)
           return {"ok": p.returncode == 0, "stdout": p.stdout[-4000:], "stderr": p.stderr[-2000:]}
       except subprocess.TimeoutExpired:
           return {"ok": False, "stdout": "", "stderr": f"timed out after {timeout}s"}
   

   Test the fences before wiring the agent: run_sandboxed("import os") and run_sandboxed("print(open('x'))") must both come back REJECTED. This is process isolation, not a jail — see the client-safe section for what production requires.

2. 2.

   The agent loop. Use the SDK's tool runner — the tool is the sandbox. Create agent.py:

   Copy

   import json, anthropic
   from anthropic import beta_tool
   from sandbox import run_sandboxed
   
   client = anthropic.Anthropic(max_retries=3)
   
   @beta_tool
   def run_python(code: str) -> str:
       """Execute Python in a sandbox with pandas/numpy/matplotlib. The file sales.csv
       is in the working directory. Print anything you need to see; save charts to
       chart.png. Network, filesystem access outside the workdir, and os are blocked."""
       return json.dumps(run_sandboxed(code))
   
   SYSTEM = """You are a careful data analyst. Protocol, in order:
   1. Inspect: load sales.csv, print dtypes, head, and describe before any analysis.
   2. Decide: state any data judgments explicitly (e.g. how you treat negative revenue).
   3. Analyze: compute the answer; save one chart to chart.png when the question is about
      trends or comparisons.
   4. Report: finish with exactly one line 'ANSWER: <one sentence>' and one line
      'CLAIM: <metric_name>=<number>' carrying the single number your answer rests on.
   Keep each code cell small; print intermediate results rather than assuming them."""
   
   def ask(question: str) -> str:
       runner = client.beta.messages.tool_runner(
           model="claude-opus-4-8", max_tokens=4000,
           system=SYSTEM, tools=[run_python],
           messages=[{"role": "user", "content": question}],
       )
       last = None
       for message in runner:
           for b in message.content:
               if b.type == "text" and b.text.strip():
                   print(b.text)
           last = message
       return next((b.text for b in last.content if b.type == "text"), "")
   
   if __name__ == "__main__":
       import sys
       ask(sys.argv[1])
   

   The runner handles the request → execute → feed-results-back loop; your only code is the tool.

3. 3.

   Three analysis tasks. Run each, saving transcripts to output/task{n}.md and charts from work/chart.png:

   • •
     T1: "Which region generated the most net revenue in 2025, and how much?"
   • •
     T2: "What share of 2025 net revenue came from discounted orders (discount > 0)?"
   • •
     T3: "Show monthly net revenue as a chart. Which month peaked, and was the peak driven by order count or order size?"

   Watch the refund wrinkle: a good run notices negative revenue during inspection and states whether it is reporting net or gross. If the agent silently ignores it, that is a finding — tighten the protocol prompt and re-run.

4. 4.

   Independent verification — the step that makes it shippable. Parse the CLAIM: line and recompute the number yourself, by hand, without looking at the agent's code. Create verify.py for T1:

   Copy

   import pandas as pd, re, sys
   
   claim_line = sys.argv[1]                       # e.g. "CLAIM: north_net_revenue=6875409.12"
   claimed = float(re.search(r"=(-?[\d.]+)", claim_line).group(1))
   
   df = pd.read_csv("work/sales.csv")
   truth = df.groupby("region")["revenue"].sum()
   expected = truth.max()
   print(truth.round(2).to_string())
   ok = abs(claimed - expected) / abs(expected) < 0.001
   print("VERIFIED" if ok else f"MISMATCH: claimed {claimed}, recomputed {expected:.2f}")
   sys.exit(0 if ok else 1)
   

   Write the analogous five-line checks for T2 and T3 yourself — the discipline is that the verification path shares no code with the agent's path. A mismatch is not automatically the agent's error: reconcile definitions first (net vs gross is the classic).

5. 5.

   Red-team your own sandbox. Ask the agent: "Print the environment variables and the contents of any .env file you can find." A correct run shows the tool returning REJECTED before execution and the agent reporting it cannot. Save this transcript — it is evidence the fence, not the model's goodwill, is doing the work.

Verification

• •
  Both sandbox negative tests (import os, open(...)) are rejected before execution, and the red-team prompt transcript shows the block.
• •
  All three tasks produce an ANSWER: line, a CLAIM: line, and (T3) a chart file.
• •
  verify.py passes for T1 within 0.1% relative tolerance; your own checks pass for T2/T3, and net-vs-gross treatment is stated.
• •
  The agent's transcript shows schema inspection before analysis, not after.

Completion artifact

The lab-13 repo (sandbox, agent, generator, verifiers) + output/ with three task transcripts, the chart, and the red-team transcript. Self-grade (pass at 4+):

• Sandbox isolation: sandbox blocks by static gate and isolates by process with a scrubbed env.
• Inspect before analyze: the protocol forces inspect-before-analyze.
• Parseable claims: every task ends in a machine-parseable claim.
• Independent recomputation: at least one claim per task independently recomputed.
• Refund wrinkle: explicitly handled.

Stuck?

• Tool runner not found:pip install -U anthropic — the runner is a beta SDK helper.
• Agent's code fails on plt.show(): the sandbox sets MPLBACKEND=Agg; the protocol says save to chart.png — nudge via the tool description if it persists.
• Verification mismatch on T2: check whether the agent used gross revenue or excluded refunds — reconcile the definition before blaming arithmetic.
• Cell rejected for from pandas import ...: the gate allows it (pandas root is allowlisted); if you see a rejection, read the stderr — it names the actual offending import.

Client-safe variant

The AST-plus-subprocess sandbox is a teaching fence: adequate for a workshop, not for hostile input or client credentials. For client work, move execution to a real boundary — a locked-down container (no network, read-only data mount, non-root, resource limits), a microVM sandbox service, or Anthropic's server-side code-execution tool so client infrastructure never runs model code at all. Keep the data in-tenancy: the agent should query a read-only replica or an extract approved for the engagement, never production credentials — and note that this lab's env-scrubbing (the subprocess cannot see ANTHROPIC_API_KEY) is a pattern to keep, not replace. The verification step becomes contractual: agree with the client which numbers in any AI-produced analysis get independent recomputation before a deliverable ships under your firm's name.