This module installs the four bounds that make an analysis agent trustworthy: a state machine that bounds its choices, a sandbox and read-only scopes that bound its blast radius, budgets that bound its spend, and trajectory evals that show when any bound is being tested.
The analyst agent answers: northeast conversion dropped because a checkout change shipped Tuesday, down to 3.1% from 4.4%. It wrote its own SQL, ran it read-only, charted it, and narrated it — the kind of end-to-end analysis that used to be Neha's afternoon.
The 3.1% came from a pandas inner-join that silently dropped every session with a null campaign id — and those nulls were disproportionately the converting ones.
Neha only catches it because of a rule she now trusts more than the agent: before any headline number leaves the system, a boring hand-written path recomputes it, and the two must agree.
The run halted and the disagreement went back to the model as a tool result to investigate.
Set weeks earlier by someone who has watched a confidently wrong number reach a VP's slide: "An agent that writes and runs its own analysis is fine — as long as it can't do anything I can't undo, and can't tell me a number nothing checked."
Those bounds: the sandbox that makes bad code survivable, the recompute that makes wrong numbers catchable, the gate that keeps a human on the irreversible calls, and the trajectory evals that tell you when any bound is being tested.
An agent is an LLM given tools and permission to decide its own next step. For data work that permission is exactly the point — "answer this business question" genuinely requires deciding what to query, whether the result looks wrong, and when to dig deeper — and exactly the risk, because an agent that writes and runs its own analysis code can be confidently, expensively wrong at machine speed. The engineering discipline this module teaches is the same one that runs through this whole track: constrain what the agent can do structurally, checkpoint what it is doing so humans can intervene, and measure what it did with the eval harness you built in M5–M6 — now extended from grading answers to grading trajectories.
You already have the pieces. The Common Core gave you the agent loop and MCP basics; M2 gave you structured outputs and guarded text-to-SQL; M6 gave you calibrated judges. This module assembles them into production-shaped agentic systems for data work: a stateful LangGraph analyst with human gates, a sandboxed code-executing agent that verifies its own numbers, governed MCP access to your data platform, and the guardrails and budgets that keep autonomy affordable.
The reliability of an agentic system lives almost entirely outside the model — in the state machine that bounds its choices, the sandbox and read-only scopes that bound its blast radius, the budgets that bound its spend, and the trajectory evals that tell you when any of those bounds are being tested. Design those four layers first; the prompt comes last.
The question that precedes any framework choice: does this task need runtime decision-making? A nightly enrichment job that classifies tickets has a known step sequence — that's a pipeline (M2), and wrapping it in an agent adds nondeterminism and cost for nothing. Ad-hoc analytical questions, investigation-shaped work ("why did conversion drop in the northeast?"), and workflows where the next step depends on what the data shows are agent-shaped: the step sequence cannot be written down in advance.
Four checks before you build one:
A "no" on any of them means stay with a pipeline, or an agent so constrained it is effectively one. The most common senior-DS failure here isn't building bad agents; it's agent-washing pipelines that were fine.
LangGraph models an agent as an explicit state machine: typed state, nodes that transform it, edges (including conditional ones) that route between them. That explicitness is why it shows up in job postings — you can checkpoint state mid-run, resume after a crash, and pause for a human inside the graph rather than bolting approval onto the outside. The worked example: an analyst agent that drafts SQL for a business question, executes it read-only, self-assesses confidence, and — on low confidence — stops and waits for a human before answering.
from typing import Literal, TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.types import Command, interrupt
class AnalystState(TypedDict):
question: str
sql: str
result: str
confidence: float
answer: str
def draft_sql(state: AnalystState):
sql = llm_draft_sql(state["question"], schema=SCHEMA_CONTEXT) # M2 pattern
return {"sql": sql}
def execute(state: AnalystState):
guard_read_only(state["sql"]) # sqlglot parse; SELECT-only, LIMIT enforced
rows = warehouse.run(state["sql"], role="analytics_readonly", timeout_s=30)
return {"result": rows.to_markdown()}
def assess(state: AnalystState):
conf = llm_self_assess(state) # structured output: 0..1 + rationale
return {"confidence": conf}
def human_gate(state: AnalystState):
decision = interrupt({ # graph pauses HERE; payload goes to the reviewer
"question": state["question"],
"sql": state["sql"],
"result_preview": state["result"][:500],
"reason": f"confidence={state['confidence']:.2f}",
})
if decision["action"] == "revise":
return {"sql": decision["sql"], "confidence": 1.0}
return {} # approved as-is
def answer(state: AnalystState):
return {"answer": llm_write_answer(state)}
def route(state: AnalystState) -> Literal["human_gate", "answer"]:
return "human_gate" if state["confidence"] < 0.7 else "answer"
builder = StateGraph(AnalystState)
for name, fn in [("draft_sql", draft_sql), ("execute", execute),
("assess", assess), ("human_gate", human_gate), ("answer", answer)]:
builder.add_node(name, fn)
builder.add_edge(START, "draft_sql")
builder.add_edge("draft_sql", "execute")
builder.add_edge("execute", "assess")
builder.add_conditional_edges("assess", route)
builder.add_edge("human_gate", "answer")
builder.add_edge("answer", END)
graph = builder.compile(checkpointer=InMemorySaver()) # Postgres checkpointer in prod
config = {"configurable": {"thread_id": "analyst-q42"}}
out = graph.invoke({"question": "Why did NE conversion drop last week?"}, config)
if "__interrupt__" in out: # paused at the human gate
payload = out["__interrupt__"][0].value
# ...surface payload to a reviewer UI; hours later, resume on the same thread:
out = graph.invoke(Command(resume={"action": "approve"}), config)
print(out["answer"])
What makes this production-shaped rather than demo-shaped:
interrupt() checkpoints state and halts; the process can die, the reviewer can respond tomorrow, and Command(resume=...) on the same thread_id picks up exactly where the graph paused. Swap InMemorySaver for the Postgres checkpointer and this survives deploys — the property that separates real HITL from a blocking input() call.interrupt() inside the gating node must be idempotent — do the expensive side effects in earlier nodes, keep the gate node thin.Design graph state as the audit record, not scratch space. If every node writes its inputs, outputs, and rationale into typed state, the checkpointer gives you replayable trajectories for free — which is exactly the raw material M5-style error analysis and this module's trajectory evals consume.
The highest-leverage data agent writes and runs its own analysis code: given a question and a dataset, it plans, queries, computes, charts, and narrates. It is also the sharpest tool in this module, because "the model writes code and we execute it" is a security decision before it's an architecture. Two rules are non-negotiable:
Execution is sandboxed, always. Model-written code runs in an isolated environment — a locked-down container or subprocess with no credentials, a read-only mount of the working dataset, no network egress, CPU/memory/time limits — or a hosted sandbox service. Never in your application process, never with warehouse write credentials. The agent's only data access is through tools you hand it, each scoped read-only.
Headline numbers get independently recomputed. LLM-written pandas is fluent and plausibly wrong — the silent inner-join that drops rows, the mean over a column with sentinel values, the filter applied after the aggregation instead of before. The cheapest strong defense: before any result leaves the system, a verification step recomputes at least one headline number by a different path and requires agreement.
def verify_headline(claim: dict, df) -> bool:
"""claim: {'metric': 'weekly_conversion', 'segment': 'NE', 'value': 0.0312}
Recompute via a hand-written, boring path — no LLM involved."""
seg = df[df.region == claim["segment"]]
recomputed = seg.orders.sum() / seg.sessions.sum()
return abs(recomputed - claim["value"]) / max(abs(recomputed), 1e-9) < 0.01
The pattern generalizes: force the agent (via structured output) to emit machine-checkable claims alongside its narrative, then check each claim with deterministic code — the M5 principle of "assertions before judges" applied to analysis. A disagreement doesn't crash the run; it's returned to the agent as a tool result ("your reported value disagrees with an independent recomputation — investigate"), which the model handles surprisingly well, or it escalates to the human gate.
The loop itself is the standard tool-use pattern. With the Anthropic SDK, a minimal shape:
import anthropic
client = anthropic.Anthropic()
TOOLS = [{
"name": "run_python",
"description": "Execute Python in the analysis sandbox. The dataframe `df` is "
"pre-loaded read-only. Call this to compute, never to guess numbers.",
"strict": True,
"input_schema": {"type": "object",
"properties": {"code": {"type": "string"}},
"required": ["code"], "additionalProperties": False},
}]
messages = [{"role": "user", "content": task_prompt}]
for _ in range(MAX_TURNS): # hard step budget
resp = client.messages.create(model="claude-opus-4-8", max_tokens=4096,
tools=TOOLS, messages=messages)
if resp.stop_reason != "tool_use":
break
messages.append({"role": "assistant", "content": resp.content})
results = [{"type": "tool_result", "tool_use_id": b.id,
"content": sandbox.run(b.input["code"])} # isolated, resource-capped
for b in resp.content if b.type == "tool_use"]
messages.append({"role": "user", "content": results})
Note the shape of the safety story: the sandbox makes bad code survivable, the verification step makes wrong numbers catchable, and the step budget makes runaway loops affordable. None of those live in the prompt.
A data-analysis agent with warehouse credentials and no sandbox is one prompt injection away from being an exfiltration tool — a poisoned CSV column or document chunk can carry instructions the model may follow. Assume the model will sometimes be persuaded; design so persuasion doesn't matter: read-only scopes, no egress, allow-listed tools.
Multi-agent designs — a researcher agent, an analyst agent, a writer agent collaborating on a recurring report — are seductive because they mirror human teams. Sometimes they're right: genuinely separable roles benefit from separate contexts (each agent sees only what its role needs, which controls context bloat), separate tool scopes, and parallelism. Often they're theater: three agents passing prose between themselves to do what one well-prompted model does in a single pass, at several times the cost.
The framework trade-off, concretely:
Published framework comparisons have reported multi-fold token-cost differences between crew-style and explicit-graph implementations of the same workflow. Treat any such number as a hypothesis about your workload, not a constant — then test it, because you own the instrumentation to do so: build the prototype in CrewAI, port the settled design to LangGraph, and run both against the same task set with per-run token accounting from your traces (M9 formalizes the dashboard). The pattern — prototype loose, productionize explicit, decide with measured cost — is itself the transferable skill, and it survives both frameworks' next major versions.
Every agent added to a system multiplies conversational overhead: context re-established per agent, results serialized into prose and re-parsed. Add an agent when roles genuinely need separate context or tool scopes — not because the org chart has those boxes.
The Common Core introduced MCP as a protocol; the senior move is using it as a governance layer over your data platform. Instead of every agent embedding its own warehouse client with its own credentials, you stand up one MCP server that exposes curated, scoped, audited tools — and every agent, IDE assistant, and chat surface in the org consumes the same governed interface. This is the pattern large platform teams have converged on: a central MCP gateway mediating agent access to internal systems, with authorization and PII redaction enforced per tool call rather than per agent.
A minimal server over a warehouse and semantic layer, with the Python MCP SDK:
from mcp.server.fastmcp import FastMCP
mcp = FastMCP("analytics-platform")
@mcp.tool()
def run_query(sql: str) -> str:
"""Run a read-only SQL query against the analytics warehouse.
Only SELECT statements are permitted; results are capped at 200 rows."""
reject_non_select(sql) # sqlglot parse — structural, not regex
df = warehouse.run(sql, role="mcp_readonly", timeout_s=30, max_rows=200)
return redact_pii(df).to_markdown() # column-policy redaction at the boundary
@mcp.tool()
def get_metric_definition(metric_name: str) -> str:
"""Return the governed definition, owner, and SQL of a business metric
from the semantic layer. Call this BEFORE writing SQL about any metric."""
return semantic_layer.lookup(metric_name)
@mcp.tool()
def get_experiment_summary(run_id: str) -> str:
"""Fetch params and metrics for an MLflow run. Read-only."""
return mlflow_summary(run_id)
The design decisions that make this governance rather than plumbing:
get_metric_definition and telling agents (in the tool description — the cheapest steering lever you have) to consult it first is how you stop five agents shipping five definitions of "active user."Autonomy needs bounds on three axes, all enforced outside the model:
strict/additionalProperties: false) plus semantic validation in the handler: does this user have access to this segment? Deny-lists where allow-lists can't be exhaustive: SQL verbs, table patterns, file paths.MAX_TURNS above — an unbounded agent loop is an unbounded bill), a per-run token/cost cap enforced by tracking usage in the loop, and per-team daily budgets at the gateway (M9). Timeouts on every tool. Small models for cheap steps — self-assessment and routing rarely need the frontier tier.Budgets are quality tools, not just cost tools. A run that blows its step budget is a signal the task was underspecified or the agent is thrashing — route budget-exhausted runs to the human gate with the trajectory attached, and mine them in error analysis; they're the richest failure examples you'll get.
Your M5–M6 harness graded final answers. Agents demand a second layer, because an agent can reach a right answer via a trajectory you'd never accept — twelve redundant queries, a scan of a table it didn't need, a guessed number that happened to be close. Trajectory evaluation grades the path, from the traces your checkpointer and tool logs already capture:
get_metric_definition before run_query") as deterministic assertions over the trace — code-based checks first, as always.Operationally, this is the M5 loop on new data: collect trajectories, open-code the failure modes (wrong-tool, thrash-loop, premature-answer, gate-skip...), convert them into a scored eval set, wire it into CI so a prompt or tool-description change that regresses trajectory quality blocks the merge. Phoenix and LangSmith ship agent-eval tooling that runs directly over traces; your job, as in M5, is owning the taxonomy and the pass bars rather than outsourcing them to defaults.
Trajectory-eval questions probe whether you know final-answer accuracy is insufficient for agents. The expected answer shape: deterministic assertions on tool choice, budgets, and safety first; judge-scored step quality second; task completion defined to include process compliance — all computed over persisted traces.
Your first 15 minutes Monday: if you have any agent that emits numbers, add one verification step before its output leaves the system — force it to emit each headline figure as a machine-checkable claim, then recompute one by a boring hand-written path and require agreement within tolerance. That single check is what stood between Neha's agent and a wrong conversion number on a VP's slide.
Check yourself: Your data-analysis agent reaches the correct final answer on a test question. Is that sufficient to trust it in production, and where does an agent's reliability actually come from?
This module anchors two labs. Lab L12 — LangGraph agent with human-in-the-loop builds the stateful RAG-analyst that routes between answer/retrieve/escalate, checkpoints to Postgres, and interrupts for approval on low-confidence answers — the artifact is a running graph plus a trace of a paused-and-resumed HITL episode. Lab L13 — Data-analysis agent, sandboxed builds the code-executing agent end-to-end: business question in, sandboxed computation, chart and narrative out, with the independent-recompute verification step in the loop — the artifact is the agent repo with three solved analysis tasks and their verification records. The playbooks provide the agent-vs-pipeline decision checklist, the sandbox hardening checklist, the MCP data-platform server template with read-only scoping and redaction hooks, the guardrail/budget configuration worksheet, and the trajectory-eval taxonomy starter. Do this at work this week → pb-41 LangGraph Stateful RAG-Agent with Human-in-the-Loop (the durable-interrupt analyst with confidence-gated escalation), pb-40 Build a Sandboxed Data-Analysis Agent (Neha's recompute-verification loop, hardened), and pb-44 Agent Guardrails and Cost Controls (step caps, deny-lists, and the kill switch). All paths run free-tier: local DuckDB or Postgres, subprocess or container sandboxes, cheap-tier models for routing and judging, step budgets keeping per-lab API spend inside the track's standard cap.
interrupt(), Command(resume=...), and resume semantics