Scheduled Retrain Agent with a Human Approval Gate

Tier: Advanced · Workflow area: Agentic systems

This playbook composes patterns you already have (code-executing agent, MLOps APIs, approval gating) into a scheduled agent that does the maintenance work and files a promotion request instead of promoting.

When to use

• •
  A model you retrain on a fixed cadence (weekly/monthly) where the procedure is stable and documented.
• •
  Retraining toil is crowding out higher-value work, but you are not ready to trust auto-promotion (usually correct).
• •
  You want a portfolio piece composing agents + MLflow + guardrails end-to-end.

Do not automate a retrain you have not run manually at least five times — the agent inherits your procedure's maturity, including its gaps.

Prerequisites

• •
  An existing training pipeline callable as a script or function (train.py --data-window ...) with MLflow tracking already wired in.
• •
  MLflow model registry (self-hosted is fine) with a champion model registered and an alias like @champion.
• •
  Guardrails from pb-44 (budgets, deny-lists) — this agent runs unattended, so they are mandatory, not optional.
• •
  A scheduler: cron, GitHub Actions schedule, or Airflow.

The workflow

1. 1.

   Write the retrain runbook as the agent's task spec. The exact steps you do manually: pull data window, run integrity checks, train, evaluate on the fixed holdout and a fresh backtest window, compare to champion metrics, decide recommend/reject. If a step relies on your judgment, write down the rule you actually use — that forcing function is half the value.

2. 2.

   Give the agent narrow tools, not a shell. run_training(config), run_backtest(model_uri, window), get_champion_metrics(), log_report(text), request_promotion(model_uri, evidence) — each a thin wrapper over your pipeline and MLflow client. No general bash, no registry-write beyond opening a request.

3. 3.

   Make request_promotion a request, not an action. It tags the run and notifies a human (Slack webhook/email) with the evidence table; it never touches the @champion alias:

   Copy

   @beta_tool
   def request_promotion(model_uri: str, evidence: str) -> str:
       """File a promotion request for human review. Does NOT promote.
   
       Args:
           model_uri: MLflow model URI of the challenger.
           evidence: Markdown table comparing challenger vs champion.
       """
       mlflow.set_tag("promotion_requested", "true")
       notify_reviewers(model_uri, evidence)
       return "Promotion request filed; awaiting human approval."
   

   Approval itself is a human running one command (or clicking in the MLflow UI) that moves the alias — the credential able to do that is never in the agent's environment.

4. 4.

   Wire the loop and schedule it. The agent (Anthropic tool runner or a LangGraph graph) executes the runbook with a step budget of ~15 and a token/dollar cap. Schedule via cron/Actions; each run writes a full report — data window, checks passed, metrics vs champion, recommendation and why — to MLflow artifacts.

5. 5.

   Handle the three exit states explicitly. (a) Challenger better → promotion request filed; (b) challenger worse or checks failed → report filed, no request, and that is a successful run; (c) budget/error → alert a human with partial findings. State (b) being normal is what keeps the gate meaningful.

6. 6.

   Shadow it for two cycles. Run the agent alongside your manual process; diff its recommendation against yours before letting it replace the manual run.

Tool variants

• LangGraph + interrupt() (pb-41): model approval as a durable pause instead of an external notification — reviewer resumes the same run with approve/reject.
• Airflow-orchestrateddeterministic steps (data pull, training) as ordinary tasks; the agent only handles evaluation, comparison, and report-writing — a good fit when the pipeline predates the agent.
• Anthropic scheduled deployments / managed agents if you want the schedule and loop hosted rather than cron on your infra.

Verification

• •
  Two shadow cycles where the agent's recommendation matched (or its mismatch taught you a runbook gap you then fixed).
• •
  A seeded degraded challenger (train on truncated data) is correctly rejected with the reason visible in the report.
• •
  The agent's environment demonstrably lacks alias-write permission — verified by attempting promotion with its credentials.
• •
  Budget breach path tested: partial report + human alert, no silent death.

Pitfalls

• Auto-promotion "just for low-risk models." The incident that ends agent programs is an unattended promotion; keep the human gate until you have quarters of evidence, then decide deliberately.
• Vague comparison rules. "Promote if better" hides threshold judgment; specify metric, margin, and slice checks (e.g. "AUC +0.005 overall and no slice worse by >0.01").
• Reports nobody reads. Route the recommendation and evidence to where reviewers already look (Slack), not only into MLflow artifacts.
• Agent and pipeline sharing one credential. Give the agent its own service account so the audit log distinguishes agent actions from yours.

Restricted-environment variant

Fully self-hostable: local model via vLLM for the agent loop, self-hosted MLflow, cron on your infra — no data or metrics leave the environment. For client engagements, the promotion gate maps naturally onto the client's existing change-approval process: the agent files the change request in their system, which usually makes this pattern easier to approve than any auto-deploy proposal.

Success criteria

• •
  Agent completes the full retrain → backtest → compare → report cycle unattended within budget.
• •
  Human gate proven structural (permission test) and exercised in a real approval.
• •
  Two shadow cycles documented; degraded-challenger test rejected correctly.
• •
  Maintenance time for this model measurably drops from hours to minutes of review per cycle.