For builders on the Databricks Data Intelligence Platform. This certification validates that you can design, build, deploy, govern, and monitor production generative AI applications there — RAG pipelines and agents, end to end, not LLM theory.
It is a builder's exam: it assumes you understand what an LLM is and instead tests whether you can assemble a working RAG pipeline or agent — chunk the documents, build the vector index, wire the chain in LangChain or the Mosaic AI Agent Framework, log it to MLflow, register it in Unity Catalog, serve it on Model Serving, and then evaluate and monitor it with judges, scorers, inference tables, and the AI Gateway.
The March 2026 exam guide refresh matters: it added the newest platform capabilities — Agent Bricks (Knowledge Assistant, Multi-Agent Supervisor, Information Extraction), Model Context Protocol (MCP) servers, Genie Spaces in multi-agent systems, and AI Gateway governance — alongside the classic RAG and deployment material. Older practice material that predates these topics will leave gaps, so treat this guide as your map of the current blueprint. Expect scenario questions of the form "a team needs X under constraint Y — which component/configuration/code snippet?" where two options are plausible and the constraint (latency, cost, governance, document structure) decides the winner.
| Exam Fact | Detail |
|---|---|
| Exam | Databricks Certified Generative AI Engineer Associate |
| Questions | 45 scored |
| Duration | 90 minutes |
| Passing score | Not published; plan for ~70% |
| Cost | $200 USD (as of July 2026) |
| Question formats | Multiple choice and multiple selection |
| Delivery | Online proctored |
| Validity | 2 years |
| Languages | English, Japanese, Portuguese (BR), Korean |
The six parts below mirror the official exam domains exactly, in official order and weight. Budget your study time proportionally: Application Development (30%) and Assembling & Deploying (22%) are more than half the exam. Governance is only 8%, but its questions are among the easiest points on the paper if you know the three or four facts it tests.
Callouts flag material worth extra attention:
Work through the parts in order — the domains build on each other the way a real project does (design → data → build → deploy → govern → monitor) — then use the Exam-Day Checklist as a rapid-review sheet in your final 48 hours.
Design questions test whether you can translate a business requirement into the right generative AI architecture before writing any code: the right model task, the right prompt, the right chain components in the right order.
Every design question starts the same way: identify what the user provides, what the business needs back, and what must happen in between. Given "customers ask questions about our product manuals and need cited answers," the input is a natural-language question, the output is a grounded answer with source citations, and the middle is retrieval over the manuals plus generation — a RAG pipeline. Given "extract invoice number, vendor, and total from scanned PDFs into a table," the input is a document image, the output is structured fields, and the middle is OCR plus extraction — not RAG at all.
Practice decomposing requirements into: input format → processing stages → output format → constraints (latency, cost, privacy, freshness). The exam frequently asks which inputs a pipeline needs to achieve a goal — e.g., serving relevant troubleshooting docs alongside a chatbot answer requires the user query, a vector index of the docs, and metadata for filtering; it does not require fine-tuning data.
Map business language to model task:
When a scenario involves questions over structured data (tables, metrics, sales figures), the answer is Genie or text-to-SQL tooling — not a vector search index. Vector search is for unstructured content. Questions that mix both point to a multi-agent design (Part 3).
When a requirement says the response must be JSON, a table, a fixed template, or "only the answer, nothing else," the tested skill is prompt design for format control:
Few-shot examples are the highest-leverage tool: one or two demonstrations of input → correctly formatted output outperform paragraphs of instructions.
To guarantee output format, combine an explicit schema in the prompt, few-shot examples, and (where supported) structured-output enforcement at the API level — and design a parser/validator step after the LLM in the chain.
A chain is a sequence of components where each output feeds the next. Know the standard inventory and when each appears:
Given "the model keeps answering from stale training data," add a retriever. Given "downstream code crashes on malformed model output," add an output parser and validation. Given "the bot forgets what the user said two turns ago," add memory.
Multi-stage questions give you a goal and ask for the correct execution order. Reason about data dependencies: a step can only run when its inputs exist. The canonical RAG order is user query → (optional query rewrite) → embed query → vector search retrieval → (optional re-rank/filter) → prompt augmentation with retrieved context → LLM generation → post-processing/parsing → response. For an agent handling "find the customer's latest order and summarize the complaint pattern": look up the customer (tool) → fetch orders (tool) → retrieve complaint tickets (retriever) → summarize (LLM). If a step consumes another step's output, it must come after it — the exam's ordering questions are solved entirely by this dependency logic.
Agent Bricks is Databricks' declarative, auto-optimized way to build production agents: you describe the task and point at your data, and the platform builds, tunes, and evaluates the agent (generating synthetic evaluation data and optimizing quality/cost automatically). Know the three flavors on the blueprint cold:
| Agent Brick | Purpose | Typical scenario cue |
|---|---|---|
| Knowledge Assistant | Q&A chatbot over your documents with citations; managed RAG end to end | "Answer employee questions from HR policies with sources, minimal engineering effort" |
| Multi-Agent Supervisor | Orchestrates multiple agents and Genie Spaces under one coordinator that routes each request | "One assistant that answers both document questions and sales-table questions by delegating" |
| Information Extraction | Turns unstructured documents into structured rows/fields at scale | "Extract fields from thousands of contracts into a Delta table" |
The differentiator between Agent Bricks and hand-building with the Agent Framework is effort vs. control. Cues like "fastest path," "no/low code," "automatically optimized" → Agent Bricks. Cues like "custom orchestration logic," "full control over the chain code" → build it yourself with LangChain/Agent Framework (Part 3).
Knowledge Assistant = answers from docs; Multi-Agent Supervisor = routes to specialists; Information Extraction = docs into tables.
RAG quality is decided before the first query runs — by what you extract, how you chunk it, what you filter out, and how you measure retrieval.
Chunking splits documents into retrievable units. The two hard constraints: chunks must fit the embedding model's context length (a chunk longer than the embedding model's max tokens gets truncated and silently loses content), and retrieved chunks must collectively fit the LLM's context window alongside the prompt. Strategy selection is driven by document structure:
| Strategy | How it works | Best for |
|---|---|---|
| Fixed-size with overlap | Split every N tokens/characters, overlap 10–20% so sentences aren't severed at boundaries | Uniform, unstructured prose; simplest baseline |
| Recursive character splitting | Try splitting on paragraphs, then sentences, then words until chunks fit | General-purpose default (LangChain RecursiveCharacterTextSplitter) |
| Structure/header-aware | Split on Markdown/HTML headers, sections, or layout elements | Manuals, wikis, policies with meaningful sections; keeps a topic per chunk |
| Semantic chunking | Split where embedding similarity between sentences drops | Documents with topic shifts not marked by headers; higher compute cost |
| Parent-document / context-window expansion | Embed small chunks for precise retrieval but pass the surrounding larger section to the LLM | When small chunks retrieve well but lack context to answer from |
Chunk size trades off precision against context: small chunks → precise matching but fragmented meaning; large chunks → complete context but diluted embeddings and wasted LLM context. Overlap mitigates boundary loss. Prepending metadata (document title, section heading) to each chunk improves both retrieval and grounding. "Advanced chunking design" scenarios expect you to combine these — e.g., header-aware splitting first, then recursive splitting of oversized sections, with title metadata prepended.
If a scenario says answers are "missing context that was in the adjacent paragraph," the fix is more overlap, larger chunks, or parent-document retrieval. If it says retrieval returns "chunks that mention the term but don't answer the question," the fix is smaller/structure-aware chunks or re-ranking — not bigger chunks.
Garbage in the index becomes garbage in the answer. Before embedding, remove: boilerplate (headers, footers, page numbers, navigation menus, legal disclaimers repeated on every page), duplicate and near-duplicate documents (they crowd the top-k with redundant results), outdated or superseded versions (retrieval can't tell 2019 policy from 2026 policy — filter or tag by date), irrelevant sections (title pages, tables of contents), and malformed extraction artifacts (OCR noise, broken encoding). Also exclude documents you are not licensed to use (Part 5). Source selection matters as much as filtering: pick the authoritative, current, in-scope corpus for the use case, not "everything on the shared drive."
Match the extraction tool to the source format:
pytesseract.image_to_string(image) → clean → chunk."Scanned" or "image" in the question = OCR = pytesseract. A normal PDF with selectable text does not need OCR.
Processed chunks land in a Delta table registered in Unity Catalog — this is the platform-idiomatic pattern because Vector Search indexes sync from Delta tables (Part 4) and Unity Catalog provides governance and lineage. The canonical schema: a primary key column (chunk id — required for Vector Search), the chunk text column, source metadata (document URI, page, section), and optionally a precomputed embedding column. Write with Spark: df.write.format("delta").mode("append").saveAsTable("catalog.schema.chunks"). Enable Change Data Feed (delta.enableChangeDataFeed = true) on the table — a Delta Sync vector index requires it to pick up incremental changes.
The prep pipeline is: extract (pytesseract/pypdf/etc.) → clean and filter → chunk → write to a Delta table in Unity Catalog with a primary key and Change Data Feed enabled → sync a Vector Search index from it.
Retrieval is evaluated separately from generation, against a ground-truth set of (query → relevant chunks/documents):
On Databricks, MLflow evaluation computes retriever metrics when traces capture the retrieval span, and LLM judges (e.g., chunk/retrieval relevance judges in mlflow.genai.evaluate()) grade whether retrieved chunks are relevant without needing exhaustive human labels (Part 6 covers the judge machinery). Use these metrics to iterate on chunk size, embedding model, and k.
First-stage vector search optimizes recall cheaply; a re-ranker (typically a cross-encoder that scores each query–chunk pair jointly) re-orders the candidates so the truly relevant ones land at the top before the context is assembled. The standard pattern: retrieve a generous candidate set (e.g., top 50) → re-rank → keep the top few for the prompt. Re-ranking raises precision and answer quality at the cost of added latency and compute — the right answer when "retrieval returns relevant documents but they're buried below marginal ones," and the wrong answer when the scenario's constraint is strict latency. Hybrid search (combining vector similarity with keyword/BM25 matching) is the related fix when queries contain exact terms, SKUs, or acronyms that pure semantic search misses.
The heaviest domain: building the actual chain or agent — prompts, LangChain, model selection, guardrails, MLflow, the Agent Framework, and multi-agent systems with Genie.
LangChain is the exam's reference orchestration library. Know the component classes and what each is for: prompt templates (ChatPromptTemplate) parameterize prompts; chat models wrap LLM endpoints (ChatDatabricks binds to a Databricks Model Serving endpoint); retrievers wrap vector stores (DatabricksVectorSearch wraps a Mosaic AI Vector Search index and exposes .as_retriever()); output parsers (StrOutputParser, JSON parsers) shape results; memory carries chat history; tools give agents actions; agents loop over reason → act → observe. LCEL composes them with pipes:
chain = (
{"context": retriever, "question": RunnablePassthrough()}
| prompt
| ChatDatabricks(endpoint="databricks-meta-llama-3-3-70b-instruct")
| StrOutputParser()
)
Tool-selection questions describe a need and ask which component satisfies it: multi-turn context → memory; grounding in company docs → retriever over a vector store; act on external systems → tools/agent; enforce structured output → output parser. Databricks also exposes Unity Catalog functions as agent tools (via the UC function toolkits), so a governed SQL/Python function becomes a callable tool with UC permissions applied.
Given a baseline prompt producing wrong-shaped output, iterate systematically: add an explicit role/persona ("You are a support agent for ACME…"), state the task and constraints precisely, add few-shot examples of desired input→output, specify output format, and add grounding instructions ("Answer only from the provided context; otherwise say you don't know"). Exam items show a prompt and a complaint ("responses are too long / off-brand / hallucinated / unparseable") and ask for the minimal edit — pick the instruction that directly targets the complaint.
Augmenting prompts with user context is its own tested skill: inject known facts about the user or session (name, plan tier, region, prior orders, conversation history) into the prompt template as variables so responses are personalized and the model doesn't ask for what the system already knows. Retrieved chunks are injected the same way — the prompt template has slots for {context} and {question}, and the chain fills both at runtime.
"The chatbot asks users for information the application already has" → augment the prompt with user/session context. "The chatbot invents facts about company policy" → augment with retrieved context and add a grounding instruction. Both are prompt-augmentation answers, but the source of the augmentation differs.
Before formal metrics, you must eyeball outputs against requirements: is the response accurate (matches the source), grounded (no claims beyond the context), complete, appropriately formatted, on-tone, and safe (no toxicity, no PII leakage, no harmful instructions)? Exam questions present a model response next to the retrieved context and ask what's wrong — the classic planted flaws are a hallucinated detail not present in the context, a leaked internal note, or an unsafe suggestion. Practicing this "spot the defect" read is worth real points.
Development is a loop: evaluate → diagnose → adjust. Map symptoms to chunking fixes: low recall@k with answers "not found" though they exist in the corpus → chunks may be too large (diluted embeddings) or the embedding model truncating them → reduce chunk size or switch strategy; retrieved chunks relevant but answers lack surrounding detail → increase chunk size/overlap or adopt parent-document retrieval; retrieval fine on some document types but broken on others → structure-aware chunking per type. The tested skill is choosing the next experiment from evaluation evidence rather than guessing.
Guardrails constrain what goes into and comes out of the model. The layers, from simplest to most robust:
A robust production app layers several: prompt instructions + AI Gateway PII/safety filters + output validation in the chain's post-processing.
Choosing the model is a recurring scenario. The attributes that decide it:
For metric-based selection, compare candidates on benchmarks relevant to the task (general reasoning vs. code vs. multilingual) and, better, on your own evaluation set with mlflow.genai.evaluate() — the exam prefers "run both models against the curated evaluation set and compare judge scores" over "pick the one with the highest leaderboard score." When browsing a model hub (Hugging Face, Databricks Marketplace), read the model card: intended use, training data, context length, license, evaluation results, and limitations. A model whose card says "research use only" is disqualified for a commercial app regardless of quality.
Model selection is requirements-driven: shortlist by hard constraints (license, context length, tool-calling, deployment mode), then decide by measured quality on your own evaluation data at acceptable cost and latency.
The embedding model determines retrieval quality and imposes the chunk-size ceiling. Selection criteria: max input tokens (chunks beyond it are truncated — the most-tested fact; pick a long-input embedding model or shrink chunks), embedding dimension (higher can be more expressive but costs more storage/compute per query), domain/language fit, and retrieval benchmark performance (MTEB). Databricks serves embedding models through Foundation Model APIs (e.g., the GTE and BGE families), and the same embedding model must embed both the documents and the queries — mixing models breaks similarity search.
One embedding model, two jobs: index-time and query-time. If you change it, re-embed and rebuild the index — and re-check that your chunk size still fits its context length.
The Mosaic AI Agent Framework is Databricks' library for authoring, logging, and deploying production agents, built on MLflow:
ResponsesAgent is the current recommended interface, superseding ChatAgent), which standardize the request/response schema so any UI, evaluation harness, and serving endpoint can talk to it. Wrap any orchestration library inside — LangChain/LangGraph, OpenAI SDK, or plain Python.mlflow.models.set_model(agent) in the file; mlflow.pyfunc.log_model(python_model="agent.py", ...)) rather than pickling objects.mlflow.genai.evaluate()) → register to Unity Catalog → agents.deploy() → collect feedback via the Review App → monitor in production. Evaluation vs. monitoring is a lifecycle distinction the exam tests directly: evaluation is pre-deployment, offline, against curated datasets with ground truth; monitoring is post-deployment, continuous, on live traffic without ground truth (Part 6). The loop closes when monitoring findings become new evaluation cases."The deployed agent gets permission errors calling the vector index" → the fix is declaring the index in the resources parameter when logging, enabling automatic auth passthrough — not embedding personal access tokens in code.
Genie Spaces provide conversational analytics over curated structured data: a Genie Space is scoped to specific Unity Catalog tables with instructions and example queries, and answers natural-language questions by generating SQL. In a multi-agent system, Genie becomes the structured-data specialist: a supervisor/orchestrator agent routes document questions to a RAG agent and data questions ("What were Q2 returns by region?") to a Genie Space. Programmatic integration uses the Genie Conversational API (start conversation → send message → poll for the SQL result/answer), which lets a custom agent — or the Agent Bricks Multi-Agent Supervisor — call Genie as a tool. This "RAG agent + Genie agent under a supervisor" pattern is the blueprint's flagship multi-agent scenario; recognize it whenever a scenario needs both unstructured documents and warehouse tables answered through one interface.
This domain takes the built chain to production: packaging, registration, vector search infrastructure, serving, batch inference, CI/CD, MCP, and user interfaces.
MLflow pyfunc (mlflow.pyfunc.PythonModel) is the universal wrapper when you need custom logic around the model. Implement load_context() (initialize clients, load the chain) and predict() (the request path). Pre-processing lives before the LLM call — input validation, prompt formatting, query rewriting; post-processing after — parsing, citation formatting, output filtering. Purpose-built flavors (mlflow.langchain.log_model for LangChain chains) capture dependencies automatically; pyfunc is the answer when the logic doesn't fit a standard flavor or needs explicit custom pre/post steps. Simple-chain coding questions show a snippet with a blank — know the LCEL pipe pattern from Part 3 and the log-then-load flow (mlflow.langchain.log_model(chain, ...) → mlflow.pyfunc.load_model(uri).predict(...)).
The elements of a logged RAG application the exam names explicitly: the model flavor (langchain/pyfunc), the embedding model reference, the retriever/vector index dependency, and the model signature — the declared input/output schema, typically inferred from an input_example. A signature is required for Unity Catalog registration and lets Model Serving validate requests.
Unity Catalog is the model registry: mlflow.set_registry_uri("databricks-uc"), then register with a three-level name — catalog.schema.model_name — either at logging time (registered_model_name=) or with mlflow.register_model(model_uri, name). Each registration creates a new version; aliases like @champion point deployments at the blessed version so promotion is an alias move, not a code change. UC registration brings access control, lineage, and cross-workspace sharing — always the "governed deployment" answer over the legacy workspace registry.
Mosaic AI Vector Search is the platform's serverless vector database, governed by Unity Catalog. Architecture: a Vector Search endpoint (the compute serving one or more indexes) plus indexes. The index types decide most questions:
| Index type | Embeddings | Sync | When to use |
|---|---|---|---|
| Delta Sync — managed embeddings | Databricks computes them from a text column using an embedding endpoint you name | Automatic from the source Delta table (Triggered or Continuous) | Default: simplest pipeline, text lives in Delta |
| Delta Sync — self-managed embeddings | You precompute an embedding column in the Delta table | Automatic from the source table | You need a custom/fine-tuned embedding model but still want auto-sync |
| Direct Vector Access | You supply vectors | None — you upsert/delete via API | Full manual control; data isn't in a Delta table or you manage updates yourself |
Delta Sync requires the source table to have Change Data Feed enabled and a primary key. Sync frequency: Triggered syncs on demand/schedule (cheaper, staleness between syncs); Continuous streams changes within seconds (fresher, always-on cost). Query with index.similarity_search(query_text=..., columns=[...], num_results=k, filters={...}) — pass query_text for managed embeddings or query_vector if you embedded the query yourself; query_type="hybrid" adds keyword matching to semantic search. Configuration trade-offs the exam probes: managed vs. self-managed embeddings (simplicity vs. control), Triggered vs. Continuous (cost vs. freshness), and endpoint sizing for latency vs. cost.
"Documents updated in the Delta table aren't appearing in search results" → the index is Triggered and hasn't synced (run a sync or switch to Continuous), or Change Data Feed isn't enabled on the source table.
Mosaic AI Model Serving exposes everything as REST endpoints. Three routes to an LLM:
| Serving mode | What it is | Choose when |
|---|---|---|
| Foundation Model APIs — pay-per-token | Shared endpoints for curated models (Llama, Claude, GTE/BGE embeddings…), billed per token, zero setup | Prototyping, variable/low traffic, instant access |
| Foundation Model APIs — provisioned throughput | Dedicated capacity for a foundation model (including fine-tuned weights), guaranteed performance, compliance-ready | Production workloads needing throughput/latency guarantees or fine-tuned models |
| External models | Endpoint proxies a third-party API (OpenAI, Anthropic…) with centralized keys and AI Gateway governance | Standardize access + governance over outside providers |
Custom chains/agents deploy as custom model serving endpoints from their UC-registered model (via the UI, API, or agents.deploy() for Agent Framework agents — which also provisions the Review App and inference tables). Endpoint access control: serving endpoints are ACL-governed — Can Query to invoke, Can Manage to edit — grant query rights to the app's service principal, and keep secrets in Databricks Secrets referenced through environment variables, never hardcoded.
For scoring whole tables rather than serving interactive traffic, use the SQL function ai_query():
SELECT id, ai_query('databricks-meta-llama-3-3-70b-instruct',
CONCAT('Summarize: ', review_text)) AS summary
FROM catalog.schema.reviews;
It runs the model over every row with automatic batching and parallelism, works with Foundation Model APIs and custom endpoints, and supports structured output via a response format. Batch cue words — "process 10 million rows nightly," "backfill summaries for the whole table" — mean ai_query(), not a REST loop over a real-time endpoint.
Production apps need durable state beyond the stateless endpoint: Delta tables in Unity Catalog for chunks, evaluation sets, logs, and extracted results; inference tables for request/response capture; and an OLTP-style store (e.g., Lakebase, Databricks' managed Postgres) when the app needs low-latency transactional reads/writes such as per-user chat history or session state. The design principle the exam rewards: keep all durable data governed in Unity Catalog rather than in ad-hoc external stores.
Treat the GenAI app like software: code in Git, deployments via Databricks Asset Bundles or similar automation, environments separated (dev → staging → prod). The blueprint names three specifics:
mlflow.genai.evaluate() against the curated evaluation set in CI and block promotion if judge scores regress below thresholds.The MLflow Prompt Registry versions prompts in Unity Catalog: mlflow.genai.register_prompt(name="catalog.schema.qa_prompt", template=...) creates immutable versions; applications load by version or alias (prompts.load_prompt("prompts:/catalog.schema.qa_prompt@production")). This decouples prompt iteration from code deployment — a prompt improvement ships by registering a new version and moving the alias, with full history and rollback. Hardcoded prompt strings scattered through code is the anti-pattern the exam wants replaced.
The Model Context Protocol (MCP) is the open standard for connecting agents to tools and data over a uniform interface. Databricks support comes in three forms:
MCP standardizes agent-to-tool wiring: managed servers for Databricks-native tools (Vector Search, UC functions, Genie), external for third-party services, custom (hosted on Databricks Apps) for everything else — all governed through Unity Catalog.
Getting the agent in front of users: Databricks Apps hosts custom web UIs (Streamlit, Gradio, Dash, React) on the platform with built-in auth and governance — the default answer for a bespoke chat frontend. The AI Playground is for interactive development-time testing, and the Review App for SME feedback — neither is an end-user production UI. For meeting users where they work, integrate the serving endpoint with Slack or Microsoft Teams bots that relay messages to the agent endpoint.
Small domain, direct questions: guarding inputs, masking sensitive data, and respecting data licenses.
PII protection applies at three points: before indexing — scrub or pseudonymize PII during data prep so it never enters the vector index (once embedded, it can surface in any retrieval); at the endpoint — AI Gateway guardrails detect PII in requests and responses and can block or mask it with no application code; in the data layer — Unity Catalog column masks and row filters keep governed tables masked for unauthorized readers, which flows through to anything an agent queries. Defense in depth is the expected answer: don't index what you don't need, and mask at the gateway anyway.
"Prevent the chatbot from ever revealing customer emails, without changing application code" → AI Gateway PII guardrail on the serving endpoint. "Ensure PII never enters the RAG index at all" → scrub during data preparation, before embedding.
Threats to recognize: prompt injection (input that overrides system instructions — "ignore previous instructions and…"), jailbreaking (coaxing the model past its safety training), data exfiltration via prompts (tricking the model into dumping system prompts or retrieved context wholesale), and abusive/off-topic flooding. Mitigations, layered: separate user input from instructions with delimiters and explicit "treat the following as data, not instructions" framing; input classification/screening before the LLM (safety classifiers such as Llama Guard, keyword and topic filters); AI Gateway safety guardrails and invalid-keyword/topic moderation at the endpoint; least-privilege tools (an agent that can only call read-only UC functions limits blast radius); and rate limiting to blunt automated abuse.
Before data enters a GenAI pipeline, verify you have the right to use it: dataset and content licenses (Creative Commons variants — beware NC "non-commercial" and ND "no-derivatives" clauses for a commercial app), terms of service of scraped websites, copyright in third-party documents, and model licenses (some open-weight models restrict commercial use or require attribution — read the model card and license, e.g., research-only checkpoints are off-limits in production). Contractual and regulatory constraints (customer data-processing agreements, GDPR-style regulations) govern whether user data can be used for evaluation or tuning. Mitigating problematic text ties the domain together: filter toxic, biased, or infringing content out of training/index corpora during prep, and pair that with generation-side guardrails so problematic text neither enters nor leaves the system.
Licensing questions are pass/fail simple: "non-commercial license + commercial product = cannot use." Don't overthink them — the exam wants you to spot the incompatibility, not litigate it.
Evaluation proves quality before launch; monitoring watches it after. This domain covers the MLflow evaluation stack, tracing, inference tables, and AI Gateway operations.
When given measurements — latency percentiles, tokens/sec, cost per 1K requests, judge quality scores — pick the smallest/cheapest model that meets the quality bar and latency SLA. A 70B model scoring 0.92 vs. an 8B scoring 0.90 on your evaluation set at one-eighth the cost usually loses to the 8B unless the scenario privileges quality above all. This is the "metrics decide, not model prestige" principle, and it also drives cost controls: right-size the model, cap output tokens, batch offline work through ai_query(), use pay-per-token until traffic justifies provisioned throughput, and apply AI Gateway rate limits so a runaway client can't burn the budget.
MLflow Tracing records each request as a tree of spans — retriever call, each tool invocation, the LLM call with its exact prompt and completion, latencies and token counts. Enable it with autologging (mlflow.langchain.autolog()) or the @mlflow.trace decorator on custom code. Traces are the raw material for everything else: debugging ("which step produced the hallucination?"), retrieval evaluation (judges read the retriever span), and production monitoring (traces from live endpoints land in inference tables). "Scoring" is attaching assessments to traces — automated judge scores or human feedback — so quality is queryable per request.
The evaluation harness: build an evaluation dataset (inputs, optionally expected_response/expected_facts ground truth), then run:
results = mlflow.genai.evaluate(
data=eval_dataset,
predict_fn=my_agent,
scorers=[Correctness(), RelevanceToQuery(), Safety(), Guidelines(...)]
)
LLM judges are the built-in scorers: ground-truth judges (Correctness) require the expected answer in the dataset and check the response against it; reference-free judges (RelevanceToQuery, Safety, groundedness against retrieved context, retrieval/chunk relevance) need no labels. Guidelines judges check plain-language rules ("must not mention competitors"). When built-ins don't fit, write a custom Scorer with the @scorer decorator — a Python function receiving inputs/outputs/expectations (and the trace) and returning a score or Feedback; custom scorers can be deterministic (regex, exact match, latency thresholds) or wrap their own LLM-judge prompt.
SME feedback closes the human loop: domain experts review responses in the Review App, guided by rubrics (explicit criteria — accuracy, completeness, tone — rated on a defined scale so feedback is consistent and actionable); their labels become ground truth for future mlflow.genai.evaluate() runs and for aligning judges with expert judgment.
Match judge to data availability: you have curated correct answers → ground-truth judge (Correctness). You have no labels but want hallucination detection → groundedness against retrieved context. You have a business rule in plain English → Guidelines. Nothing built-in fits → custom Scorer.
Evaluate before deployment on curated data with ground truth; monitor after deployment on live traffic without it. Same scorers can run in both worlds — that symmetry (offline judges rerun as scheduled production scorers) is the platform's headline design.
Inference tables auto-capture every request and response payload from a serving endpoint into a Delta table in Unity Catalog — the foundation of production observability. For RAG/agent endpoints, captured traces include retrieval spans, so you can log which chunks were retrieved for every production query and mine failures. Agent Monitoring builds on this: attach scorers/judges to a deployed agent so a sample of live traffic is continuously scored (groundedness, safety, guideline adherence), with dashboards over quality, latency, token usage, and request volume, and alerts on degradation. Monitoring metrics worth naming: request volume, error rate, latency percentiles, token consumption/cost, judge quality scores over time, and user feedback rates.
Mosaic AI Gateway wraps serving endpoints (including external models) with centralized operations — enable features per endpoint, no app changes:
If the requirement is centralized, code-free governance of model traffic — usage attribution, logging, PII/safety filtering, rate limits, provider fallback — the answer is AI Gateway on the serving endpoint.
Design (14%)
Data Prep (14%)
Application Development (30%)
ChatDatabricks + DatabricksVectorSearch; UC functions as tools.agents.deploy().Assembling & Deploying (22%)
databricks-uc, catalog.schema.model, aliases for promotion.ai_query().mlflow.genai.evaluate().Governance (8%)
Evaluation & Monitoring (12%)
ai_query().@mlflow.trace; traces power debugging, evaluation, monitoring.@scorer for custom.mlflow.genai.evaluate(), scorers, judges), tracing, and the Prompt Registry.Good luck — build one real pipeline, break it, fix it with the evaluation loop, and the exam will feel like a walkthrough of your own project.