Machine Learning Operations Engineer Associate (AI-300) — Complete Study Guide

Exam Code: AI-300 · Learn → Practice → Apply · Build Smart. Scale Fast.

AI-300, Microsoft Certified: Machine Learning Operations Engineer Associate, is the successor to DP-100 (Designing and Implementing a Data Science Solution on Azure), which retired on June 1, 2026. The change is more than a renumbering. Where DP-100 centred on the data scientist building and training models, AI-300 targets the operations engineer who takes models from experiment to production — the shift the exam is built around.

The most important shift to internalise before you study anything else: roughly half of AI-300 (45–55%) is GenAIOps on Microsoft Foundry. Domains 3, 4, and 5 — Foundry infrastructure, generative AI quality assurance and observability, and generative AI optimization — together carry 52% of the weighted content. If you prepared for DP-100 in the past, the classical ML material in Domains 1 and 2 will feel familiar, but you cannot pass AI-300 on classical MLOps knowledge alone. Foundry projects, serverless model deployment, provisioned throughput units, prompt versioning, AI-assisted evaluation metrics, and RAG optimization are first-class exam subjects.

The exam went to beta in March 2026 and is now generally available. It is currently offered in English only.

Weightings on the official skills outline are given as ranges (for example, 25–30% for Domain 2). This guide uses midpoint-style single figures — 19%, 29%, 24%, 14%, 14% — so you can budget study time proportionally. Domain 2 is the single largest domain; Domains 3–5 combined are the largest theme.

How to Use This Guide

Each Part maps one-to-one to an official skills area, in official order, and covers every topic listed in the skills outline for that domain. Throughout you will find four kinds of callout:

Work through each Part, practise the CLI/Bicep/YAML shapes shown (the exam expects you to recognise correct structure, flags, and schema keys), then re-read only the callouts the day before the exam.

Part 1 · Design and Implement an MLOps Infrastructure (19%)

This domain covers the foundation everything else stands on: the Azure Machine Learning workspace and the assets inside it, identity and access, shared registries, and — critically for an operations exam — deploying all of it as code with Bicep, the Azure CLI, and GitHub Actions, behind appropriate network restrictions.

1.1 Creating and Managing an Azure ML Workspace

The workspace is the top-level resource for Azure Machine Learning. It groups jobs, models, endpoints, environments, components, data assets, and compute, and it carries the security boundary (identity, networking, encryption). A workspace depends on four associated resources, created automatically if you don't supply them:

• Azure Storage accountdefault datastore for artifacts, snapshots, and job outputs.
• Azure Key Vaultsecrets, connection strings, and keys used by the workspace.
• Azure Container Registry (ACR)environment images built for training and inference (created lazily on first image build if not pre-provisioned).
• Application Insightstelemetry for deployed endpoints.

You can create a workspace through the Azure portal, the Python SDK v2, ARM/Bicep templates, or the CLI:

az extension add -n ml
az ml workspace create \
  --name mlw-prod \
  --resource-group rg-mlops \
  --location eastus

1.2 Datastores and Data Assets

A datastore is a reference to an Azure storage service plus the credentials to reach it — it stores connection information, not data. Supported services include Azure Blob Storage, Azure Data Lake Storage Gen2, and Azure Files. Credentials (account key, SAS token, or identity-based access) live in the workspace Key Vault.

A data asset is a versioned pointer to specific data, referenced by name and version in jobs. Data assets come in three types:

• •
  uri_file — a single file.
• •
  uri_folder — a folder of files.
• •
  mltable — a schematised, tabular abstraction defined by an MLTable file (required for AutoML training data).

# data-asset.yml
$schema: https://azuremlschemas.azureedge.net/latest/data.schema.json
name: taxi-training-data
version: 3
type: mltable
path: azureml://datastores/adls_training/paths/taxi/2026-06/

Hold the distinction firmly: a datastore is where and how to connect (credentials, service); a data asset is what data, which version. Jobs consume data assets; data assets resolve through datastores.

1.3 Compute Targets

Know the four compute categories and when each is the right answer:

• Compute instancea single-node, per-user development VM for notebooks and interactive work. Not for production training at scale; one user at a time.
• Compute clustermulti-node, autoscaling training compute. Scales from min_instances (set 0 to avoid idle cost) to max_instances; supports dedicated and low-priority VMs.
• Serverless computeon-demand, Microsoft-managed compute for jobs; no cluster to create or manage. You specify instance type and count per job.
• Kubernetes (attached AKS/Arc)bring-your-own Kubernetes for training or inference when you need full control or on-premises/hybrid placement.

For managed inference you'll use managed online endpoints (real-time) and batch endpoints with compute clusters or serverless compute behind them — covered in Part 2.

Scenario mapping to practise: "minimise cost for an intermittently used training cluster" → compute cluster with min_instances: 0; "run a job without managing any compute at all" → serverless compute; "data scientist needs an interactive notebook environment" → compute instance.

1.4 Workspace IAM: Roles and Identity

Azure ML uses Azure RBAC. The built-in roles you must be able to distinguish:

Workspaces and compute resources carry managed identities (system-assigned or user-assigned) used to reach the storage account, Key Vault, and ACR without embedded credentials. Identity-based datastore access means the identity of the compute or the user, not a stored key, is authorised on the storage account (for example via the Storage Blob Data Reader role).

Role-selection shorthand: submit jobs but not create clusters → AzureML Data Scientist; add cluster management only → also AzureML Compute Operator. Reach for custom roles only when built-ins genuinely can't express the requirement.

1.5 Environments and Components

An environment captures the software context of a job or deployment: Docker base image plus a conda/pip dependency specification. Curated environments are Microsoft-maintained and ready to use; custom environments are versioned assets you define and Azure ML builds into images stored in the workspace ACR.

A component is a reusable, versioned, self-contained pipeline step — name, version, typed inputs/outputs, environment, and the command it runs. Components are the unit of reuse across pipelines and teams:

# component: prep-data
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
name: prep_data
version: 1.2.0
type: command
inputs:
  raw_data:
    type: uri_folder
outputs:
  prepped_data:
    type: uri_folder
environment: azureml:sklearn-env:4
command: >-
  python prep.py --raw ${{inputs.raw_data}} --out ${{outputs.prepped_data}}

1.6 Cross-Workspace Sharing with Registries

An Azure ML registry is an organisation-level catalogue for sharing models, environments, components, and data assets across workspaces — including workspaces in different regions and subscriptions. The canonical MLOps pattern: train and validate in a dev workspace, publish the approved model/component to the registry, and have staging and production workspaces consume from the registry (azureml://registries/<registry-name>/models/<name>/versions/<v>).

1.7 Infrastructure as Code: Bicep and the Azure CLI

AI-300 expects you to recognise correct Bicep for provisioning ML infrastructure. A minimal workspace with dependencies:

resource stg 'Microsoft.Storage/storageAccounts@2023-05-01' = {
  name: 'stmlopsprod'
  location: location
  sku: { name: 'Standard_LRS' }
  kind: 'StorageV2'
}

resource mlw 'Microsoft.MachineLearningServices/workspaces@2024-10-01' = {
  name: 'mlw-prod'
  location: location
  identity: { type: 'SystemAssigned' }
  properties: {
    storageAccount: stg.id
    keyVault: kv.id
    applicationInsights: appi.id
    containerRegistry: acr.id
    publicNetworkAccess: 'Disabled'
  }
}

Deploy with az deployment group create --template-file main.bicep --resource-group rg-mlops. Compute, too, can be declared in Bicep (as child computes resources) or created with az ml compute create.

Know which layer each tool owns: Bicep/ARM provisions Azure resources (workspace, storage, networking, compute), while the az ml CLI v2 with YAML manages workspace assets and operations (jobs, data assets, environments, components, endpoints, deployments). Questions often hand you a task and four tools — match the layer.

1.8 GitHub Integration and CI/CD Provisioning with GitHub Actions

Git is the system of record: training code, pipeline YAML, environment definitions, and infrastructure templates all live in the repository; the workspace clones/snapshots code at job submission. Compute instances support Git integration for development, but the operational flow is repo-driven.

For automation, GitHub Actions authenticates to Azure with OpenID Connect (OIDC) federated credentials — no long-lived secrets stored in GitHub:

# .github/workflows/provision.yml
permissions:
  id-token: write
  contents: read
jobs:
  provision:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - name: Deploy infrastructure
        run: |
          az deployment group create \
            --resource-group rg-mlops \
            --template-file infra/main.bicep
      - name: Submit training pipeline
        run: az ml job create -f pipelines/train.yml -w mlw-prod -g rg-mlops

1.9 Network Restrictions

Locking down a workspace involves three moving parts:

• Managed virtual network (managed VNet)Azure ML provisions and manages an isolated network for the workspace and its managed computes. Two isolation modes: allow_internet_outbound and allow_only_approved_outbound (the stricter mode requires explicit outbound rules, e.g., private endpoints to your data services).
• Private endpointsprivate IP access to the workspace (and to storage, Key Vault, ACR) from your VNet; pair with publicNetworkAccess: Disabled to remove public exposure entirely.
• No public IP computeclusters and instances inside the managed VNet can be created without public IPs.

Part 2 · Implement Machine Learning Model Lifecycle and Operations (29%)

The heaviest domain on the exam: training and tracking with MLflow, AutoML, hyperparameter tuning, distributed training, pipelines, model registration and lifecycle, responsible-AI evaluation, real-time and batch endpoints, progressive rollout, and drift-triggered retraining.

2.1 Experiment Tracking with MLflow

Azure ML workspaces are MLflow-compatible tracking servers. You track experiments with the standard MLflow API — mlflow.log_param, mlflow.log_metric, mlflow.log_artifact — and with autologging (mlflow.autolog() or framework-specific variants such as mlflow.sklearn.autolog()), which automatically captures parameters, metrics, and the model itself for supported frameworks.

Key facts to hold onto:

• •
  Jobs submitted to Azure ML are automatically configured for MLflow tracking; from outside (e.g., local runs), set the tracking URI to the workspace's MLflow endpoint.
• •
  Models saved in MLflow format carry the MLmodel file, environment specification, and signature — which is what enables no-code deployment later.
• •
  Runs are queryable across experiments with mlflow.search_runs(), which returns a DataFrame you can sort by metric.

2.2 AutoML and Notebook Exploration

Automated ML explores algorithms, preprocessing, and hyperparameters for a task type (classification, regression, forecasting, plus vision and NLP tasks). Operationally, know:

• •
  Training data must be an mltable data asset.
• •
  You configure the task, target column, primary_metric to optimise, and limits — timeout_minutes, trial_timeout_minutes, max_trials, and enable_early_termination.
• •
  You can restrict the search with allowed_training_algorithms / blocked lists, and enable ONNX-compatible models.
• •
  Every trial is logged like any other run, so the leaderboard is comparable in the studio and via MLflow.

Notebooks on compute instances are for exploration; the exam's operational stance is that exploratory notebook code gets refactored into scripts and components before it enters a pipeline.

2.3 Hyperparameter Tuning with Sweep Jobs

A sweep job automates hyperparameter search over a command job. You define:

• Search spacediscrete (choice) and continuous (uniform, loguniform, normal) distributions.
• Samplinggrid (exhaustive, discrete only), random, or bayesian (learns from prior trials).
• Objectivethe metric name your script logs, and goal: maximize or minimize.
• Early termination policybandit (slack factor/amount from the best run), median_stopping (below running median), or truncation_selection (cancel bottom X%).

type: sweep
trial: azureml:train_component:1
search_space:
  learning_rate:
    type: loguniform
    min_value: -6.9
    max_value: -2.3
sampling_algorithm: bayesian
objective:
  primary_metric: val_auc
  goal: maximize
limits:
  max_total_trials: 40
  max_concurrent_trials: 4
early_termination:
  type: bandit
  slack_factor: 0.1
  evaluation_interval: 2

2.4 Training Scripts and Distributed Training

Production training runs as a command job: a script, an environment, a compute target, and typed inputs/outputs. For large models and datasets, AI-300 covers distributed training configuration:

• PyTorchdistribution: { type: pytorch, process_count_per_instance: N } for native DDP.
• MPIdistribution: { type: mpi } for Horovod-style workloads.
• TensorFlowdistribution: { type: tensorflow, worker_count: N }.

Set resources: { instance_count: M } for multi-node runs; Azure ML wires up the environment variables (ranks, world size) for you. Data parallelism (replicate model, shard data — the common case) versus model parallelism (shard the model itself, for models too large for one device) is a tested conceptual distinction.

2.5 Training Pipelines

A pipeline job chains components into a DAG. Outputs of one step bind to inputs of the next; steps run in dependency order and independent steps run in parallel. Pipelines support per-step compute and reuse of cached step results when inputs and code are unchanged.

$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
inputs:
  raw: 
    type: uri_folder
    path: azureml:taxi-training-data:3
jobs:
  prep:
    type: command
    component: azureml:prep_data:1.2.0
    inputs:
      raw_data: ${{parent.inputs.raw}}
  train:
    type: command
    component: azureml:train_model:2.0.1
    inputs:
      training_data: ${{parent.jobs.prep.outputs.prepped_data}}

Pipelines are scheduled with schedules (cron or recurrence triggers on a pipeline job) — the standard mechanism for periodic retraining.

2.6 Comparing Models Across Jobs

Cross-job comparison is an explicit skill: use the studio's experiment view to chart metrics across runs, or mlflow.search_runs() to pull runs from one or more experiments into a DataFrame and rank by the primary metric. Consistent metric names across training scripts are what make this possible — an operational convention worth stating in design questions.

2.7 Feature Retrieval Specification Packaging

With managed feature store, features are defined once (feature sets on source data with transformations) and consumed consistently in training and inference. The feature retrieval specification (feature_retrieval_spec.yaml) lists the feature sets and features a model consumes. It is generated when you select features for training and — critically — is packaged with the model artifact so that online/batch inference retrieves exactly the same features, eliminating training/serving skew.

2.8 Model Registration and MLflow Models

Register models to make them versioned, deployable assets. Three model types: mlflow_model (preferred — enables no-code deployment), custom_model (arbitrary files; you supply scoring script + environment), and triton_model. Register from a completed job's output to preserve lineage:

az ml model create --name churn-model --version 4 \
  --type mlflow_model \
  --path azureml://jobs/<job-id>/outputs/artifacts/model

2.9 Responsible-AI Evaluation

Before promotion, models are evaluated with the Responsible AI dashboard, generated by a pipeline of RAI components (constructor → tool components → gather). Components include error analysis (where the model fails), model explanations/interpretability (global and local feature importance), fairness assessment (performance across sensitive cohorts), counterfactual what-if analysis, and causal analysis. The RAI scorecard exports a shareable PDF summary for stakeholders and governance gates.

2.10 Model Lifecycle and Archiving

Model versions accumulate; lifecycle hygiene is tested. Archiving a model (or a whole model container) hides it from listings and blocks new deployments while preserving history; archived assets can be restored. Stages/labels (dev → staging → production) are represented with tags and registry promotion. Deleting is restricted when deployments reference the version — archive first, retire deployments, then clean up.

2.11 Real-Time and Batch Endpoints

The deployment split is one of the highest-yield topics on the exam:

An endpoint is the stable URI + auth; a deployment is a model + environment + compute behind it. One endpoint hosts multiple deployments with traffic split percentages. MLflow models get no-code deployment (no scoring script or environment needed); custom models require code_configuration (scoring script with init() and run()) plus an environment.

2.12 Progressive Rollout and Safe Rollback

The blue-green pattern on managed online endpoints:

1. 1.
   blue deployment serves 100% of traffic.
2. 2.
   Create green with the new model version — 0% traffic.
3. 3.
   Test green directly (you can invoke a specific deployment by name, or use mirrored/shadow traffic to copy a percentage of live traffic to green without returning its responses).
4. 4.
   Shift traffic gradually: az ml online-endpoint update --traffic "blue=90 green=10", then 50/50, then 0/100.
5. 5.
   Rollback = set traffic back to blue=100 instantly. Delete blue only when confident.

2.13 Drift Detection and Retraining Triggers

Model monitoring in Azure ML watches production signals: data drift (feature distributions shift vs. the reference/training baseline), prediction drift (output distribution shifts), data quality (nulls, type mismatch, out-of-range), and feature attribution drift (importance rankings change). Monitors compare production inference data (via built-in data collection on online endpoints) against a reference dataset, on a schedule, with per-signal metrics and thresholds.

When a threshold trips, the monitor raises an alert (Azure Monitor / email), and the operational pattern is to trigger the retraining pipeline — via an Event Grid–driven automation or a scheduled pipeline gated on monitor results — then flow the retrained model through registration, evaluation, staged rollout.

Part 3 · Design and Implement a GenAIOps Infrastructure (24%)

Here the exam pivots to Microsoft Foundry (the evolution of Azure AI Studio / Azure AI Foundry): provisioning Foundry resources and projects as code, securing them, deploying foundation models on serverless APIs or managed compute, capacity planning with PTUs, and treating prompts as versioned engineering artifacts.

3.1 Foundry Resources and Project Environments

The Foundry resource is the top-level Azure resource for generative AI development; projects are child environments inside it where teams organise models, agents, connections, evaluations, and data for a use case. Projects provide isolation (per-project access, files, and history) while sharing the parent resource's deployments and quota. Connections attach external services — Azure AI Search, storage, other model providers — at the resource or project level.

Typical environment strategy mirrors classical MLOps: separate Foundry resources (or at minimum separate projects) for dev, staging, and production, provisioned identically from the same templates.

Terminology check: older content refers to "hub-based projects" (the hub + project pattern from Azure AI Studio). AI-300 is written against the current Foundry resource + projects model. If a question contrasts them, the Foundry project is the current, recommended type for new work — including agents and evaluations.

3.2 Managed Identities and RBAC for Foundry

Foundry resources use managed identities to reach dependent services (storage, Azure AI Search, Key Vault) without keys. RBAC roles to know:

• Azure AI Useruse the project: run deployed models, evaluations, agents; no resource management.
• Azure AI Project Managermanage projects and their assets, assign the Azure AI User role at project scope.
• Azure AI Account Ownerfull control at the account (Foundry resource) level, including creating projects and deployments.

Disable key-based auth and rely on Microsoft Entra ID tokens for production — questions about "eliminate API keys" resolve to Entra ID auth plus managed identity.

3.3 Private Networking for Foundry

The same zero-public-exposure pattern as Part 1, applied to Foundry: private endpoints on the Foundry resource, publicNetworkAccess: Disabled, private endpoints on connected services (Search, storage), and network-injected/managed VNet options for agent workloads. Data never traversing the public internet is the design requirement to recognise.

3.4 Deploying Foundry with Bicep and the CLI

Foundry resources are Microsoft.CognitiveServices/accounts with kind: 'AIServices'; projects and model deployments are child resources — so the whole GenAI stack is expressible in Bicep:

resource foundry 'Microsoft.CognitiveServices/accounts@2025-04-01-preview' = {
  name: 'fdy-prod'
  location: location
  kind: 'AIServices'
  sku: { name: 'S0' }
  identity: { type: 'SystemAssigned' }
  properties: {
    allowProjectManagement: true
    customSubDomainName: 'fdy-prod'
    publicNetworkAccess: 'Disabled'
  }
}

resource gpt 'Microsoft.CognitiveServices/accounts/deployments@2025-04-01-preview' = {
  parent: foundry
  name: 'gpt-4o'
  sku: { name: 'GlobalStandard', capacity: 50 }
  properties: {
    model: { format: 'OpenAI', name: 'gpt-4o', version: '2024-11-20' }
  }
}

CLI equivalents: az cognitiveservices account create and az cognitiveservices account deployment create. Either way, the exam point is that Foundry infrastructure is code, deployed by the same GitHub Actions OIDC pipelines as Part 1.

3.5 Model Deployment: Serverless API vs. Managed Compute

Two hosting models for foundation models, and choosing between them is a guaranteed exam topic:

3.6 Model Selection, Versioning, and Production Strategies

The model catalogue is where you compare candidates — by quality benchmarks, cost, throughput, and capability (context length, tool calling, modalities). Operational practices the exam rewards:

• •
  Pin model versions in production deployments; don't ride auto-upgrade defaults blind. Deployment update policies control when a deployed model version is upgraded or retired — track retirement dates.
• •
  Use deployment names as stable aliases (apps call chat-prod, which you repoint across model versions), enabling blue-green-style model swaps without client changes.
• •
  Evaluate a new model version against your evaluation suite (Part 4) before shifting traffic.

3.7 Provisioned Throughput Units (PTUs)

PTUs buy reserved, predictable model-processing capacity — a fixed throughput allocation with stable latency, versus standard (pay-as-you-go) deployments that share capacity and can throttle (429s) under load.

• •
  PTUs are purchased per model family/version; deployment types include regional, data-zone, and global provisioned.
• •
  Cost is fixed per PTU-hour regardless of usage — with monthly/yearly reservations dramatically cheaper than hourly for steady workloads.
• •
  Sizing: estimate from expected tokens-per-minute (input + output mix) using the capacity calculator; utilisation is observable via the Provisioned-Managed Utilization metric.
• •
  The spillover pattern: primary PTU deployment for baseline traffic, overflow routed to a standard deployment when PTU capacity is exceeded.

3.8 Prompt Design, Variants, and Comparison

Prompts are operational artifacts. In Foundry you author prompts with system messages, few-shot examples, and template variables, and create prompt variants — alternative phrasings, formats, or parameter settings (temperature, max tokens) for the same task. Variants are compared by running them over the same test dataset and scoring with evaluation metrics (Part 4) — not by eyeballing single responses.

3.9 Prompt Version Control in Git

The GenAIOps discipline the exam wants: prompts live in the repository — as prompt/template files (e.g., Prompty-style .prompty assets or YAML/JSON templates) — versioned, code-reviewed, and promoted through environments like any other artifact. A prompt change triggers the same CI evaluation gates as a code change: run the evaluation suite, compare scores against the baseline, block regression.

Part 4 · Implement Generative AI Quality Assurance and Observability (14%)

Domain 4 is how you know a generative system is good — before release (evaluation) and after (monitoring, tracing, cost).

4.1 Test Datasets and Data Mapping

Evaluations run over test datasets — typically JSONL, with fields such as query, response, context, and ground_truth. Data mapping binds your dataset's column names to the inputs each evaluator expects (e.g., map your question column to the evaluator's query input; ${data.question} / ${target.response} style bindings). You can evaluate a static dataset of pre-generated responses, or point the evaluation at a target (model/app) that generates responses on the fly.

Each evaluator needs specific fields: groundedness needs context (and query/response); relevance needs query and response; fluency needs only response. If an evaluation "fails to run," the first suspect is a missing or mis-mapped required field.

4.2 AI Quality Metrics: Groundedness, Relevance, Coherence, Fluency

The four core AI-assisted quality metrics — scored (typically 1–5, higher is better) by a judge LLM:

Complementary measures: similarity (vs. a ground-truth answer), classic NLP overlap metrics (F1, BLEU, ROUGE, METEOR) when ground truth exists, and RAG-specific retrieval quality scoring.

4.3 Risk and Safety Evaluations

Safety evaluators score defect rates across severity levels for hateful/unfair content, sexual content, violent content, self-harm, plus protected material and jailbreak/indirect prompt-injection vulnerability. They work by simulating adversarial interactions (AI red teaming / adversarial simulators can generate the attack datasets) and grading responses. Safety evaluation complements, not replaces, runtime content filtering — evaluation tells you how the system behaves under attack; filters block harm live.

4.4 Automated Evaluation Workflows: Built-In and Custom Metrics

Evaluations run in the Foundry portal, or programmatically with the Azure AI Evaluation SDK — which is what CI/CD needs. Built-in evaluators cover quality, safety, and RAG; custom evaluators come in two shapes:

• Code-baseda Python function/class scoring each row (exact-match, regex, business rules).
• Prompt-basedyour own judge prompt (e.g., a Prompty file) defining a bespoke rubric such as "brand-tone adherence, 1–5."

The operational pattern: evaluation suite runs in the pipeline on every prompt/model/retrieval change; results are logged to the project; the gate compares scores to the baseline and fails the build on regression.

In short: built-in metrics for universal qualities, custom evaluators for your domain's definition of good — and both wired into CI so no prompt or model change ships unevaluated.

4.5 Continuous Monitoring in Foundry

Post-deployment, monitoring closes the loop: operational dashboards track request volumes, latency, errors, and token consumption, and continuous evaluation samples production traffic and scores it with the same quality/safety evaluators used pre-release — surfacing degradation trends (e.g., groundedness sliding after a data source change). Alerts route through Azure Monitor / Application Insights.

4.6 Performance Metrics: Latency and Throughput

For LLM serving, know the specific latency vocabulary:

• TTFT (time to first token)dominates perceived responsiveness in streaming UIs.
• End-to-end latency and time between tokens / tokens-per-second generation rate.
• Throughputrequests per minute and tokens per minute (TPM), the unit quotas and PTUs are denominated in.

Levers: streaming (improves perceived latency, not total compute), max_tokens limits, smaller/faster model variants, PTUs for tail-latency stability, and prompt/context trimming.

4.7 Cost Tracking: Tokens and Resources

Generative cost is token-denominated: input tokens + output tokens, priced per model (output typically costs more). Tracking practices: capture per-request token usage from API responses, aggregate by app/feature/tenant, use Azure Cost Management with tags for resource-level views, and set budgets/alerts. Cost reducers the exam likes: prompt caching (discounted repeated prefixes), shorter system prompts and trimmed retrieved context, right-sizing the model per task (small model for classification, flagship for synthesis), batch/async processing where latency allows, and PTU reservations for steady load.

4.8 Logging, Tracing, and Debugging

Tracing instruments a GenAI app end-to-end — capturing each step (retrieval call, tool invocation, model call with prompt/response, token counts, durations) as spans. Foundry tracing is OpenTelemetry-based, viewable in the portal's trace view and exportable to Application Insights. This is how you debug a multi-step agent/RAG flow: find the span where the wrong context, tool output, or prompt entered the chain.

Part 5 · Optimize Generative AI Systems and Model Performance (14%)

The final domain assumes a working RAG or fine-tuned system and asks: how do you make it better — retrieval tuning, embeddings, hybrid search, disciplined A/B experimentation, and the full fine-tuning lifecycle.

5.1 Retrieval Optimization: Chunks, Thresholds, Strategies

Retrieval quality caps RAG quality. The tunables:

• Chunk size and overlapsmall chunks give precise matches but fragment context; large chunks preserve context but dilute relevance and burn prompt tokens. Typical starting range is a few hundred tokens with ~10–15% overlap; tune empirically against your evaluation set. Prefer structure-aware/semantic chunking (split on headings/paragraphs) over blind fixed-size splits.
• Similarity thresholds and top-kraising the minimum similarity score or lowering k cuts noise (helps groundedness) but risks missing evidence (hurts recall); loosening does the opposite. There is no universal value — the exam answer is "tune against evaluation metrics."
• Query strategiesquery rewriting/expansion, filtering on metadata, and semantic ranker re-scoring of initial results.

Think of it as the retrieval trade-off triangle: precision (threshold/top-k) vs. recall (looser retrieval) vs. token cost (chunk size × k). Every retrieval-tuning question is asking which corner the scenario is bleeding on.

5.2 Embedding Model Selection and Fine-Tuning

Embedding model choice drives retrieval ceiling: compare on dimensionality (storage/latency vs. expressiveness — e.g., text-embedding-3 models support shortened dimensions), max input tokens (must fit your chunk size), domain fit, multilingual coverage, and cost per million tokens. Fine-tuning or adapting embeddings on domain pairs (query ↔ relevant passage) helps when vocabulary is highly specialised and generic embeddings mis-rank. Critical operational rule:

5.3 Hybrid Search Optimization

In Azure AI Search, hybrid search runs keyword (BM25) and vector queries together and fuses rankings with Reciprocal Rank Fusion (RRF); adding semantic ranker on top re-ranks the fused list with a deep model. Keyword search wins on exact terms (part numbers, names, acronyms); vector search wins on paraphrase and concept matching; hybrid + semantic ranker is Microsoft's benchmark-backed default recommendation for RAG. Tuning: field weighting, filters, and evaluating retrieval with metrics (e.g., NDCG-style relevance judgments or RAG retrieval evaluators) rather than anecdotes.

5.4 RAG Evaluation and A/B Testing

Systematic comparison discipline:

1. 1.
   Fix an evaluation dataset (real user queries + ground truth where possible).
2. 2.
   Change one variable per experiment — chunking, k, threshold, embedding model, search mode, prompt.
3. 3.
   Run the evaluation suite (groundedness, relevance, retrieval score, plus latency and token cost) on both variants.
4. 4.
   Compare offline first; then A/B test online — route a traffic fraction to the variant, compare quality metrics via continuous evaluation plus behavioural signals (thumbs-up rate, deflection, task completion).
5. 5.
   Promote or roll back based on data; record the experiment.

5.5 Advanced Fine-Tuning Methods

When prompt engineering and RAG plateau, fine-tuning adjusts the model itself:

• Supervised fine-tuning (SFT) on prompt/completion pairs — the baseline method; on Azure it uses LoRA-style parameter-efficient (PEFT) adaptation, training small adapter weights instead of all parameters.
• Preference alignment (e.g., DPO — direct preference optimization)trains on chosen-vs-rejected response pairs to align tone/style/judgment where "correct" is preferential.
• Distillationuse a large teacher model's outputs (e.g., stored completions) to train a smaller, cheaper student model.

Decision heuristics: fine-tune for style, format, and behaviour; use RAG for knowledge that changes or must be cited. Data quality beats quantity — hundreds to a few thousand high-quality examples in the required JSONL chat format.

5.6 Synthetic Data for Fine-Tuning

When real examples are scarce, generate training data: use a strong model to produce candidate pairs from your documents (the Azure AI Evaluation SDK's simulator can generate query/response datasets from source text, including adversarial sets), then filter and human-review for quality, deduplicate, and hold out an untouched real-data evaluation set. Synthetic data risks: distribution narrowing and inherited teacher biases — always validate the fine-tuned model on real, non-synthetic evaluation data.

5.7 Monitoring Fine-Tuned Models

During training, watch training loss vs. validation loss (divergence = overfitting) and run post-training evaluation against the base model on the same suite — a fine-tune that doesn't beat its base model on your metrics doesn't ship. In production, fine-tuned deployments are monitored like any other (Part 4): continuous quality evaluation, safety monitoring (fine-tuning can shift safety behaviour — re-run safety evaluations after every training run), latency, and token cost. Watch for drift: as your domain evolves, the fine-tune's static behaviour ages — schedule periodic re-evaluation.

5.8 Fine-Tuned Model Lifecycle to Production

The lifecycle, end to end: curate dataset (JSONL, train/validation split) → submit fine-tuning job in Foundry → evaluate the resulting model vs. base and vs. current production on the fixed suite → register/version the model → deploy alongside the current model → progressive rollout with A/B comparison → promote or roll back → monitor continuously → retrain when evaluation or drift signals demand. Version everything: the dataset, the base model + version, hyperparameters, and evaluation results, so any deployed fine-tune is reproducible.

Note the hosting interaction: fine-tuned Azure OpenAI models deploy as standard/provisioned deployments (some models also as developer-tier for experimentation), and hosting fine-tunes may carry an hourly hosting cost on top of tokens — factor that into the "should we fine-tune or keep prompt-engineering" cost question.

Exam-Day Checklist

Work through this list the day before the exam. Every line maps to sections above.

• •
  I can list the four workspace-dependent resources (Storage, Key Vault, ACR, App Insights) and what each does.
• •
  I can pick the right compute (instance / cluster / serverless / Kubernetes) and the right RBAC role (Data Scientist vs. Compute Operator) for a scenario.
• •
  I know datastore vs. data asset, and the three data asset types (uri_file, uri_folder, mltable — AutoML needs mltable).
• •
  I can explain when to use a registry (cross-workspace sharing/promotion) vs. a workspace.
• •
  I can read a Bicep workspace/Foundry template and a GitHub Actions OIDC login workflow and spot what's wrong.
• •
  I know the managed VNet isolation modes and the full private-endpoint lockdown pattern.
• •
  MLflow: autologging, model format → no-code deployment, search_runs for cross-job comparison.
• •
  Sweep jobs: three sampling methods, three early-termination policies, and the grid/Bayesian constraints.
• •
  Distributed training: pytorch/mpi/tensorflow distribution blocks; data vs. model parallelism.
• •
  I can order the blue-green rollout steps and state the instant-rollback move (traffic back to blue).
• •
  Real-time vs. batch endpoint selection is automatic for me.
• •
  Drift: data vs. prediction vs. feature-attribution vs. data quality — and monitor → alert → retraining pipeline.
• •
  RAI dashboard components matched to verbs (explain, fairness, counterfactual, error analysis).
• •
  Serverless API vs. managed compute trade-offs, and when PTUs (plus spillover) win on cost/latency.
• •
  Foundry RBAC (AI User / Project Manager / Account Owner), managed identity, Entra-only auth.
• •
  Prompts in Git + evaluation gates in CI — the GenAIOps loop.
• •
  The four quality metrics — and that groundedness needs context and is the hallucination metric.
• •
  Safety evaluation categories and adversarial simulation vs. runtime content filtering.
• •
  TTFT vs. end-to-end latency; TPM; token-based cost levers (caching, trimming, right-sizing).
• •
  Tracing (OpenTelemetry → App Insights) for debugging agent/RAG flows.
• •
  Retrieval tuning trade-offs; re-embed the corpus when the embedding model changes.
• •
  Hybrid search = BM25 + vector + RRF (+ semantic ranker) — the recommended RAG default.
• •
  Fine-tune for behaviour, RAG for knowledge, distillation for cost; SFT/LoRA vs. DPO; synthetic data needs human review and real-data evaluation.
• •
  Time budget: ~50 questions in 120 minutes ≈ 2 minutes each; case studies come in blocks — bank time on short items.

Official Resources

• Official exam page and skills outline: learn.microsoft.com — search "AI-300 Machine Learning Operations Engineer Associate" for the current study guide (skills outline updated 2026; always re-check shortly before your exam date).
• Microsoft Learn training paths: the AI-300 learning paths covering Azure Machine Learning (CLI v2 / SDK v2), MLOps with GitHub Actions, and GenAIOps on Microsoft Foundry.
• Azure Machine Learning documentation: learn.microsoft.com/azure/machine-learning — workspaces, compute, jobs, pipelines, endpoints, monitoring, registries.
• Microsoft Foundry documentation: learn.microsoft.com/azure/ai-foundry — projects, model catalogue, deployments, PTUs, evaluation, tracing, fine-tuning.
• Azure AI Evaluation SDK reference: built-in evaluators, custom evaluators, simulators for synthetic and adversarial data.
• Microsoft Practice Assessment: free official practice questions for AI-300 on the exam page — take it at least twice: early to find gaps, and again in your final week.
• Exam sandbox: aka.ms/examdemo — rehearse the question formats (drag-and-drop, build list, case studies) so the interface costs you zero time on exam day.

Good luck — pace yourself, answer every question, and trust the weightings: master Domain 2 deeply, and give Domains 3–5 the half-of-the-exam respect they demand.