Your team already has pytest, CI runners, JUnit dashboards, and twenty habits built around them. DeepEval lets LLM feature tests live inside that world — G-Eval rubrics, relevancy and faithfulness metrics as pytest assertions — instead of in a separate eval platform nobody checks. Same dashboards, new test type.
Tier: Core · Workflow area: Testing AI systems — foundations
This playbook writes LLM feature tests as ordinary pytest functions with DeepEval metrics, runs them through your existing CI with JUnit XML output, and keeps the metric thresholds honest with a calibration habit.
pip install deepeval pytest in the feature repo; provider API key for the metric-judge model exported (e.g. OPENAI_API_KEY or DeepEval's Anthropic config).Write the first test exactly like an integration test — with an LLM metric at the end.
# tests/llm/test_support_summarizer.py
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from app.summarizer import summarize_ticket
from tests.llm.golden import load_cases
@pytest.mark.parametrize("case", load_cases("summarizer", strata=["happy_path", "known_failures"]))
def test_summary_relevancy(case):
output = summarize_ticket(case.input) # calls the real feature
tc = LLMTestCase(
input=case.input,
actual_output=output,
expected_output=case.reference_summary,
)
assert_test(tc, [AnswerRelevancyMetric(threshold=0.7, model="gpt-5.2")])
Everything is familiar: parametrize over golden cases, call the feature, assert. The only new element is that the final assertion is a graded metric instead of an equality check.
Encode judgment-shaped requirements as G-Eval metrics with explicit steps.
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams
professional_tone = GEval(
name="ProfessionalTone",
evaluation_steps=[
"Check the summary never repeats abusive language from the input",
"Check the summary does not editorialize about the customer",
"Penalize any promise of refunds or compensation",
],
evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
threshold=0.8,
)
Numbered evaluation steps beat a one-line rubric for the same reason checklists beat vibes: the judge model grades each step, and failures tell you which expectation broke. Keep deterministic requirements (length, schema, banned phrases) as plain Python asserts before the metric call — cheapest layer first, always.
Add RAG metrics where the feature retrieves.
from deepeval.metrics import FaithfulnessMetric, ContextualRelevancyMetric
tc = LLMTestCase(
input=case.question,
actual_output=answer.text,
retrieval_context=answer.chunks, # what the retriever actually returned
)
assert_test(tc, [
FaithfulnessMetric(threshold=0.85), # answer claims ⊆ context
ContextualRelevancyMetric(threshold=0.6), # retrieved chunks on-topic
])
Faithfulness is the hallucination gate: it checks that the answer's claims are supported by the retrieved context. A faithfulness failure with high relevancy means the model embellished; the reverse means the retriever failed — the metric pair localizes the bug, which is exactly what QA needs from a failing test.
Run through pytest with JUnit XML, like every other suite.
deepeval test run tests/llm/ --junit-xml=reports/llm-tests.xml
# or plain: pytest tests/llm/ --junitxml=reports/llm-tests.xml
tests/llm/test_support_summarizer.py::test_summary_relevancy[gs-0087] PASSED
tests/llm/test_support_summarizer.py::test_summary_relevancy[gs-0104] FAILED
AnswerRelevancyMetric: score 0.58 < threshold 0.7
Reason: summary omits the damaged-item issue raised in the ticket
The score-plus-reason failure output goes straight into the JUnit XML, so your existing dashboard shows LLM regressions beside unit-test failures — one pane of glass, which is the entire strategic point of the pytest-native route.
Wire into CI with cost and variance controls. In the workflow: run the LLM test directory only when feature code, prompts, or golden data change (path filters); cache enabled (deepeval caches metric calls); mark variance-prone cases with @pytest.mark.flaky(reruns=0) explicitly disabled and use repeat-with-threshold logic instead — a plain rerun-on-fail plugin silently converts your statistical gate into retry-until-green:
def test_ambiguous_case_statistical(case):
passes = sum(run_and_grade(case) for _ in range(5))
assert passes >= 4, f"{passes}/5 below 0.8 threshold"
Calibrate the metrics quarterly against human grades. Take 30 recent outputs, have two humans pass/fail them against the same criteria, and compare with the metric verdicts. Agreement below ~85% means the threshold or the evaluation steps need editing — adjust, re-run, record the calibration date in the repo. An uncalibrated G-Eval score is an opinion with decimals.
The pytest-native route wins on organizational physics, not on metrics: LLM quality signals land in the dashboards, alert rules, and review rituals your team already trusts. A slightly better metric in a platform nobody opens loses to a good-enough metric in the pipeline everyone watches.
threshold=0.7 nobody can explain is technical debt with authority. Every threshold traces to a calibration or a written decision.Name LLM test failures after their metric reason in the report ("relevancy: omitted second issue"), not just the case id — after a month, the failure-reason histogram becomes your prompt-improvement backlog, generated for free by CI.