Bloom-aligned objectives

• Understand: evaluation dimensions for GenAI systems
• Analyze: failure modes across chat, RAG, and agents
• Design: an evaluation blueprint tied to user acceptance criteria

Topics

• Why LLM testing differs from deterministic software testing
• Evaluation pyramid for GenAI:
  • unit checks (schemas, validators)
  • component evaluation (retrieval, prompt response quality)
  • end-to-end scenario evaluation (multi-turn, tool workflows)
• Defining acceptance criteria:
  • correctness and usefulness
  • groundedness/citations
  • safety and compliance
  • performance and cost

Labs

• Lab 1.1: Evaluation blueprint — Define evaluation dimensions, metrics, and minimum thresholds for a target use case.
• Lab 1.2: Failure mode mapping — Create a failure taxonomy for the target app (RAG + tool agent).

Bloom-aligned objectives

• Create: golden and adversarial evaluation datasets
• Apply: labeling strategies and expected-output contracts
• Analyze: dataset coverage and bias gaps

Topics

• Dataset types:
  • golden set (representative tasks)
  • regression set (historical failures)
  • adversarial set (injection, ambiguity, edge cases)
  • load/performance set (high volume, long inputs)
• Labeling strategies:
  • expected answer characteristics (not always exact text)
  • expected sources/citations for RAG
  • expected tool calls and sequence for agents
• Coverage planning:
  • intent coverage, topic coverage, user personas, error-prone areas
• Data governance:
  • handling PII, sanitization, retention, and audit constraints

Labs

• Lab 2.1: Golden set build — Create a 50-item dataset with labeled intents, expected characteristics, and pass/fail checks.
• Lab 2.2: Adversarial set build — Create an injection and tool misuse dataset; define safety expectations.
• Lab 2.3: Coverage heatmap — Create a coverage matrix mapping dataset items to intents and risk areas.

Bloom-aligned objectives

• Implement: automated evaluators for quality, groundedness, and safety
• Design: scoring rubrics that are stable and repeatable
• Evaluate: tradeoffs between precision and cost of evaluation

Topics

• Evaluator types:
  • rule-based (schemas, citations present, length constraints)
  • model-graded (LLM-as-judge) with calibration
  • hybrid evaluators (rules + judge + heuristics)
• Key scoring dimensions:
  • relevance, completeness, clarity
  • groundedness/faithfulness and citation correctness
  • safety/policy adherence, hallucination detection patterns (heuristics + judge)
• Calibration and consistency:
  • reducing judge variability, evaluator prompts and rubrics
  • inter-rater agreement concepts
• Reporting:
  • score distributions, failure clusters, examples

Labs

• Lab 3.1: Citation validator — Implement checks for citation presence, validity, and mapping to retrieved chunks.
• Lab 3.2: Groundedness evaluator — Implement a groundedness scoring rubric and evaluator for RAG answers.
• Lab 3.3: Safety evaluator — Implement a policy compliance evaluator for unsafe content/tool misuse attempts.
• Lab 3.4: Evaluator calibration drill — Run evaluators across a dataset, identify inconsistent scoring, and refine rubrics.

Bloom-aligned objectives

• Evaluate: retrieval quality and grounding correctness
• Analyze: root causes (chunking, indexing, ranking, filtering)
• Create: targeted regression tests for retrieval failures

Topics

• Retrieval evaluation:
  • hit rate, recall@k, MRR, NDCG (conceptual + applied)
  • metadata filter correctness and access boundary tests
• Chunking and indexing tests:
  • chunk quality heuristics (length, overlap, metadata completeness)
  • duplicate and stale content detection
• Grounding tests:
  • citation coverage, snippet correctness, “insufficient evidence” behavior
• Drift and freshness:
  • content updates, re-indexing integrity, regression detection

Labs

• Lab 4.1: Retrieval benchmark — Build a retrieval benchmark suite and compare configurations.
• Lab 4.2: Access boundary test — Validate that unauthorized content is never retrieved under role/tenant filters.
• Lab 4.3: “No answer” routing — Implement and test refusal behavior when evidence is weak or missing.

Bloom-aligned objectives

• Implement: tests for tool selection, tool correctness, and side-effect safety
• Evaluate: multi-step workflow success rates and recovery behavior
• Analyze: agent failures (looping, wrong tool, partial completion, state drift)

Topics

• Tool correctness testing:
  • schema validation for tool inputs
  • deterministic checks on tool outputs
  • timeout/retry/idempotency tests
• Workflow evaluation:
  • task completion metrics
  • step correctness and sequence constraints
  • bounded autonomy (max steps, max tool calls)
• Multi-agent handoffs:
  • role correctness, handoff quality, conflict resolution tests
• Failure recovery:
  • fallback paths when tools fail
  • safe escalation to human review

Labs

• Lab 5.1: Tool gateway simulator — Build a simulator that injects controlled failures (timeouts, partial data, rate limits).
• Lab 5.2: Agent workflow test suite — Create end-to-end tests for 20 workflows with expected tool usage patterns.
• Lab 5.3: Side-effect safety test — Validate idempotency and no duplicate actions under retries.
• Lab 5.4: Handoff evaluation — Evaluate a supervisor/specialist setup on consistency and completeness.

Bloom-aligned objectives

• Design: experiments that produce reliable decisions
• Apply: A/B and canary strategies for LLM applications
• Evaluate: improvements with statistical and practical significance

Topics

• Offline vs online experiments:
  • when to trust offline evaluations
  • how to detect “offline-online” mismatch
• A/B testing design:
  • success metrics, guardrail metrics (safety, latency, cost)
  • sampling and cohort considerations
• Canary and shadow testing:
  • staged rollout, rollback triggers
• Regression prevention:
  • mandatory gates for prompts, retrieval configs, tool schemas

Labs

• Lab 6.1: A/B plan — Create an A/B experiment design with success and guardrail metrics.
• Lab 6.2: Release gate policy — Implement a gating policy that blocks release if any guardrail metric regresses.

Bloom-aligned objectives

• Implement: automated evaluation in CI/CD
• Create: standardized evaluation reports and promotion criteria
• Analyze: pipeline results and decide promotion vs rollback

Topics

• CI pipeline design:
  • run evaluators on PRs and merges
  • publish evaluation artifacts
  • compare baseline vs candidate
• Promotion criteria:
  • threshold-based and weighted scoring models
  • must-pass safety and security checks
• Governance integration:
  • approvals for changes affecting safety or data boundaries

Labs

• Lab 7.1: CI evaluation pipeline — Build an automated pipeline that runs evaluation suites and blocks merges on regressions.
• Lab 7.2: Promotion report — Generate an executive-friendly evaluation report with failure analysis and recommendation.

Bloom-aligned objectives

• Apply: continuous evaluation and drift detection
• Analyze: production failures using trace-linked evidence
• Create: incident playbooks and feedback loops

Topics

• Continuous evaluation: sampling strategy, privacy considerations, cost controls
• Drift detection: retrieval drift, prompt regressions, tool schema changes
• Incident playbooks: quality drop, safety incident, tool outage
• Feedback loops: labeling production failures into regression sets

Labs

• Lab 8.1: Continuous evaluation config — Configure sampling-based continuous evaluation and dashboards.
• Lab 8.2: Incident drill — Run a simulated “quality drop” incident; identify root cause and add regression tests.

Deliverable

A complete testing and evaluation package that includes:

• Evaluation blueprint (metrics + thresholds + acceptance criteria)
• Datasets (golden + adversarial + regression)
• Evaluators (rules + judge + hybrid) with calibration notes
• Automated CI pipeline with gating rules
• A production monitoring and continuous evaluation plan
• A failure taxonomy and remediation playbook

Tools and platforms used

• Evaluation dataset formats and labeling templates (golden/regression/adversarial)
• Automated evaluators (rule-based + model-graded + hybrid)
• Tool gateway simulator for agent workflow testing
• CI/CD integration for automated evaluation and promotion gates
• Telemetry and monitoring dashboards for continuous evaluation and drift detection