Bloom-aligned objectives

• Understand: what makes GenAIOps distinct from classic MLOps
• Analyze: key failure modes (prompt regressions, retrieval drift, safety violations, tool errors)
• Design: an operating model (roles, artifacts, gates, KPIs)

Topics

• Lifecycle overview for LLM applications: prompt/flow engineering, evaluation, deployment, monitoring, governance
• Artifacts to version: prompts, flows, evaluator configs, datasets, tool schemas, system policies
• Environment strategy: dev/test/prod separation and controlled promotion

Labs

• Lab 1.1: Operating model blueprint — Define artifacts, owners, release gates, and KPIs for a chosen LLM app (RAG assistant or tool-using agent).
• Lab 1.2: Repo structure standard — Create a baseline repository layout for flows, evaluation sets, and CI pipelines.

Bloom-aligned objectives

• Apply: prompt flow to build executable LLM app workflows
• Create: variants and controlled experiments
• Analyze: experiment results and choose a promotion candidate

Topics

• prompt flow capabilities in Microsoft Foundry: prototype, iterate, and deploy AI applications via executable flows
• Flow composition: prompts + Python tools + external calls (retrieval/tools)
• Variant management: prompt variants, parameters, environment configs
• Template-driven GenAIOps: Microsoft GenAIOps prompt flow template (repo scaffold, lifecycle management concepts)

Labs

• Lab 2.1: Build a baseline flow — Implement a prompt flow for a question-answering or RAG workflow with structured outputs.
• Lab 2.2: Variant experiment — Create 3 prompt variants and compare outcomes on a small test set; document selection criteria.

Lab 2.3: Deploy a flow — Package and deploy the selected flow, capturing deployment config as code.

Bloom-aligned objectives

• Create: evaluation datasets (golden + adversarial)
• Implement: automated offline evaluation in CI/CD
• Evaluate: changes using quality/safety metrics before release

Topics

• Offline evaluation concepts: measuring quality/safety metrics on test datasets before production
• Evaluation in CI/CD: GitHub Actions approach for running evaluations and producing reports
• Evaluator selection: quality metrics (relevance, coherence, fluency) and safety metrics
• Statistical considerations: meaningful improvement vs random variation (confidence/consistency expectations)
• Optional enterprise pipeline pattern: Azure DevOps end-to-end GenAIOps with prompt flow (concepts transferable to Foundry-centered pipelines)

Labs

• Lab 3.1: Golden set builder — Create an evaluation dataset with expected characteristics and failure labels.
• Lab 3.2: GitHub Actions evaluation gate — Implement a workflow that runs offline evaluation on pull requests and blocks merge on regression.
• Lab 3.3: Promotion decision report — Generate a standardized evaluation report (metrics + samples + failure clusters + go/no-go).

Bloom-aligned objectives

• Implement: monitoring for quality and token usage in production
• Analyze: live telemetry to detect regressions and drift
• Create: alerting policies and SLOs

Topics

• Monitoring deployed prompt flow applications: collect inference data and monitor quality/safety metrics and token usage
• Operational metrics: request counts, latency, error rate; recurring monitoring and alerts
• Safety telemetry: abuse monitoring components and signals (content classification contributing to monitoring)
• Dashboard design: release impact view (before/after), cohort analysis, top failure intents

Labs

• Lab 4.1: Telemetry instrumentation — Add structured logging (prompt version, flow version, tokens, latency, error codes) and route to dashboards.
• Lab 4.2: Quality monitoring setup — Configure monitoring for groundedness/coherence/relevance (or equivalent metrics) and define alert thresholds.
• Lab 4.3: Regression triage drill — Simulate a regression (token spike + quality drop), identify root cause, and propose rollback.

Bloom-aligned objectives

• Understand: continuous evaluation sampling and tradeoffs
• Apply: near real-time quality/safety evaluation on live traffic
• Evaluate: agent behaviors using trace-linked diagnostics

Topics

• Continuous evaluation for agents:
  • near real-time observability at a sampling rate with metrics surfaced in an observability dashboard
  • evaluation results connected to traces for debugging and root cause analysis
• Agent evaluation via SDK:
  • converting agent thread data into evaluation-ready data for evaluators
• Operationalization:
  • sampling policies, privacy considerations, and cost management

Labs

Lab 5.1: Enable continuous evaluation — Configure continuous evaluation sampling and verify metrics + traces for a sample agent/app.

Lab 5.2: Agent run evaluation via SDK — Convert agent thread/run data and run an evaluator; produce an analysis summary.

Bloom-aligned objectives

• Apply: platform safety controls for Azure OpenAI usage
• Design: governance for safety configuration changes
• Evaluate: application behavior under unsafe inputs and policy constraints

Topics

• Azure OpenAI content filtering:
  • filters applied to prompts and completions to detect harmful content
  • severity thresholds and approval requirements for turning filters down/off
• Default safety policies:
  • default safety configurations and features applied broadly to models
• Safety ops playbooks:
  • incident categories (harmful content, injection attempts, data leakage)
  • audit logging and review workflows

Labs

• Lab 6.1: Safety configuration review — Define a change-control process for content filter modifications (approvals, testing, rollback).
• Lab 6.2: Safety regression tests — Build an adversarial prompt set and run it in CI; block releases on safety regression.

Bloom-aligned objectives

• Create: release strategies for LLM apps and agents
• Apply: canary/A/B with measurable success criteria
• Analyze: rollout decisions using evaluation + monitoring signals

Topics

• Release strategies: canary release, shadow testing, A/B experimentation
• Versioning strategy: prompt versioning, flow versioning, evaluator versioning
• Rollback discipline: rapid rollback triggers based on monitored metrics

Labs

• Lab 7.1: Release plan — Create a rollout plan with explicit gates (offline eval pass + monitoring thresholds).
• Lab 7.2: A/B analysis drill — Compare two flow versions using an evaluation report and a monitoring slice; decide promotion vs rollback.

Bloom-aligned objectives

• Analyze: cost and latency drivers (tokens, retries, tool calls)
• Implement: practical optimization levers
• Create: runbooks for common incidents

Topics

• Cost controls: token budgeting, context trimming, caching, rate limiting
• Reliability: retries, circuit breakers for external tools, fallback responses
• Runbooks: incident response, postmortems, regression prevention

Labs

• Lab 8.1: Cost guardrails — Implement token caps and alerting on token spikes.
• Lab 8.2: Ops runbook — Create a concise runbook for “quality drop”, “token spike”, and “safety incident”.

Deliverable

A working LLM app (prompt flow + API) with:

• versioned artifacts in Git
• CI pipeline that runs offline evaluation and blocks regressions
• production monitoring for quality + token usage + operational KPIs
• a governance plan for safety controls and rollback procedures

Tools and platforms used

• Microsoft Foundry prompt flow (build, iterate, deploy)
• Microsoft Foundry evaluations + CI integration (GitHub Actions evaluation)
• Microsoft Foundry observability / continuous evaluation (sampling, metrics, trace linkage)
• Monitoring quality/safety and token usage for deployed flows
• Azure OpenAI safety controls (content filtering, default safety policies, abuse monitoring)
• Optional pipeline patterns: Azure DevOps integration concepts with prompt flow