Bloom-aligned objectives

• Understand: what differentiates agents from chatbots (planning, tool use, execution, state)
• Analyze: when agentic architecture is appropriate vs overkill
• Design: a reference architecture for an enterprise agent runtime

Topics

• Agentic system components: policy layer, orchestration layer, tool layer, memory layer, evaluation layer, observability layer
• Agent Framework positioning and core capabilities (Python/.NET, agents + workflows)
• Foundry Agent Service runtime view: persisted agent definitions and runtime interaction history
• Architecture patterns:
  • single tool-using agent
  • supervisor + specialist agents
  • workflow/graph-based orchestration (deterministic handoffs)
  • parallel agents with aggregator

Labs

• Lab 1.1: Target architecture blueprint — Create a reference architecture and deployment topology for a multi-agent system (services, tools, identity, logs).
• Lab 1.2: Orchestration pattern selection — Map three enterprise tasks to the best orchestration pattern and justify tradeoffs (latency, risk, determinism).

Bloom-aligned objectives

• Apply: persistent conversation state to multi-turn and long-running tasks
• Create: a thread/run lifecycle model with retries and idempotency
• Analyze: failure modes caused by state drift and partial tool completion

Topics

• Threads, runs, and messages: purpose and lifecycle concepts
• Handling long-running work:
  • run status polling and streaming patterns
  • run expiration constraints and implications for tool execution timeouts
• Agent Framework abstractions for multi-turn conversation and threading
• Idempotency and replay safety:
  • tool call deduplication
  • run resumption design

Labs

• Lab 2.1: Threaded agent runtime — Implement a minimal runtime that creates threads, appends messages, and starts runs; persist and rehydrate state.
• Lab 2.2: Run retry + dedupe — Simulate tool timeout and implement safe retries without duplicating side effects.

Bloom-aligned objectives

• Implement: function calling end-to-end (define → request → execute → return results)
• Design: typed tool contracts and error handling
• Evaluate: tool correctness, robustness, and security

Topics

• Function calling mechanics in Foundry Agent Service:
  • metadata returned by agent (function name + arguments)
  • application executes the function and returns results
• Tool contract engineering:
  • schemas, validation, typed inputs/outputs
  • error models (transient vs permanent), retry policy, timeouts
• Tool safety rails:
  • tool allowlists, parameter constraints, PII redaction, audit logging
• Tool categories:
  • data tools (DB/query, search)
  • action tools (ticket creation, email sending via internal gateway, workflow triggers)
  • compute tools (ETL tasks, batch jobs)
• Observability for tools:
  • per-tool latency, error rates, failure taxonomy, correlation IDs

Labs

• Lab 3.1: Build a tool server — Implement a tool gateway (REST) exposing 5 tools: search, create ticket, update ticket, fetch customer profile, write note.
• Lab 3.2: Function calling integration — Connect the agent to tools using function calling; validate arguments and handle errors safely.
• Lab 3.3: Tool safety drill — Add allowlists + parameter constraints; simulate an unsafe request and verify the agent refuses or routes to safe alternatives.

Bloom-aligned objectives

• Understand: different memory types and their risks
• Apply: memory strategies to improve continuity without leaking sensitive context
• Create: a memory policy (what to store, where, for how long, and why)

Topics

• Agent Framework memory options: in-memory history vs persistent stores and variations by agent type
• Memory design:
  • short-term context window management
  • long-term memory (preferences, user profile abstractions, episodic summaries)
  • task memory (tool outputs, intermediate reasoning artifacts)
• Safety and governance:
  • retention rules, tenant boundaries, “right to forget” patterns
  • PII handling and redaction before persistence

Labs

• Lab 4.1: Memory policy implementation — Implement a memory layer that stores conversation summaries and key entities with retention and tenant isolation rules.
• Lab 4.2: Context budget optimization — Implement a context assembly step that selects only relevant memory and evidence for the next turn.

Bloom-aligned objectives

• Design: multi-agent roles and coordination protocols
• Implement: supervisor/specialist and workflow-based orchestration
• Evaluate: handoff quality and outcome consistency

Topics

• Role specialization:
  • planner/supervisor agent
  • executor/tool agent
  • reviewer/validator agent
  • domain specialist agents (policy, finance, HR, IT ops)
• Orchestration styles:
  • supervisor-managed sequential handoffs
  • parallel agents + aggregator
  • deterministic workflows (graph/YAML/visual orchestration in Foundry Agent Service)
• Conflict resolution:
  • voting/consensus, reviewer arbitration, confidence-based selection
• Reliability patterns:
  • “plan then act” separation
  • bounded autonomy (max steps, cost ceilings, tool call limits)

Labs

• Lab 5.1: Supervisor + specialists — Build a supervisor agent that assigns tasks to 3 specialists (Research, Tool Executor, Compliance Reviewer) and merges outputs with citations and audit notes.
• Lab 5.2: Multi-agent workflow — Implement a workflow where agents run in parallel, then an aggregator produces final output; add stopping criteria and escalation routing.
• Lab 5.3: Handoff audit trail — Produce a structured handoff log (who decided what, which tool outputs were used, which constraints were applied).

Bloom-aligned objectives

• Evaluate: agent effectiveness and reliability with agent-specific evaluators
• Analyze: failure clusters (wrong tool selection, incomplete handoff, state drift, poor intent resolution)
• Create: promotion gates and regression suites

Topics

• Foundry agent evaluators and what they measure (agentic workflow evaluation concepts)
• SDK-based evaluation:
  • converting agent thread data into evaluation data
  • running evaluators over agent runs
• Custom evaluators (prompt-based / code-based) for:
  • tool correctness
  • policy adherence
  • handoff quality
  • completeness and consistency
• Regression strategy:
  • golden tasks, adversarial tasks, tool-failure simulations
  • drift detection (tool schema changes, behavior changes)

Labs

• Lab 6.1: Build an evaluation suite — Create a golden set of 30 tasks with expected tool usage and expected outcomes; add pass/fail criteria.
• Lab 6.2: Evaluate agent runs via SDK — Convert thread data and run evaluators; generate a promotion report.

Bloom-aligned objectives

• Apply: safeguards against prompt injection and unsafe tool use
• Design: governance controls for human-in-the-loop and approvals
• Evaluate: safety under adversarial prompts and tool misuse attempts

Topics

• Threat model for agents:
  • tool abuse, escalation of privileges, data exfiltration, instruction hijacking
• Policy controls:
  • approval steps for high-impact actions
  • environment boundaries (dev/test/prod tools)
• Auditability:
  • tool call logs, arguments, outputs, decision traces
• Fail-safe design:
  • safe defaults, rollback hooks, manual override

Labs

• Lab 7.1: Adversarial tool misuse simulation — Attempt to coerce the agent into unauthorized actions; validate refusal and escalation behavior.
• Lab 7.2: HITL approvals — Implement “approval required” workflow for destructive actions (e.g., account changes, refunds).

Bloom-aligned objectives

• Analyze: cost/latency drivers in multi-agent systems
• Implement: practical optimizations and guardrails
• Create: an operational runbook for incidents and rollbacks

Topics

• Performance engineering:
  • concurrency controls, parallelism caps, caching of tool outputs
  • step limits and budget policies
• Observability:
  • end-to-end tracing across agents and tools
  • run-level metrics (steps, tool calls, retries, failures)
• Release management:
  • versioning agents/workflows, canary testing, rollback plans

Labs

• Lab 8.1: Ops dashboard — Build a minimal dashboard and log schema for run tracing, tool metrics, and failure analysis.
• Lab 8.2: Cost guardrail policy — Implement max-step, max-tool-call, and max-token policies and validate enforcement.

Deliverable

Build and demo a multi-agent assistant that:

• Uses persistent state (threads/runs/messages)
• Executes a toolchain via function calling (ticketing + knowledge lookup + customer/profile)
• Uses supervisor + specialists with bounded autonomy
• Produces an auditable handoff trail
• Passes an evaluation suite with defined promotion gates

Tools and platforms used

• Microsoft Agent Framework (Python/.NET agents and multi-agent workflows)
• Microsoft Foundry Agent Service (agent runtime components, threads/runs/messages, agent definitions)
• Function calling / tools integration (Foundry agent API)
• Agent evaluation and custom evaluators (Foundry evaluators + SDK-based evaluation)