Bloom-aligned objectives

• Understand: why GenAI systems introduce new security and governance risks
• Analyze: where traditional AppSec controls are insufficient
• Design: a security-first reference architecture for GenAI applications

Topics

• GenAI system anatomy and attack surfaces: prompts, context, retrieval, tools, memory, endpoints, telemetry
• Threat categories:
  • prompt injection and instruction hijacking
  • data leakage (PII, secrets, proprietary information)
  • retrieval poisoning and misinformation injection
  • tool misuse and unauthorized actions
  • insecure model access and endpoint abuse
  • operational risks: drift, regressions, policy misconfiguration
• Security objectives for GenAI:
  • confidentiality, integrity, availability
  • safety, compliance, auditability, non-repudiation

Labs

• Lab 1.1: GenAI risk inventory — Build a risk inventory for an enterprise assistant (data sources, users, tools, actions).
• Lab 1.2: Security architecture blueprint — Draw a reference architecture with trust boundaries and control points.

Bloom-aligned objectives

• Apply: structured threat modelling methods to GenAI systems
• Analyze: threats across components and trust boundaries
• Create: a prioritized mitigation plan with owners and timelines

Topics

• Threat modelling approach:
  • system decomposition, trust boundaries, data flows
  • threat identification across prompts/retrieval/tools/memory
  • impact/likelihood scoring and prioritization
• GenAI threat deep dives:
  • direct prompt injection and jailbreak patterns
  • indirect prompt injection through retrieved documents
  • data exfiltration via tool calls or retrieval overreach, cross-tenant data access failures
  • model endpoint abuse (rate, brute force, extraction attempts), prompt and tool schema leakage
• Control selection and placement:
  • pre-processing, retrieval-time controls, generation-time controls, post-processing validation

Labs

• Lab 2.1: Full threat model workshop — Create a complete threat model with data flow diagrams and trust boundaries.
• Lab 2.2: Mitigation backlog — Convert threats into a prioritized backlog with severity, owners, and implementation roadmap.

Bloom-aligned objectives

• Understand: why retrieval introduces unique integrity and confidentiality risks
• Implement: data boundary controls and grounded response policies
• Evaluate: RAG-specific security regressions

Topics

• RAG risks:
  • retrieval overreach (exposing unauthorized documents)
  • retrieval poisoning and misinformation injection
  • malicious instructions embedded in content (indirect injection), citation manipulation and fabricated sources
• Secure retrieval patterns:
  • filter-first retrieval (tenant/user/role labels), metadata-based access control enforcement
  • document classification and sensitivity labels, retrieval provenance and audit trails
• Grounding and refusal policies:
  • evidence thresholds and “insufficient evidence” safe-fail behavior
  • citation validation and snippet verification
• Data handling:
  • PII/PHI redaction before indexing
  • encryption and key management expectations, secure content ingestion pipelines with integrity checks

Labs

• Lab 3.1: Secure retrieval policy — Implement and test access filters and validate zero cross-tenant leakage.
• Lab 3.2: Indirect injection simulation — Inject malicious instructions into documents and validate defenses (instruction hierarchy + validators).
• Lab 3.3: Citation integrity gate — Implement post-generation checks to block answers without valid citations.

Bloom-aligned objectives

• Design: safe tool access patterns with least privilege
• Implement: guardrails for tool invocation and side-effect control
• Analyze: tool misuse failures and escalation paths

Topics

• Tool risk landscape:
  • privilege escalation through tools
  • unauthorized actions (refunds, account changes, approvals)
  • prompt-induced tool misuse, tool output poisoning into future decisions
• Tool governance controls:
  • allowlisted tools and scoped permissions
  • typed schemas and strict argument validation
  • timeouts, retries, circuit breakers, idempotency keys and side-effect protections
• Human-in-the-loop patterns:
  • approvals for high-risk actions
  • two-person rule and segregation of duties, step-up authentication patterns (conceptual)
• Auditability:
  • full tool call logs (arguments, outputs, actor identity, timestamps)
  • decision trace capture and reproducibility

Labs

• Lab 4.1: Tool allowlist gateway — Build a tool gateway policy with validation, permission checks, and auditing.
• Lab 4.2: High-risk action approval flow — Implement an approval workflow for destructive actions and validate enforcement.
• Lab 4.3: Side-effect safety drill — Simulate tool timeouts and retries; validate idempotent behavior and no duplicate actions.

Bloom-aligned objectives

• Create: a GenAI governance framework aligned to enterprise controls
• Apply: change control to prompts, policies, tools, and models
• Evaluate: audit readiness and evidence completeness

Topics

• Governance structure:
  • roles and responsibilities (product, security, legal, compliance, engineering)
  • approval checkpoints and decision rights
• Policy pack design:
  • acceptable use policy for GenAI
  • data usage and retention policy, tool access policy and escalation rules
  • model selection and validation policy, logging and audit policy
• Risk management:
  • risk register formats, periodic reviews, control effectiveness scoring
  • third-party/vendor risk (model providers, SaaS tool integrations)
• Change control:
  • versioning prompts/flows/agents/tool schemas
  • staged environments and promotion gates
  • rollback and emergency change procedures

Labs

• Lab 5.1: Governance playbook — Create a governance playbook (RACI, approvals, risk reviews, and audit artifacts).
• Lab 5.2: Control mapping — Map GenAI controls to internal security/compliance standards and define evidence requirements.
• Lab 5.3: Change control workflow — Design a change-control pipeline for prompt/tool/model changes with gating and audit logs.

Bloom-aligned objectives

• Understand: Responsible AI principles and their practical implications
• Apply: safety and privacy-by-design measures for LLM applications
• Create: measurable acceptance criteria for Responsible AI

Topics

• Responsible AI principles in practice:
  • safety and harm prevention
  • privacy and data minimization, fairness and bias considerations
  • transparency: disclosures, citations, limitations
  • accountability: ownership, review, escalation
• Safety controls:
  • content moderation concepts (inputs and outputs)
  • policy-based refusal behavior and safe completion templates
• Privacy engineering:
  • data minimization for prompts and logs
  • secure redaction of sensitive data in telemetry, retention and deletion workflows
• User experience safeguards:
  • disclaimers and user guidance
  • safe suggestions and escalation to human experts

Labs

• Lab 6.1: Responsible AI acceptance criteria — Define measurable criteria for safety, privacy, and transparency for a target use case.
• Lab 6.2: Transparency design — Implement a response format with citations, limitations, and escalation guidance.
• Lab 6.3: Privacy logging policy — Define what is logged, what is redacted, retention windows, and review workflow.

Bloom-aligned objectives

• Evaluate: system robustness using adversarial and regression suites
• Analyze: failure clusters and create prevention mechanisms
• Create: a continuous validation plan integrated with release cycles

Topics

• Testing strategy:
  • unit tests for tool schemas and validators
  • regression tests for prompts and workflows
  • adversarial testing for injection and data leakage
• Red-teaming:
  • test packs: jailbreak attempts, indirect injection, tool misuse, cross-tenant access attempts
  • scoring and reporting
• Release gating:
  • quality gates and safety gates
  • blocking criteria and rollback triggers

Labs

• Lab 7.1: Red-team test pack — Build and run an adversarial test set and define pass/fail thresholds.
• Lab 7.2: Safety regressionAG pipeline — Implement a CI gate that blocks releases on safety regressions (policy violation, leakage, unsafe tool calls).

Bloom-aligned objectives

• Apply: operational monitoring and alerting patterns
• Analyze: incidents and perform structured postmortems
• Create: incident playbooks for GenAI-specific failures

Topics

• What to monitor:
  • prompt injection signals, refusal rates, anomalous tool activity
  • access violations, unusual retrieval patterns, token spikes
  • safety incidents and policy violations
• Incident response:
  • containment (disable tools, restrict retrieval scope, roll back prompt versions)
  • investigation (trace review, tool logs, retrieval logs)
  • communication and documentation for audit
• Postmortems:
  • root cause analysis, corrective actions, regression tests, governance improvements

Labs

• Lab 8.1: Incident drill — Run a simulated injection + data leakage incident; execute containment and rollback steps.
• Lab 8.2: Postmortem template — Produce a complete postmortem report with corrective action backlog and prevention gates.

Deliverable

A complete security and governance package for a GenAI application that includes:

• Threat model + prioritized mitigations
• Governance playbook + control mapping + audit evidence checklist
• Responsible AI acceptance criteria + validation plan
• Red-team test pack + CI safety gate design
• Incident response runbooks and rollback triggers

Tools and platforms used

• Threat modelling templates and risk registers (enterprise-ready formats)
• Security control catalog (IAM, network security, data security, logging/audit, application security)
• Adversarial testing and red-teaming harness (prompt packs + tool misuse simulations)
• Evaluation datasets and gating strategies (offline + continuous validation)
• Governance artifacts (RACI, approvals, change control, audit evidence pack)