Retrieval contract
Define which questions the system should answer and when it must decline.
MODULE 09 / 10
All modules ↑KNOWLEDGE · TARGET L3 · 5–7 days
09
Internal RAG assistant
Answer operational questions from approved runbooks with citations and measurable grounding quality.
Build a useful internal assistant without weakening document authorization, freshness or evidence standards.
Prerequisites
- Modules 01–02 and 05–06
- S3 and IAM at L3
- Basic retrieval concepts
01 / CONCEPT LESSONS
Build the mental model first.
Document lifecycle
Ownership, approval, classification, version, expiry and deletion begin before chunking.
Chunk and metadata
Structure retrieval around operational meaning and authorization attributes.
Access-aware retrieval
Filter before retrieval and verify again before response; citations must not leak titles or paths.
Evaluation
Measure retrieval relevance, grounded correctness, citation validity and refusal behavior.
02 / SYSTEM FLOW
Trace the complete path.
Authorized user→Query classification→Metadata filter→Knowledge Base retrieval→Grounded generation→Citation check→Answer / refuse
For every arrow, the engineer must name the data, identity, trust boundary, failure mode and evidence produced.
03 / GUIDED LAB
Build it in bounded steps.
AI and documentation are allowed. The learner owns every generated artifact and must preserve the stated safety boundaries.
- 01Select only approved internal runbooks.
- 02Add owner, classification and freshness metadata.
- 03Create a question and refusal evaluation set.
- 04Build access-aware retrieval.
- 05Require source citations in answers.
- 06Measure results and fix the largest error category.
04 / INCIDENT SIMULATION
The assistant confidently cites a stale runbook the user should not access.
Mentor injects
- Metadata filter is applied after retrieval
- Superseded document remains indexed
- Citation validation checks format only
Required evidence
- Retrieval and authorization trace
- Index lifecycle correction
- Evaluation result after remediation
Investigation loop
Symptom → impact → hypothesis → evidence → test → root cause → containment → correction → verification.
05 / DESIGN CHALLENGE
Start with a blank page.
Design ingestion and revocation for future client documents with tenant isolation and deletion guarantees.
06 / VERIFY & SUBMIT
Verification checklist
Submission package
- Data inventory
- Retrieval design
- Evaluation set and results
- Access incident report
- Operations runbook
Review questions
- Who owns document truth?
- What happens when access changes?
- How is stale knowledge detected?
- What is a valid citation?
- When must the assistant refuse?
07 / MENTOR-ONLY GUIDANCE
Ask a question whose answer exists only in a restricted document. The correct behavior is a clean refusal without leaking metadata.
SCORING1 · cannot explain2 · explains with gaps3 · implements and verifies4 · designs and handles trade-offs
Record the achieved L-level only from observed evidence. Course completion and certification do not automatically change the skill matrix.