Inference contract
Define input, expected output, allowed variability and unacceptable behavior before choosing a model.
MODULE 02 / 10
All modules ↑FOUNDATION · TARGET L3 · 4–5 days
02
Bedrock & model economics
Run a repeatable evaluation and defend a model-routing decision using measured evidence.
Teach model selection as an engineering trade-off across task quality, latency, context, privacy, quotas and unit cost.
Prerequisites
- Module 01 passed
- Basic Python and JSON
- AWS budgets and CloudWatch basics
01 / CONCEPT LESSONS
Build the mental model first.
Tokens and context
Tokens drive cost and latency. Context is a bounded resource; more context can reduce rather than improve quality.
Evaluation before preference
A small task-specific dataset is more useful than generic benchmark claims. Score correctness, completeness, safety and format.
Routing and fallback
Use the least expensive model that meets the task threshold, with explicit conditions for escalation and fallback.
Quota and privacy
Capacity, region, logging and data-handling assumptions must be verified for the actual deployment.
02 / SYSTEM FLOW
Trace the complete path.
Evaluation cases→Prompt template→Bedrock model A/B→Structured scoring→Latency + tokens→Decision threshold→Routing policy
For every arrow, the engineer must name the data, identity, trust boundary, failure mode and evidence produced.
03 / GUIDED LAB
Build it in bounded steps.
AI and documentation are allowed. The learner owns every generated artifact and must preserve the stated safety boundaries.
- 01Create 20 sanitized operational questions with expected evidence.
- 02Define a four-dimension scoring rubric before running models.
- 03Invoke two suitable Bedrock models with the same settings.
- 04Capture input/output tokens, latency, score and failure reason.
- 05Calculate cost per accepted answer.
- 06Write a routing decision and one fallback rule.
04 / INCIDENT SIMULATION
Cost triples while accepted-answer quality falls.
Mentor injects
- Prompt now includes repeated context
- Retries are not capped
- Fallback routes all requests to the largest model
Required evidence
- Per-request token comparison
- Retry and routing trace
- Corrected budget and quality thresholds
Investigation loop
Symptom → impact → hypothesis → evidence → test → root cause → containment → correction → verification.
05 / DESIGN CHALLENGE
Start with a blank page.
Design model routing for classification, investigation summary and high-risk recommendation tasks. State when not to use an LLM.
06 / VERIFY & SUBMIT
Verification checklist
Submission package
- Evaluation dataset
- Results table
- Cost worksheet
- Routing ADR
- Failure analysis
Review questions
- Why can a larger context reduce quality?
- Which metric determines acceptance?
- How would model drift be detected?
- What is the cost of a failed answer?
- When should routing refuse the request?
07 / MENTOR-ONLY GUIDANCE
Do not accept 'model A felt better.' Ask for the exact task threshold and show how the decision changes if price or latency doubles.
SCORING1 · cannot explain2 · explains with gaps3 · implements and verifies4 · designs and handles trade-offs
Record the achieved L-level only from observed evidence. Course completion and certification do not automatically change the skill matrix.