Telemetry model
Start from questions operators must answer, then define events, metrics and spans.
MODULE 06 / 10
All modules ↑OBSERVE · TARGET L4 · 5–7 days
06
GenAI observability with OTEL
Use correlated telemetry to find a latency or quality regression and connect it to cost.
Make an agent run explainable across application, model, tool, infrastructure and business outcome.
Prerequisites
- Modules 01–05
- OTEL fundamentals at L2
- CloudWatch dashboards and alarms
01 / CONCEPT LESSONS
Build the mental model first.
Trace hierarchy
One investigation trace contains model, retrieval, tool, approval and verification spans.
Semantic attributes
Record stable identifiers, versions, token counts and outcomes without sensitive prompt content.
Quality signals
Acceptance, correction and escalation provide operational feedback beyond technical success.
Sampling and retention
Preserve incidents and anomalies while controlling observability cost and privacy risk.
02 / SYSTEM FLOW
Trace the complete path.
Correlation ID→Agent span→Model span→Tool span→Approval span→Outcome event→Dashboard + alert
For every arrow, the engineer must name the data, identity, trust boundary, failure mode and evidence produced.
03 / GUIDED LAB
Build it in bounded steps.
AI and documentation are allowed. The learner owns every generated artifact and must preserve the stated safety boundaries.
- 01Define the minimum telemetry schema.
- 02Instrument model and tool calls.
- 03Propagate correlation across async steps.
- 04Record token, latency and outcome attributes.
- 05Build latency, error and quality views.
- 06Find one deliberately injected regression.
04 / INCIDENT SIMULATION
P95 latency doubles, but infrastructure metrics appear normal.
Mentor injects
- One tool waits on a slow API
- Approval duration is mixed with compute time
- Trace sampling drops failed requests
Required evidence
- Critical-path trace
- Corrected duration breakdown
- Sampling change with cost impact
Investigation loop
Symptom → impact → hypothesis → evidence → test → root cause → containment → correction → verification.
05 / DESIGN CHALLENGE
Start with a blank page.
Design telemetry for a future client deployment that cannot store raw prompts. Preserve diagnosis value without sensitive content.
06 / VERIFY & SUBMIT
Verification checklist
Submission package
- Telemetry schema
- Trace screenshot or export
- Dashboard definition
- Regression incident report
- Retention ADR
Review questions
- Which span owns token count?
- How is queue time separated?
- What is safe to retain?
- How do you measure false positives?
- What would sampling hide?
07 / MENTOR-ONLY GUIDANCE
Ask the engineer to explain one slow trace from top to bottom and identify the single critical path.
SCORING1 · cannot explain2 · explains with gaps3 · implements and verifies4 · designs and handles trade-offs
Record the achieved L-level only from observed evidence. Course completion and certification do not automatically change the skill matrix.