Observability Skills

Skills for logging, monitoring, tracing, and alerting in production systems.

logging-strategy

Structured logging strategy including log levels, correlation IDs, context propagation, and PII avoidance. Use when designing logging, reviewing log statements, or setting up log aggregation.

Triggers: Designing a logging approach, reviewing existing log statements, setting up log aggregation (ELK, Loki, CloudWatch), adding correlation IDs, deciding what to log and what to avoid. Tools: Bash Read Write References: structured-logging.md

Key capabilities:

• Six log levels with clear usage guidance (TRACE through FATAL) and rules for choosing between them
• Structured logging (JSON/key-value) over unstructured text for machine parseability
• Correlation IDs: generate request_id at system boundary, propagate via X-Request-ID header, include in every log line
• W3C Trace Context propagation with trace_id and span_id for distributed systems
• MDC (Mapped Diagnostic Context) for transparent ID propagation
• What to log: request summaries, state transitions, decision points, errors, performance data, lifecycle events, retries, external calls
• What NOT to log: PII, secrets, sensitive business data, high-cardinality user input
• Log aggregation: centralized systems, JSON ingestion, retention policies (hot/warm/cold), rotation by size or time
• Performance considerations: lazy evaluation, avoid logging in tight loops, sampling, async appenders
• Anti-pattern detection: log-and-throw, everything-at-INFO, string concatenation, missing context, swallowed exceptions

metrics-monitoring

Application metrics and monitoring using RED/USE methods, Prometheus patterns, and SLO-based alerting. Use when instrumenting applications, designing dashboards, or setting up monitoring.

Triggers: Instrumenting an application with metrics, designing monitoring dashboards, setting up alerting, choosing metric types, defining SLIs/SLOs, applying observability methodology (RED, USE, golden signals). Tools: Bash Read Write References: metric-types.md

Key capabilities:

• Four Golden Signals (Google SRE): latency, traffic, errors, saturation
• RED method for request-driven services: rate, errors, duration (p50/p95/p99)
• USE method for infrastructure resources: utilization, saturation, errors
• Prometheus metric types: counter, gauge, histogram, summary with naming conventions
• Dashboard design: golden signals first, percentile latency, layered dashboards, deployment markers
• Alerting thresholds: symptom-based over cause-based, SLO burn-rate alerting, baseline-derived thresholds, multi-window alerting
• SLI/SLO/SLA framework: quantitative indicators, target objectives, error budgets, budget-based feature freezes
• Instrumentation checklist: endpoint RED metrics, dependency metrics, queue depth, connection pools, business metrics, runtime metrics

distributed-tracing

Distributed tracing with OpenTelemetry including spans, traces, context propagation, and sampling strategies. Use when instrumenting distributed systems, debugging request flows, or setting up tracing infrastructure.

Triggers: Instrumenting a distributed system with tracing, debugging requests spanning multiple services, setting up OpenTelemetry, choosing sampling strategies, understanding latency across service boundaries. Tools: Bash Read Write References: None

Key capabilities:

• Core concepts: traces, spans (with parent-child tree structure), context propagation, baggage
• OpenTelemetry architecture: SDK, API, Exporter, Collector, auto-instrumentation
• Instrumentation strategy: auto-instrumentation first, then manual spans for business logic
• Span naming (component.operation), attributes (HTTP, DB, business context), and error recording
• Context propagation: W3C Trace Context (traceparent/tracestate), B3, message queues, async boundaries
• Head-based sampling: probabilistic and rate-limiting approaches
• Tail-based sampling via OTel Collector: error-based, latency-based, and policy-based rules
• Trace analysis for debugging: critical path identification, gap detection, fast-vs-slow comparison, waterfall analysis
• Common pitfalls: missing propagation, over-instrumentation, sensitive data in attributes, no production sampling, broken async traces

alerting-oncall

Alert design and on-call practices including severity levels, runbooks, SLO-based alerting, and escalation policies. Use when designing alerts, writing runbooks, or improving on-call processes.

Triggers: Designing alerts for a service, writing runbooks, reducing alert fatigue, setting up escalation policies, implementing SLO-based alerting, improving on-call processes. Tools: None References: None

Key capabilities:

• Alert severity levels: P1 (immediate, page), P2 (30 min, page), P3 (4 hours, ticket), P4 (1 business day, ticket)
• Page vs ticket decision framework based on user impact and intervention urgency
• SLO-based burn-rate alerting with multi-window thresholds (14.4x/5min+1hr for P1, 6x/30min+6hr for P2, 3x/2hr+24hr for P3)
• Runbook template: what it means, likely causes, diagnosis steps, mitigation, escalation
• Alert fatigue prevention: track volume (<2 pages/shift target), weekly review, merge related alerts, auto-resolve transients
• Escalation policies: primary to secondary (10 min P1, 30 min P2), to engineering lead (1hr P1, 4hr P2), incident declaration
• On-call handoff practices: consistent timing, written summaries, acknowledgment, shadow rotations for new members
• Incident communication: dedicated channel, initial summary, 15-30 min updates, status page, post-incident review within 48 hours