Designs monitoring and alerting systems. Covers metric selection, dashboard design, alert rules, SLOs/SLIs, on-call runbooks, and observability architecture. Use when setting up monitoring, defining SLOs, creating dashboards, or improving alerting.

Monitoring & Alerting

Monitor for user impact, not system internals. A server at 95% CPU is fine if users are happy. A server at 20% CPU is broken if requests are timing out.

The Four Golden Signals

Monitor these for every service:

Signal	What It Measures	Example Metric
Latency	How long requests take	p50, p95, p99 response time
Traffic	How much demand	Requests per second, concurrent users
Errors	How often things fail	Error rate (%), 5xx count
Saturation	How full the system is	CPU, memory, disk, queue depth, connection pool

If you can only monitor four things, monitor these.

SLOs, SLIs, SLAs

SLI (Service Level Indicator)

A metric that measures a user-facing behavior.

SLI: Percentage of HTTP requests completing in < 500ms
Current: 99.2%

SLO (Service Level Objective)

A target for an SLI. Internal commitment.

SLO: 99.5% of requests complete in < 500ms, measured monthly
Error budget: 0.5% = ~3.6 hours of allowed degradation per month

SLA (Service Level Agreement)

An SLO with consequences. External contract.

SLA: 99.9% uptime. Credits issued below this threshold.

Define SLOs before building dashboards. They tell you what to measure and when to alert.

Dashboard Design

Layout

┌─────────────────────────────────────────────┐
│              Service Health Summary          │
│  ✅ API  ✅ Database  ⚠️ Worker  ✅ Cache    │
├──────────────────────┬──────────────────────┤
│   Request Rate       │   Error Rate         │
│   📈 (last 6h)      │   📈 (last 6h)      │
├──────────────────────┼──────────────────────┤
│   Latency (p50/p95)  │   Saturation         │
│   📈 (last 6h)      │   📈 (last 6h)      │
├──────────────────────┴──────────────────────┤
│              Recent Deployments              │
│  ▼ deploy markers on all graphs above       │
└─────────────────────────────────────────────┘

Principles

Top-level dashboard: Red/yellow/green health for each service. Glanceable.
Service dashboard: The four golden signals for one service. Drill-down.
Investigation dashboard: Detailed metrics for debugging. On-demand.
Deploy markers on every graph (most issues correlate with deploys)
Default time range: last 6 hours (most investigations start here)
Include "what's normal" baselines (yesterday, last week)

Alert Design

When to Alert

SLO error budget is burning faster than expected
Service is down or unreachable
Error rate exceeds baseline by 3x+
Latency p95 exceeds SLO threshold for > 5 minutes

When NOT to Alert

CPU/memory/disk at a threshold (alert on user impact, not system state)
A single transient error (wait for pattern)
Something that can wait until business hours (use a ticket instead)
A dashbaord already shows it (redundant noise)

Alert Template

ALERT: [clear, specific name]
SEVERITY: critical / warning / info
CONDITION: [metric] [operator] [threshold] for [duration]
IMPACT: [what users experience]
RUNBOOK: [link to resolution steps]
ESCALATION: [who to contact if not resolved in X minutes]

Alert Rules

Every alert must be actionable. If the response is "wait and see," it's not an alert.
Every alert needs a runbook. No one remembers what to do at 3 AM.
Review alerts monthly. Delete alerts that fire but require no action (noise).
Page on symptoms, ticket on causes. "Requests failing" is a page. "Disk at 80%" is a ticket.

Metric Naming Convention

<service>.<subsystem>.<metric>.<unit>

# Examples
api.http.request_duration.milliseconds
api.http.requests.total
api.database.query_duration.milliseconds
api.database.connections.active
worker.jobs.processed.total
worker.jobs.failed.total
worker.queue.depth.count

Common Monitoring Stack

Need	Tools
Metrics	Prometheus, Datadog, CloudWatch, Grafana
Logs	Loki, ELK, CloudWatch Logs, Datadog Logs
Traces	Jaeger, Zipkin, Datadog APM, Honeycomb
Alerts	PagerDuty, OpsGenie, Grafana Alerting
Uptime	Pingdom, UptimeRobot, Checkly
Dashboards	Grafana, Datadog, CloudWatch

Reference: reference/alerting-patterns.md - golden signals by service type, severity levels, alert fatigue prevention, SLO/SLI templates, dashboard layouts, PromQL patterns, escalation decision tree.

Monitoring Maturity Checklist

Level 1: Basic

Health check endpoint exists and is monitored
Error rate tracked and alerted on
Basic uptime monitoring (is it responding?)

Level 2: Operational

Four golden signals dashboarded per service
Alerts have runbooks
Deploy markers on graphs
Log aggregation searchable

Level 3: SLO-Driven

SLOs defined for user-facing services
Error budget tracking and burn rate alerts
Distributed tracing for request flow
Alert noise reviewed and reduced monthly

Level 4: Proactive

Anomaly detection on key metrics
Capacity planning based on traffic trends
Chaos testing validates monitoring coverage
On-call load balanced and sustainable