Load Testing

"Works on my machine" is not a performance guarantee. Load testing answers: "will it work when 1000 users hit it at once?"

Types of Load Tests

Type	Purpose	Pattern
Load test	Does it handle expected traffic?	Ramp to target RPS, hold for 10-30 min
Stress test	Where does it break?	Ramp until failure, note the breaking point
Soak test	Does it leak resources over time?	Normal load for 4-24 hours
Spike test	Does it handle sudden bursts?	Sudden jump to 5-10x normal, then back down
Breakpoint test	What's the absolute max?	Incrementally increase until system fails

Step 1: Define Targets

Before writing tests, define what "good" looks like:

TARGET SCENARIO: [what you're simulating]
EXPECTED USERS: [concurrent users]
EXPECTED RPS: [requests per second]
ACCEPTABLE LATENCY: p50 < Xms, p95 < Xms, p99 < Xms
ACCEPTABLE ERROR RATE: < X%
DURATION: [how long to sustain]

Where do targets come from?

• Current production traffic (baseline) - check your monitoring
• Growth projections (2x, 5x, 10x current traffic)
• Specific events (launch, Black Friday, viral marketing campaign)
• SLO commitments (99.9% = 8.7h downtime/year)

Step 2: Design Realistic Scenarios

Model Real User Behavior

Don't just hammer one endpoint. Model actual usage patterns:

Scenario: E-commerce browsing
  - 60% browse products (GET /products)
  - 20% search (GET /search?q=...)
  - 10% add to cart (POST /cart)
  - 5% checkout (POST /orders)
  - 5% view account (GET /account)

Include Think Time

Real users pause between actions (reading, deciding). Without think time, your test generates unrealistic burst traffic.

User flow:
  Browse product → wait 3-8s → Add to cart → wait 1-3s → Checkout

Use Realistic Data

• Varied product IDs (not the same cached item)
• Varied user accounts (not one user doing everything)
• Varied payload sizes (not identical requests)
• Varied search queries (different cache keys)

Step 3: Set Up the Test

Environment

• Test against a staging environment that mirrors production (same infra, same data volume)
• Never load test production without explicit approval and monitoring
• Use production-like data volume (not an empty database)
• Monitor the test environment's resources during the test

Tools

Tool	Type	Best For
k6	Script (JS)	Developer-friendly, CI-friendly, free
Locust	Script (Python)	Python teams, distributed tests
Gatling	Script (Scala)	JVM teams, detailed reporting
Artillery	Script (YAML/JS)	Simple scenarios, serverless
Apache JMeter	GUI + Script	Complex scenarios, legacy standard
wrk / hey	CLI	Quick single-endpoint benchmarks
Grafana k6 Cloud	SaaS	Distributed load from multiple regions

Example: k6 Script

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },   // Ramp up to 100 users
    { duration: '10m', target: 100 },   // Hold at 100 users
    { duration: '2m', target: 0 },      // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],    // 95% of requests < 500ms
    http_req_failed: ['rate<0.01'],      // < 1% error rate
  },
};

export default function () {
  const res = http.get('https://staging.example.com/api/products');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });
  sleep(Math.random() * 3 + 1);  // Think time: 1-4 seconds
}

Step 4: Execute and Monitor

During the test, watch:

Metric	Where	Red Flag
Response time (p50, p95, p99)	Load test tool	p95 exceeding target
Error rate	Load test tool	> 1%
CPU utilization	Server monitoring	> 80% sustained
Memory usage	Server monitoring	Growing without stabilizing
Database connections	DB monitoring	Pool exhausted
Queue depth	Queue monitoring	Growing (consumers can't keep up)
Disk I/O	Server monitoring	Saturated
Network throughput	Server monitoring	Bandwidth cap hit

Step 5: Analyze Results

Finding Bottlenecks

Is latency increasing with load?
  → YES at low concurrency: Application bottleneck (slow code, N+1 queries)
  → YES at high concurrency: Resource saturation (CPU, memory, connections)
  → NO: System is handling the load well

Is error rate increasing with load?
  → 5xx errors: Server overwhelmed (timeouts, OOM, connection exhaustion)
  → 429 errors: Rate limiting (increase limits or optimize client)
  → Connection errors: Network or load balancer limits

Common Bottlenecks

Bottleneck	Symptom	Fix
Database queries	Latency spikes, connection pool exhausted	Add indexes, optimize queries, connection pooling
No connection pooling	New TCP connection per request	Use connection pool for DB, HTTP clients
Synchronous I/O	Threads blocked, low throughput	Async I/O or increase thread pool
Single instance	CPU maxed on one server	Horizontal scaling, load balancer
Missing cache	Same expensive computation repeated	Add caching layer
Large payloads	Network bandwidth saturated	Compress, paginate, reduce payload
Lock contention	Threads waiting on shared resources	Reduce critical section, use optimistic locking

Step 6: Report

## Load Test Report

### Test Configuration
- Target: [X] concurrent users / [X] RPS
- Duration: [X] minutes
- Environment: [staging, production-mirror]
- Date: [YYYY-MM-DD]

### Results
| Metric | Target | Actual | Status |
|--------|--------|--------|--------|
| p50 latency | < 200ms | 180ms | PASS |
| p95 latency | < 500ms | 620ms | FAIL |
| p99 latency | < 1000ms | 2400ms | FAIL |
| Error rate | < 1% | 0.3% | PASS |
| Max RPS achieved | 500 | 420 | FAIL |

### Bottlenecks Identified
1. [Bottleneck] - [evidence] - [recommended fix]

### Recommendations
1. [Action] - [expected improvement]

### Next Steps
- [ ] Fix identified bottlenecks
- [ ] Re-run test to verify improvements
- [ ] Schedule soak test for [date]

When to Run Load Tests

Trigger	Test Type
Before launch	Full load + stress + soak
Before major feature release	Load test for affected endpoints
Before expected traffic spike	Load test at expected peak
After significant architecture change	Full load test
Quarterly (routine)	Soak test (catch resource leaks)