Rollback

The fastest way to end an incident is often to undo the change that caused it. But rollbacks have their own failure modes - especially when data has changed.

Decision: Rollback vs Fix Forward

Is the fix obvious and small (< 15 min)?
  YES → Fix forward (patch + deploy)
  NO  →
    Has data been mutated by the bad code?
      YES → Rollback is complex (see Data Rollback below)
      NO  → Rollback the deployment
        Were database migrations applied?
          YES → Assess if migration is reversible (see below)
          NO  → Clean code rollback - safest option

When to Rollback

• Root cause is unclear
• Fix will take > 15 minutes
• Impact is severe and growing
• Multiple systems are affected

When to Fix Forward

• Root cause is identified and fix is small
• Rollback would cause data issues
• The deployed migration can't be reversed
• The bad code hasn't reached all users yet (canary)

Code Rollback

Revert the Deployment

# Option 1: Redeploy the previous version
# (Best - deploys a known-good artifact)
deploy --version <previous-version>

# Option 2: Git revert + fast deploy
git revert <bad-commit-sha>
git push
# CI/CD deploys the revert

Verify After Rollback

• Health checks passing
• Error rate back to baseline
• Key user flows working (smoke test)
• No new error types appearing
• Latency within normal range
• Customer-facing issues resolved

Database Rollback

The hard one. Data changes aren't automatically reversible.

Schema Rollback

Migration Type	Reversible?	How
Add column (nullable)	Yes	Drop the column
Add column (NOT NULL with default)	Yes	Drop the column
Add index	Yes	Drop the index
Add table	Yes	Drop the table
Drop column	NO	Data is gone (restore from backup)
Drop table	NO	Data is gone (restore from backup)
Rename column	Depends	Rename back (if no code depends on new name yet)
Change column type	Depends	Change back (if no data loss in conversion)
Data backfill	Depends	Reverse backfill (if old values are known)

Rules:

1. Always write down migrations. Test the down migration.
2. Never drop columns in the same deploy as the code change.
3. Separate schema migration from data migration.
4. Before deploying irreversible migrations: take a snapshot/backup.

Data Rollback

When bad code wrote wrong data:

1. STOP THE BLEEDING - disable the code path (feature flag, rollback)
2. ASSESS DAMAGE - what records are affected? Query for anomalies.
3. DETERMINE CORRECT STATE - can you derive correct values from logs, events, or audit trail?
4. FIX DATA - scripted repair (never manual edits)
5. VERIFY - query to confirm all affected records are fixed
6. PREVENT - add validation to prevent bad data from being written again

Config Rollback

# If config is in environment variables:
# Revert the config change in your config management system

# If config is in a config file:
git revert <config-change-commit>

# If config is in a feature flag service:
# Toggle the flag off (instant, no deploy needed)

Config rollback should be the fastest of all - no build, no deploy.

Feature Flag Rollback

The safest rollback - no deployment needed.

1. Toggle the flag OFF in your flag management system
2. Effect is immediate for new requests
3. Verify: feature is disabled, baseline behavior restored
4. Investigate at leisure - the incident is over

This is why feature flags exist. If a feature is behind a flag, rollback is a button click.

Multi-Service Rollback

When multiple services were deployed together:

1. Identify which service(s) are actually broken
   (don't roll back everything - that causes more disruption)
2. Rollback in reverse deployment order
   (last deployed → first rolled back)
3. Check cross-service API compatibility at each step
   (old Service A + new Service B - does it work?)
4. Verify inter-service communication after each rollback

The safest approach: Services should be independently deployable. If they're not, that's tech debt to address.

Rollback Checklist

Before Deploying (Prepare)

• Know the previous good version/SHA
• Verify rollback mechanism works (have you tested it recently?)
• Understand what data changes the deploy includes
• Know if database migrations are reversible
• Feature flags in place for risky features

During Rollback (Execute)

• Communicate to team: "Rolling back deployment X"
• Execute rollback (redeploy previous version)
• Reverse database migration if applicable
• Toggle feature flags if applicable
• Monitor health checks and error rates

After Rollback (Verify)

• Health checks green
• Error rate at pre-deploy baseline
• Key user flows verified
• Data integrity checked (if data was affected)
• Communicate: "Rollback complete, service restored"
• Create post-mortem task

Anti-Patterns

Anti-Pattern	Problem	Fix
No rollback plan	Panic during incident, slow recovery	Require rollback plan for every deploy
Rollback untested	Rollback mechanism itself is broken	Test rollbacks regularly (in staging)
Rolling back everything	Unnecessary disruption	Identify the broken service, roll back only that
Ignoring data impact	Rollback code but bad data persists	Check for data side effects before rolling back
Fix forward under pressure	Bad fix causes second incident	Default to rollback when stressed. Fix forward only when root cause is clear.