Infrastructure as Code

IaC makes infrastructure reproducible, reviewable, and version-controlled. It also makes it possible to delete your production database with a typo. Respect the power.

Core Principles

1. Everything in code. If it exists in production, it's defined in code. No manual console changes.
2. Plan before apply. Always review the diff before making changes. Always.
3. Small changes. One concern per PR. Don't refactor modules while adding a new service.
4. Immutable infrastructure. Replace, don't mutate. New AMI > patch existing server.
5. Blast radius awareness. Know what a change can destroy before you apply it.

Tool Comparison

Feature	Terraform	Pulumi	CloudFormation	CDK
Language	HCL	Python/TS/Go	YAML/JSON	Python/TS
State	Remote (S3, etc.)	Pulumi Cloud / self-managed	AWS-managed	AWS-managed
Multi-cloud	Yes	Yes	AWS only	AWS only
Learning curve	Low-medium	Lower (real language)	Medium	Medium
Community	Largest	Growing	AWS-focused	AWS-focused

Project Structure

infrastructure/
├── modules/              # Reusable modules
│   ├── vpc/
│   ├── database/
│   ├── service/
│   └── monitoring/
├── environments/          # Environment-specific config
│   ├── dev/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── terraform.tfvars
│   ├── staging/
│   └── production/
├── global/                # Shared resources (DNS, IAM)
└── README.md

Rules:

• Separate state per environment (dev state can't affect prod)
• Modules are reusable (same module, different parameters per env)
• Environment differences are in variables, not in code duplication

State Management

State is the mapping between your code and real infrastructure. Lose it or corrupt it and you're in trouble.

Remote State (Required for Teams)

# Terraform: S3 + DynamoDB for locking
terraform {
  backend "s3" {
    bucket         = "mycompany-terraform-state"
    key            = "production/main.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

State Rules

• Never edit state manually (unless you truly understand the consequences)
• Lock state during operations (prevents concurrent modifications)
• Encrypt state at rest (it contains secrets and resource details)
• Separate state per environment (one state file per env)
• Back up state (enable versioning on the state bucket)

State Operations (Use Rarely, Carefully)

# Import existing resource into state
terraform import aws_s3_bucket.my_bucket my-existing-bucket

# Remove resource from state (without destroying it)
terraform state rm aws_s3_bucket.my_bucket

# Move resource within state (rename without destroy/recreate)
terraform state mv aws_s3_bucket.old_name aws_s3_bucket.new_name

Module Design

Good Module

module "api_service" {
  source = "../modules/service"

  name        = "order-api"
  environment = "production"
  image       = "order-api:v1.2.3"
  cpu         = 512
  memory      = 1024
  port        = 3000
  replicas    = 3

  database_url = module.database.connection_string
  vpc_id       = module.vpc.id
  subnet_ids   = module.vpc.private_subnet_ids
}

Module Principles

• Inputs are explicit - no hardcoded values inside modules
• Outputs expose what consumers need - connection strings, IDs, ARNs
• Defaults are sensible - dev-friendly defaults, override for prod
• Scope is clear - a module does one thing (a service, a database, a VPC)
• Version your modules - use git tags or registry versions

Change Safety

The Plan Review

terraform plan -out=tfplan    # Always save the plan

In the plan output, check:

• + create - new resources (usually safe)
• ~ update in-place - modify existing (review what's changing)
• -/+ destroy and recreate - DANGEROUS - resource will be deleted and recreated
• - destroy - DANGEROUS - resource will be deleted

Dangerous Changes (Require Extra Review)

Change	Risk	Mitigation
Rename a resource	Destroy + recreate	Use moved block or state mv
Change resource type	Destroy + recreate	Import new, remove old from state
Modify immutable fields	Destroy + recreate	Check if the field triggers replacement
Remove a resource	Deletion	Verify it's truly unused
Change database instance type	Potential downtime	Check if it requires restart
Modify security groups	Access changes	Review rules before and after

Protect Critical Resources

resource "aws_rds_instance" "production" {
  # ...
  lifecycle {
    prevent_destroy = true    # Terraform will refuse to destroy this
  }
}

Secrets in IaC

Never hardcode secrets in IaC files. Not even in .tfvars.

Approach	How
Environment variables	TF_VAR_database_password
Secret manager reference	data.aws_secretsmanager_secret_version
CI/CD secrets	Injected during pipeline execution
SOPS / age	Encrypted files in repo, decrypted at apply time

Drift Detection

When someone makes a manual change that doesn't match the code:

terraform plan    # Shows diff between code and reality

If there's drift:

1. Don't just apply - understand what changed and why
2. If manual change was correct: update the code to match
3. If manual change was wrong: apply to restore code's state
4. Prevent future drift: enforce "no console changes" policy, use CI/CD for all changes

CI/CD for Infrastructure

PR opened:
  → terraform fmt -check (formatting)
  → terraform validate (syntax)
  → terraform plan (show what would change)
  → Post plan output as PR comment

PR merged to main:
  → terraform plan (regenerate)
  → Manual approval gate (for production)
  → terraform apply
  → Verify health checks

Rules:

• Never apply without a plan review
• Production requires manual approval
• Plan output visible in PR (reviewers see what will change)
• State locking prevents concurrent applies

Reference: reference/iac-patterns.md - tool comparison, state management strategies, module patterns, common resource table, drift remediation, secrets handling, tagging strategy.

Review Checklist

• Plan reviewed - no unexpected destroys or recreates
• Secrets not hardcoded (using secret manager or env vars)
• Critical resources have prevent_destroy
• State is remote, encrypted, and locked
• Module inputs are explicit (no hidden assumptions)
• Environment differences are in variables, not code
• Naming is consistent and includes environment
• Tags applied for cost tracking and ownership