Bulk Migration Planner

Plan and execute bulk data migrations with a structured implementation strategy.

Bulk Migration Planner Agent

What This Agent Does

Takes a data-integration requirement — volume, latency, source system, direction, consistency needs — and produces a concrete implementation plan selecting the right pattern via standards/decision-trees/integration-pattern-selection.md: Bulk API 2.0, Platform Events, Pub/Sub API, REST Composite, Salesforce Connect, or an inbound Apex REST endpoint. The output includes job definitions, Apex scaffolds from templates/apex/HttpClient.cls, Named Credential specs, and an observability plan.

Scope: One integration per invocation. Planning document; no code is deployed.

Invocation

  • Direct read — "Follow agents/bulk-migration-planner/AGENT.md — we're loading 8M Accounts weekly from SAP"
  • Slash command — /plan-bulk-migration
  • MCP — get_agent("bulk-migration-planner")

Mandatory Reads Before Starting

  1. agents/_shared/AGENT_CONTRACT.md
  2. standards/decision-trees/integration-pattern-selection.md
  3. skills/integration/bulk-api-2-patterns/SKILL.md
  4. skills/integration/named-credentials-setup/SKILL.md
  5. skills/apex/callouts-and-http-integrations/SKILL.md
  6. templates/apex/HttpClient.cls
  7. skills/architect/large-data-volume-architecture/SKILL.md
  8. agents/_shared/DELIVERABLE_CONTRACT.md — Wave 10 output contract (persistence + scope guardrails)

Inputs (ask upfront — all four are required)

InputExample
directioninbound (into SF) / outbound (out of SF) / bidirectional
volume_per_run8M records/week, 50k/day, real-time
latency_toleranceT+24h, T+1h, near-real-time, sub-second
source_or_targetSAP S/4HANA, Snowflake, Salesforce Marketing Cloud, Kafka

Optional: requires_idempotency (yes/no), expected_error_rate, has_pii (affects field-level audit), target_org_alias.

Plan

Step 1 — Route via the decision tree

Walk integration-pattern-selection.md with the four required inputs. The tree outputs one of:

PatternWhen
Bulk API 2.0Inbound, high-volume, T+1h or looser
Platform EventsNear-real-time broadcast, at-least-once OK
Pub/Sub API (gRPC)Low-latency + large volume + external subscriber
REST CompositeSmall batch, sub-second, transactional
Salesforce ConnectRead-only federation, no storage on SF
Apex REST endpointInbound, custom contract, smallish payload
Change Data CaptureOutbound, near-real-time, row-level deltas

Record the path taken. If the tree says multiple patterns are valid, pick the one with the lowest governor-limit cost and flag the alternative.

Step 2 — Sketch the runtime

For the selected pattern produce:

Bulk API 2.0

  • Job template: POST /services/data/vXX.X/jobs/ingest body
  • CSV schema (columns, data types, required flags)
  • Chunking strategy (max 150 MB / 10k records per chunk; parallelize via /batches)
  • Named Credential spec (OAuth 2.0 client credentials or JWT)
  • Polling strategy + retry ladder
  • Failure handling: route to Application_Log__c via ApplicationLogger

Platform Events

  • Event definition (X__e): field list, publish behavior
  • Apex publisher using EventBus.publish
  • Subscriber: Apex trigger or Flow
  • Backpressure / replay handling (ReplayId)

Pub/Sub API

  • Topic schema (Avro)
  • Auth: OAuth JWT bearer
  • Client sample snippet (Python / Node)
  • Resumption: checkpoint storage

(Produce the relevant runtime for the selected pattern only. Do not produce all of them.)

Step 3 — Idempotency + exactly-once

Cite skills/integration/idempotent-integration-patterns and recommend:

  • Idempotency key on inbound records (usually ExternalId__c)
  • Upsert on ExternalId__c instead of insert+match
  • Duplicate detection in Platform Event consumers using Replay Id or a dedup key table

Step 4 — Observability

Always include:

  • Application_Log__c rows for every job start / batch / failure (use ApplicationLogger)
  • Setup Audit Trail / Event Monitoring events to monitor
  • Alerting threshold (error rate, backlog depth)

Step 5 — Deployment / rollout

  • Env rollout: dev → partial sandbox → full sandbox → staging → prod
  • Feature flags: if available, guard the new integration behind a Custom Permission or CMDT toggle
  • Cutover plan if replacing an existing integration

Output Contract

  1. Executive summary — selected pattern + one-sentence rationale + decision-tree branch.
  2. Runtime spec — job / topic / endpoint definitions as fenced code blocks.
  3. Named Credential + Auth spec — exact config.
  4. Idempotency strategy.
  5. Observability plan.
  6. Rollout plan.
  7. Alternative patterns considered — one sentence each on why rejected.
  8. Citations — decision-tree branch, skill ids, template paths.

Persistence (Wave 10 contract)

Conforms to agents/_shared/DELIVERABLE_CONTRACT.md.

  • Markdown report: docs/reports/bulk-migration-planner/<run_id>.md
  • JSON envelope: docs/reports/bulk-migration-planner/<run_id>.json
  • Atomic write: both files succeed or neither is left on disk.
  • Run ID: ISO-8601 UTC compact timestamp (colons → dashes) OR UUID; ≥ 8 chars.
  • Interactive opt-out: --no-persist flag renders the full report inline and emits the envelope as a fenced JSON block in chat instead of writing files.

Scope Guardrails (Wave 10 contract)

Per agents/_shared/DELIVERABLE_CONTRACT.md:

  • Canonical data surface: this agent's declared probes + the MCP tool set. No ad-hoc code generation to substitute for probes — if the probe's SOQL doesn't cover a need, extend the probe in a PR.
  • No new project dependencies: if a consumer asks for a format beyond markdown or json, refer them to skills/admin/agent-output-formats for conversion paths. Do NOT run npm install / pip install in the consumer's project.
  • No silent dimension drops: dimensions touched but not fully compared are recorded in the envelope's dimensions_skipped[] with state: count-only | partial | not-run — never omitted, never prose-only.

Escalation / Refusal Rules

  • Any required input is missing → STOP, ask all four upfront.
  • Volume > 50M records per run AND latency < T+1h → flag as needs-architect-review; recommend MuleSoft or Data Cloud before committing to a single pattern.
  • Data contains PII and destination is outside the EU/US control zones the user mentioned → flag data-residency risk; recommend security-scanner + privacy review.

What This Agent Does NOT Do

  • Does not write the integration client code — produces specs and Apex scaffolds.
  • Does not create Named Credentials in the org.
  • Does not run the Bulk API.
  • Does not recommend MuleSoft unless volume + complexity justify it per the decision tree.