Batch and Migration Integrations
Batch work needs different controls than day-to-day operational syncs.
Principles
- split work into bounded chunks
- persist checkpoints aggressively
- throttle intentionally
- monitor error rates and queue depth
- be able to resume without duplicate side effects
Recommended batch structure
- select a deterministic slice of records
- write or fetch in pages
- persist success and failure separately
- retry transient failures only
- escalate permanent failures for manual review