A backfill can fill in missing data in a connected system, but it can also repeat actions that do more than save records. Before running one, define what will be reprocessed, which system is authoritative, and which side effects might be triggered. The goal is not simply to send old data: it is to fix a verifiable discrepancy without disrupting current operations.
Backfills, migrations, and retries are different jobs
A backfill reprocesses a defined set of historical data to complete or correct information in a connected system. For example, it may send records from before an integration was enabled or fill in a field that was not previously transmitted.
A migration, by contrast, moves data between systems or changes its structure, model, or destination. It may include a backfill, but usually involves additional decisions about transformation, field mapping, and how systems coexist. A retry of failed events reprocesses specific operations that did not complete, typically identified by an error or a pending queue. It is not the same as processing the entire history.
This distinction helps you choose the right solution. If records are missing because they were never sent, a backfill may be needed. If the receiving system has the data and only its structure needs to change, that is a migration. If events have known errors, it is usually better to fix and retry them than to resend all historical records.
Decide whether reprocessing is the right option
Before designing the process, describe the problem as an observable discrepancy: what data is missing, where it should be, and since when. If the source contains the correct value and the destination does not, a backfill may be appropriate. If the value is wrong in the source, sending it will only reproduce the problem; correct it there first or define an explicit transformation.
Synchronizing the current state may be enough. If the destination only needs to know the latest state and does not need every historical change, sending all intermediate states adds complexity without necessarily adding value. However, if events are needed for auditing, traceability, or calculations, preserving their order and historical detail may be important.
- A backfill makes sense when eligible, verifiable records never reached the destination, or an integration rule left fields incomplete.
- Correct the source when the authoritative data is wrong or incomplete before transfer.
- Retry events when the problem is limited to identifiable failed operations and the mechanism supports resuming them.
- Synchronize the current state when the destination does not need to reproduce the history of changes.
Also assess the operational impact. A record might trigger a notification, a user-visible update, a task assignment, or a financial operation. If you cannot determine which actions will be triggered, do not start with the full history.
Define the records and identify the source of truth
Specify the data set using rules that another person could repeat. Define dates and time zone, included statuses, record types, exclusions, and eligibility conditions. Clarify whether the time range uses the creation date, update date, or event occurrence date: these are not necessarily equivalent.
Also identify the key that links each record across systems. A stable key makes it possible to determine whether the destination already contains the object and whether to create, update, skip, or review it. Do not assume two records are the same just because they share a name or address; use the identifiers defined by the integration and check how missing or repeated values are handled.
Designate a source of truth for each field. The source system may be authoritative for an email address while the destination controls another attribute. Without this decision, a backfill could overwrite legitimate changes made in the receiving system. Document which fields are written, which are preserved, and what to do when values conflict.
Choose a method and protect against side effects
A batch upload may be suitable when transferring a known data set and the receiving system supports controlled imports. An API can process records with validations and per-item responses, but requires handling limits, failures, and resumption. Event replay can be useful when change history matters and there is a safe way to identify events that have already been processed. No option is automatically better: consider the guarantees it offers for ordering, duplicates, errors, and observability.
Before sending data, check whether the operation is idempotent: repeating the same request with the same key should prevent a second unwanted effect. If it is not, define how to detect duplicates and resume after an interruption without starting over. Use operation identifiers or progress markers when the system supports them, and keep a record of what was attempted and how the system responded.
Where possible, separate writing the data from its associated effects. For example, an import mode that suppresses notifications or a temporary suppression rule may reduce risk, if available and approved. If charges, communications, or irreversible state changes are involved, establish explicit controls before processing. Do not assume a historical update will be treated like a routine update.
Order can also change the outcome. If records are related or changes are sequential, determine whether parent records should be processed before their dependencies or whether each change must follow its chronological sequence. If only the final state matters, document that rule and avoid replaying transitions that could trigger actions.
Test, run in stages, and stop at the right time
Test a representative sample that includes different statuses and dates, incomplete cases, and situations with possible duplicates. Compare the expected outcome with the destination’s actual behavior, including side effects. A sample made up only of straightforward records can hide the riskiest cases.
Agree on metrics before starting: records selected, processed, created, updated, skipped, and failed. Also define how to measure the difference between source and destination. The figures must refer to the same data set and eligibility rules; comparing overall totals can confuse excluded records with missing ones.
Run the backfill in stages: start with a small sample, move to a limited range, and process the rest only if the results are acceptable. Assign someone to monitor the process and set stop conditions, such as an error rate above the agreed threshold, unexpected duplicates, an increase in side effects, or growing discrepancies. Stopping should be a planned decision, not an improvisation.
To resume, preserve the progress point and separate recoverable failures from those requiring intervention. Fix the cause before retrying; otherwise, the same error will recur. Rollback is not always possible, especially if communications were sent or external actions were performed. Define in advance whether the recovery plan is to restore data, compensate for effects, or stop further writes while the issue is investigated.
Reconcile and close out the work
When the process is complete, compare the expected records with the destination results using the agreed keys. Classify discrepancies as ineligible records, technical errors, data conflicts, or cases requiring manual review. Do not declare success just because the process finished without interruption; reconciliation must account for discrepancies.
Document the scope processed, exclusions, outstanding errors, decisions about conflicts, and any observed side effects. Keep enough information for support and operations teams to answer later questions. If the discrepancy involves data that continues to change, define how updates made during the backfill will be captured so they do not create a new gap.
Pre-production checklist
- Is it clear which discrepancy is being fixed and why a retry or current-state sync is not enough?
- Are dates, statuses, exclusions, identifiers, and the source of truth for each field defined?
- Has behavior been checked for duplicates, repeated requests, and partial failures?
- Are notifications, charges, state changes, and other triggered actions understood and controlled?
- Did the test sample cover representative cases, and were results verified in the destination?
- Are there metrics, an operational owner, stop conditions, and a recovery plan?
- Is there a reconciliation method and a list of discrepancies that may need review?
If a key answer is unknown, limit the scope and resolve that uncertainty before scaling up. A safe backfill depends less on processing data quickly than on being able to explain what was sent, what changed, what remains outstanding, and how unwanted effects will be prevented from happening again.