An integration receives an address update and then an event saying that the customer still has the old address. If it applies both messages in arrival order, the record ends up out of date. The problem is not that the second event is false: it may be valid, but it happened earlier.
Handling out-of-order events means distinguishing when a change happened from when it was received, then deciding which source of information can establish what should take precedence. There is no universal rule based on timestamps. The right choice depends on the event type, how the data is produced, and which system is authoritative for it.
Why events arrive out of order
A queue, webhook, or integration process can introduce delays between an event being generated and consumed. Deliveries may be retried, processed in parallel, or sent along routes with different latencies. As a result, two events for the same entity can arrive in a different order from the one in which they were created. Events may also be duplicated when a sender retries after receiving no confirmation.
Order may be preserved within one stream but not across different streams. For example, account changes might be distributed across partitions while changes to that account’s orders are processed independently. Do not assume that a queue or API guarantees global ordering unless that property is documented and holds throughout the actual processing path.
The effect depends on the message content:
- A full-state event describes a record as it stands at a particular point in time. Applying an old state later can replace more recent field values.
- A delta event describes a change, such as adding a label or incrementing a counter. Discarding it because it is older could lose a legitimate change that has not yet been applied.
Before choosing a rule, identify whether messages contain full states, partial changes, or a combination. The right strategy for a profile update may not work for a financial operation or an order-status transition.
Receipt time, event time, and business sequence
Receipt time indicates when the consuming system observed the message. It helps measure delay and order what that consumer received, but it does not prove when the original change occurred.
Event time is intended to reflect when a change happened or was recorded. It can help with diagnosis, but depends on the sender’s clock, precision, and conventions. Two systems may have clocks that are out of sync, and an event generated during a recovery may carry a timestamp that does not accurately represent business order. A later timestamp is not, on its own, sufficient proof of precedence.
Business sequence expresses the order that matters for an entity: for example, version 12 of a record follows version 11, or one operation is confirmed after another. If the source system provides a monotonic version or a reliable sequence number tied to the same entity, it is usually a stronger signal than a timestamp. Check its scope: it may reset, apply only within a process, or be unsuitable for comparing changes to different entities.
Record these values separately. Do not reuse a single field called “date” to determine precedence, measure delays, and preserve an audit trail; each purpose needs a clear interpretation.
How to identify stale updates without losing valid changes
An update is stale only in relation to a defined rule and the current state known to the consumer. For example, if it holds version 18 of a customer record and receives a full state for version 16, it can reject it as older. By contrast, it should not automatically discard a valid delta just because its timestamp is earlier: the delta may add information that is missing from the later event.
Evaluate each message by checking:
- Identity and scope: Which entity, field, or aggregate does it concern? Two changes may be independent and need no choice between them.
- Type and version: Does it contain a full state or a delta, and can its sequence be compared with the one already applied?
- Preconditions: Is the transition valid from the current state? A change to “shipped” should not reverse an order that is already “delivered” without an explanation.
- Authority: Which system can determine the value of this field? A profile update and a correction from the master system do not necessarily carry equal weight.
The comparison and write must be consistent. If two consumers read version 16 and both try to save changes, a check performed before writing may become stale. Use a write conditional on the observed version, or an equivalent storage operation. This lets the consumer detect if the record changed before its own update was committed.
Criteria for resolving conflicts
Prioritize the most reliable signal that actually exists; do not invent a hierarchy from fields that do not guarantee order. These criteria can be combined, provided their meaning is documented:
- Per-entity version or sequence: Accept a full state if its version advances; ignore a version that has already been applied or is older. Define how to handle gaps, resets, and missing versions.
- State and permitted transition: Validate the change against a state machine or business rules. An invalid transition should trigger review, not silently become an update.
- Authority by field: Assign ownership of each data item to a responsible system. If two sources can legitimately edit the same field, set an explicit policy, such as review or an agreed priority, rather than improvising a “last message received” rule.
- Content comparison: Apply idempotent deltas where possible and avoid replacing an entire record with a partial message. If the event does not reveal what changed, query the source of truth before overwriting.
“Last write wins” may be acceptable for low-impact fields if clocks and policy are sufficiently controlled. It is not a neutral rule: it can erase valid changes and conceal conflicts. For sensitive decisions, preserve both versions or prevent automatic resolution.
What to do when there is no reliable sequence
If the sender does not provide a comparable version, do not try to infer an exact order from receipt time. Choose a response based on the impact of getting it wrong:
- Read the current state from the source system before updating, if a reliable read API is available. This reduces reliance on a delayed message, although the read and write can still race with later changes.
- Apply only independent fields when the event clearly identifies what it changed. Keep the other values and avoid full replacements.
- Hold the conflict for reconciliation when context is missing or the data is important. Keep the message, entity, and reason for the decision so the case can be reviewed.
- Define source priority when the business has determined which system governs a data item. Communicate the rule to the teams and review its exceptions.
A review queue or a later query must not become an invisible limbo. Decide who handles these cases, how they are retried, and what indicates they have been resolved. If an automatic decision is not possible, showing uncertainty is safer than presenting doubtful data as current.
Testing and logs for verifying behavior
Test the integration contract, not just the ideal path. For one entity, send two changes in reverse order; replay an already processed event; delay one during a retry; and cause two updates to be written almost simultaneously. Include equal, missing, or unexpected versions if those cases are possible at the source.
Check not only the final value, but also the decision: accepted, ignored as a duplicate or stale, merged, held, or rejected because of an invalid transition. Verify that repeated messages do not cause additional effects and that valid events are not lost because of a date-only rule.
Log the event identifier, entity, source, available version or sequence, event and receipt times, previous and resulting versions, and reason for the decision. Avoid including unnecessary personal data in logs. Monitor message age and conflict volume: an increase may point to delays, retries, or a sender-side change, not necessarily a consumer failure.
Decisions to make before launching the integration
- Does each message represent a full state, a delta, or either type?
- Which field establishes precedence, and who guarantees its scope and continuity?
- Which system is authoritative for each data item, and which changes can each source accept?
- How are duplicates and concurrent writes that invalidate a check prevented?
- What happens when sequences are missing, transitions are invalid, or conflicts are ambiguous?
- Who reviews held cases, and how is the time to resolution measured?
The practical approach is to use the most reliable ordering signal available, limit each event to the data it actually changes, and make uncertainty explicit. That way, a delayed message can be recognized without blindly discarding legitimate changes or letting an old update overwrite the current state.