Transport success is not business success
An HTTP 200 response proves that one server accepted a request. It does not prove that a product was mapped correctly, an order was stored exactly once or a downstream system completed its own processing. Integration quality is determined by the full business outcome.
A robust design therefore separates transport, validation, processing state and business result. Each stage can fail differently and needs enough evidence for operators to understand what happened without reading production code.
Start with a precise data contract
The contract should define fields, types, units, identifiers, optional values, time zones and the meaning of absence. “Price” is incomplete without currency, tax semantics and precision. “Updated at” is incomplete without time zone and conflict rules.
Validation should reject structurally invalid data early and report actionable reasons. Business validation may require a later stage—for example, checking whether a referenced customer or warehouse exists. Those two failure classes should not be collapsed into an unspecified error.
Contract questions that prevent ambiguity
- Which system owns each field?
- Which identifier remains stable across systems?
- Does an omitted value mean unchanged, empty or invalid?
- How are money, units and time represented?
- What is the maximum payload and expected throughput?
- How are schema versions introduced and retired?
Design idempotency before adding retries
Network timeouts create uncertainty: the caller may not know whether the receiver completed the request. Retrying without an idempotency strategy can create duplicate orders, payments or stock movements.
An idempotency key identifies one logical operation across repeated deliveries. The receiver stores the result associated with that key and returns the same outcome for a safe retry. The key needs a defined scope and retention period; it is not merely a random header.
Some operations are naturally idempotent, such as setting a product field to a known value. Others require a business identifier or a dedicated request ledger. The choice belongs in the contract.
Retry only failures that may become successful
A timeout, a temporary provider outage or rate limiting can justify a retry. A malformed payload, unknown product identifier or violated business rule normally requires correction, not repeated traffic. Retrying every error hides permanent failures and creates unnecessary load.
Backoff and jitter spread repeated attempts over time. The retry budget should include maximum attempts or age, because an event that is technically deliverable days later may no longer be correct for the business.
A practical failure classification
- Transient: retry automatically with backoff
- Rate-limited: respect provider guidance and shared quotas
- Permanent data error: move to a correction path
- Authentication or configuration: alert an operator
- Unknown outcome: reconcile using an idempotency key or status endpoint
Make asynchronous processing explicit
Queues can decouple a slow or unreliable target from the originating request. They also introduce state: accepted, queued, processing, completed, failed and possibly awaiting correction. Returning “accepted” should not be presented as “completed”.
The sender or operational team needs a way to obtain final state. That may be a status endpoint, a callback, an event stream or a dashboard. The mechanism should use durable identifiers that connect the original request to later processing.
Each transition is recorded against a stable correlation identifier.
Log for reconstruction, not for volume
Useful logs answer which operation ran, against which business object, in which integration version, with which outcome and why. A correlation identifier should connect inbound request, queue message, downstream call and final state.
Logs must not become an uncontrolled copy of personal data, secrets or full payment payloads. Structured fields, redaction and retention rules should be designed together. Operators need enough context to recover without creating a second sensitive datastore.
Useful structured fields
- Correlation and idempotency identifiers
- Business object type and non-sensitive identifier
- Integration and schema version
- Attempt number and processing duration
- Normalised outcome or error class
- Target service and response category
Protect data consistency across system boundaries
A database transaction cannot usually span a shop, ERP and external provider. Each system can commit independently, which means partial success is normal rather than exceptional. The design must define how such states are detected and corrected.
An outbox can record the business change and the event to be published in the same local transaction. A consumer can record processed message identifiers with its own update. Compensating actions may reverse or neutralise a completed step when a later step fails.
The right pattern depends on business semantics. Reversing a notification is different from reversing a payment. Technical symmetry should not be assumed where the business cannot actually undo an operation.
Add reconciliation as an independent control
Even a well-designed event flow can miss a message because of configuration errors, retained dead letters or provider defects. Reconciliation compares authoritative states independently of the normal transport path.
Examples include comparing order counts and totals, checking that every exported product has a current remote identifier, or verifying that payment status agrees with the provider. Differences become explicit work items rather than silent drift.
Version integrations with overlap
A breaking contract change should not require every participant to switch at the same instant. Additive changes, explicit versions and a defined overlap period allow consumers to migrate independently.
Observability should reveal which version is still in use. A deprecation date without usage data is a hope, not a migration plan. Old versions can be removed when traffic, ownership and fallback have been checked.
Operate the integration as a product
A production integration needs ownership, runbooks and meaningful service indicators. Queue age, failure rate, retry volume, reconciliation differences and processing duration are often more useful than raw request counts.
Alerts should indicate an actionable condition. A single temporary failure may be normal; a growing queue, exhausted retry budget or sustained inconsistency requires attention. The correction process should be tested before an incident.