Most organizations run many systems that need to share data: a customer database, billing, a warehouse system, a web shop, reporting tools. Connecting each system directly to every other one creates a fragile web of point-to-point links. An integration layer provides a consistent way for systems to communicate, and it is a key enabler for modernizing legacy systems without breaking everything around them.
Synchronous APIs
An API lets one system request data or actions from another and wait for a response. HTTP-based APIs, often designed in a REST style or using gRPC, are the most common form. Synchronous calls are a good fit when the caller needs an immediate answer, such as checking stock before confirming an order.
Their limitation is coupling at runtime: if the called system is slow or unavailable, the caller is affected. Timeouts, retries with backoff and circuit breakers, which stop calling a failing service for a while, help contain these failures.
Message queues
A message queue, provided by brokers such as RabbitMQ or managed cloud queue services, lets a sender place a message that a receiver processes later. The sender does not wait and does not need the receiver to be online. Queues smooth out load peaks and allow work to be retried when processing fails.
Queues are well suited to tasks like sending emails, generating documents or forwarding orders to a fulfilment system. Designs must handle messages that cannot be processed, usually by moving them to a dead-letter queue for investigation.
Event streams and publish-subscribe
In publish-subscribe integration, a system publishes events such as "order placed" or "customer address changed", and any number of other systems subscribe. Platforms such as Apache Kafka store events in durable logs that consumers read at their own pace. This decouples producers from consumers: new consumers can be added without changing the producer.
Event-driven designs require agreement on event formats and versioning, and consumers must cope with duplicates and with events arriving in an unexpected order.
Adapters
An adapter wraps a system's native interface, which might be a file drop, a database table, a proprietary protocol or an old SOAP service, and exposes it in the form the integration layer expects. Adapters are especially useful for legacy systems that cannot be changed. They keep the rest of the architecture independent of the legacy interface, which can later be replaced behind the same adapter.
Data mapping and canonical models
Different systems describe the same thing differently: field names, formats, codes and identifiers vary. The integration layer translates between them. Some organizations define a canonical data model, a shared representation used for messages between systems, to reduce the number of translations. Others map directly between pairs of systems where only a few connections exist. Either way, document the mappings and their owners.
Reliability patterns
- Idempotent receivers: processing the same message twice should not cause duplicate effects.
- Correlation identifiers: pass an ID through every call and message so a transaction can be traced end to end.
- Transactional outbox: store outgoing events in the same database transaction as the business change, then publish them reliably.
- Schema versioning: evolve message and API formats in backward-compatible ways.
Monitoring and operations
Integration failures are often silent: a message waits in a queue or a nightly file is never picked up. Monitor queue depth, consumer lag, error rates and the age of the oldest unprocessed message. Log enough context to reprocess failed items, and give someone clear ownership of each integration.
Choosing the right mechanism
| Situation | Typical choice |
|---|---|
| Caller needs an immediate answer | Synchronous API |
| Work can happen later, one consumer | Message queue |
| Many systems react to the same change | Event stream / publish-subscribe |
| System cannot be modified | Adapter in front of it |
Key takeaways
- Use APIs for request-response needs and queues or events for decoupled work.
- Wrap legacy interfaces in adapters and translate data models explicitly.
- Design for duplicates, failures and tracing from the start.
- Monitor integrations as carefully as the systems they connect.