System integration is the work of making separate applications behave as one — exchanging data on agreed contracts, in agreed formats, at an agreed pace. It is less a technology choice than a set of decisions about coupling, ownership and failure. This guide covers the patterns, the constraints and the way integration programmes actually run.
Quick Overview
What you’ll learn:
- What integration delivers beyond “the systems can talk to each other”
- The directions integration runs in, and why direction changes what is hard
- Which foundations decide whether an integration ages well or rots
- How the main patterns differ, and which one an estate has usually outgrown
- Where regulation constrains the design rather than decorating it
- What an integration programme looks like from requirements to steady-state operation
Who this article is for:
- Architects choosing between direct connections, a broker and an event backbone
- Engineering leads who have inherited an estate of hand-written connectors
- Programme managers responsible for adoption once the pipes are working
Reading time: 11 minutes
What Integration Actually Solves
Integration is usually commissioned in the language of plumbing: connect the ERP to the shop, push leads into the marketing platform, stop the finance team re-keying payroll data. The plumbing is real, but it is the smaller part of the work. What an integration establishes is an agreement — about which system owns a fact, what that fact is called, how often it changes hands, and what happens when the exchange fails.
That agreement is where the return comes from. Removing silos, ending duplicate data entry and making better use of systems already paid for are all downstream of a single decision: naming one owner for each piece of data and making every other system a reader of it. Estates that skip that decision end up connected and still inconsistent — the same customer carrying different addresses in different places, each of them technically synchronised.
It also explains why the brief matters more than the tooling. “Connect these systems” produces a connector. “Make the order status in the shop match the order status in the warehouse inside a defined window, and make it obvious when it does not” produces a design. The first brief is finished when data moves; the second is finished when someone can answer for the data. Our guide to IT systems integration and the removal of data-flow barriers works through the diagnostic side of that gap.
The Directions Integration Runs In
Four directions cover almost every project, and the direction usually decides what will be hard about it.
Vertical integration connects systems at different levels of the same organisation — production or shop-floor systems into an ERP, for instance. The technical surface tends to be narrow, but the data is operational and time-sensitive: a delayed message is a wrong stock figure, not a stale report.
Horizontal integration connects systems at the same level across departments — CRM into marketing automation, service desk into asset management. The technical work here is often the easy part, because two departments have to agree on definitions they have each been using differently for years.
Internal integration stays inside a single unit. It is the cheapest place to settle conventions — naming, error handling, retry policy, who gets paged — before those conventions have to be negotiated across a boundary where nobody reports to the same person.
External integration reaches out to partners, suppliers and customers. Everything tightens: contracts are versioned in public, credentials belong to somebody else, and the failure you are debugging may be a partner’s incident rather than your own. Organisations routinely underestimate this step, because internal experience transfers less well than it looks.
Foundations: Architecture, Interfaces and Data Contracts
Three foundations decide whether an integration is still serviceable years later, and none of them is a product choice.
Architecture has to be additive: connecting another system must not require re-opening the ones already connected. That is a property of how responsibilities are divided, not of how many boxes the diagram carries. Failure isolation belongs here too — an integration layer that takes the estate down with it has traded a set of small outages for one large one.
Interfaces are the contract, and they should be written as one. Describing an HTTP interface in a machine-readable document is the practical minimum; the OpenAPI Specification, now on its 3.1 line, exists so that the description, the generated client and the tests come from a single artefact rather than from three people’s recollections. Whether the style is REST, GraphQL or gRPC matters less than whether the contract is versioned and whether a breaking change is visible before it ships. For teams making that choice deliberately, the API Design & Management — REST, GraphQL, gRPC course covers the trade-offs and the governance that has to sit around them.
Data contracts are where integrations quietly fail. Standardising formats and structures across an organisation sounds like a data-quality exercise; it is really the precondition for the interfaces above to mean anything. Where an industry has already done this work — clinical messaging, logistics document exchange — adopting the established standard is almost always cheaper than inventing a local dialect, because the standard arrives with a community that has already found the ambiguities.
Patterns: Point-to-Point, Hub-and-Spoke, Service Bus and Event Bus
Four patterns cover the field, and an estate can usually tell you which one it has outgrown.
Point-to-point wires two systems directly. It is the fastest thing to build and the cheapest thing to start, which is why almost every estate begins here. The cost arrives later and grows faster than the system count: each new participant can need a connector to several existing ones, and every one of those connectors is a place where a schema change breaks something unrelated at an inconvenient hour.
Hub-and-spoke puts a broker in the middle. Translation and routing happen once, in a place with a name and an owner, so adding a participant is a change to the hub rather than to every peer. The trade is explicit: the hub becomes a capacity constraint and a single point of failure, and it needs the operational attention that implies.
Middleware and the enterprise service bus generalise the hub into a shared channel with transformation, routing and protocol mediation built in. The flexibility is real and so is the cost — a bus that nobody owns turns into a place where business logic accumulates invisibly, which is the failure mode that gave the pattern its reputation.
The event bus inverts the conversation: producers publish facts, consumers subscribe, and neither needs to know the other exists. The Apache Kafka Documentation puts concrete numbers behind that promise — the broker’s default retention keeps a topic’s records for 168 hours whether or not anyone has read them, so a consumer that was down over a weekend catches up instead of losing the data. The discipline this pattern demands is schema governance: without it, publishing a fact becomes publishing whatever the producer happened to serialise. Teams heading in this direction usually meet the same design questions as service decomposition, which our complete guide to microservices architecture sets out with the pitfalls attached.
Integration in Practice: Common Shapes
The same patterns look different depending on who is applying them.
A large enterprise consolidating ERP, CRM and supply-chain systems generally lands on a broker or a bus, because the participant count makes anything else unmaintainable. The hard part is rarely the technology; it is agreeing a canonical definition of a customer, an order and a product across divisions that have each been optimising their own.
A public-sector body integrating registry and benefits systems faces the opposite balance. The technical patterns are ordinary, but the constraints — statutory retention, lawful basis, auditability of every read — shape the design from the start, and a service-oriented approach with strict, standardised contracts tends to survive changes of policy better than direct connections do.
A small software firm integrating with its clients’ systems has no estate to speak of; it has other people’s. Direct connections and a thin middleware layer are usually the right answer here, and the real discipline is versioning: the interface belongs to a contract with a customer, so a breaking change is a commercial event rather than a deployment.
Security and Compliance at the Integration Boundary
An integration is a new path into data that previously sat behind one application’s access control, and it deserves to be treated as such. Encryption in transit and at rest, authentication that identifies the calling system rather than a shared secret pasted into a configuration file, and authorisation scoped to the fields actually needed are the baseline. So is logging that records which system read what, because an integration without an audit trail turns every later question into speculation.
Regulation is not decoration here; it constrains the design. The General Data Protection Regulation sets the clock explicitly — a personal-data breach has to be reported to the supervisory authority without undue delay and, where feasible, within 72 hours of becoming aware of it. An estate that cannot say which systems held a given record, and which interfaces moved it, cannot meet that obligation on time regardless of how good its security tooling is. Data-flow documentation is therefore a compliance artefact, not architectural housekeeping.
The practical consequence is a design rule: minimise what crosses the boundary. An interface that returns a whole customer record because it was easier than defining a narrower one has made every future review harder and every future incident wider. Deciding what a partner genuinely needs is cheaper before the contract is published than after.
How an Integration Programme Actually Runs
Five stages describe most programmes, and the ones that go wrong usually compressed the first.
Requirements means naming the systems, the data and the owner of each fact, plus the acceptable delay and the behaviour on failure. Design turns that into a pattern choice, an interface specification and a rollout order. Implementation builds the connectors, the transformations and the migration of whatever already exists in the wrong shape. Testing covers correctness, load and security, and it has to include the paths where the far side is slow or unavailable, because those paths are the ones production will exercise. Operation is where the programme actually lives: monitoring, alerting, and a route by which a broken synchronisation becomes somebody’s task rather than a mystery.
The stage that gets least attention is adoption. An integration that works technically still changes how people do their jobs — the double entry they no longer perform, the report they can no longer reconcile by hand, the exception process that now has a system of record. Where that change is not managed, teams keep the old spreadsheet running alongside the new pipe and the estate ends up with both. The Adaptation in the project — strategies for effective change management course addresses exactly that gap between a working system and a used one.
Tooling: Platforms, Pipelines and Observability
Tooling divides into families, and mixing up what each is for is a common source of expensive detours.
Integration platforms — the commercial iPaaS offerings and the open-source engines alike — provide connectors, transformation and routing so that common exchanges do not have to be hand-written. They earn their cost when the estate has many similar integrations, and they earn very little when it has a handful of unusual ones.
Data movement and preparation tools cover the batch end: extraction, transformation and loading between operational systems and analytical stores, where the constraint is volume and correctness rather than latency. Protocol choice belongs in this conversation too, especially where devices are involved. MQTT Version 5.0 defines three delivery guarantees — at most once, at least once, exactly once — and choosing the wrong one is how telemetry pipelines end up either lossy or full of duplicates that nobody expected. Our article on IoT connectivity in the factory: network architecture and protocols works through those choices where they bite hardest.
Observability is the family most often deferred. Integrations fail quietly: a queue backs up, a nightly job stops running, a field arrives empty. Monitoring that reports on the pipe rather than the payload will show green throughout. What is needed is a measure of the business fact — orders synchronised against orders placed — with an alert when the difference stops being zero.
Where Integration Programmes Go Wrong
The recurring failures are not exotic. Complexity gets underestimated because the first connector is genuinely easy and the tenth is not; effort estimated from the first is wrong by an order of magnitude. Cost is counted narrowly, covering licences and build while omitting the operating burden, the training and the downtime during cutover — the parts that continue after the project closes.
Ownership stays vague. An integration with no named owner degrades the moment either endpoint changes, and endpoints change constantly. Security is bolted on after the exchange works, at which point narrowing an over-broad interface means renegotiating with everyone already consuming it.
And the data model is treated as a detail. Two systems can exchange messages perfectly while disagreeing about what a customer is, and that disagreement surfaces months later as a reconciliation problem nobody can attribute. Cheap to settle at design time; expensive to unpick once both systems have been operating on their own definition.
Build Your Skills
The distance between “the systems are connected” and “the estate is integrated” is mostly the material above — ownership decided, contracts versioned, patterns chosen for the estate you actually have, and operations designed before the first cutover. The Advanced SOA — architecture, governance, integration and cloud readiness course works through that ground with the governance and cloud-readiness questions attached rather than deferred. Check the programme and sign up to build your skills with EITT’s experts.
Frequently Asked Questions (FAQ)
Which integration pattern should we start with?
Start with the pattern that matches the estate’s trajectory, not its current size. If the systems involved are unlikely to multiply, direct connections are honest and cheap. If more participants are coming, a broker or an event bus pays for itself well before the connector count becomes uncomfortable — and retrofitting one over a mesh of point-to-point links is considerably more expensive than choosing it early.
How do we know an integration is working?
Not from the pipe. A working integration is one where a business fact on one side matches the same fact on the other inside the window you agreed, and where a divergence raises an alert rather than a support ticket weeks later. If the only monitoring you have reports on connectivity, you will find out about failures from users.
Do we need an integration platform, or is custom code enough?
It depends on repetition. Custom code is reasonable when the estate has a small number of unusual exchanges and a team that will still be there to maintain them. A platform pays off when there are many similar exchanges, when non-specialists have to make changes, or when connector maintenance has started to consume a meaningful share of an engineering team’s time.
Who should own an integration once it is live?
Someone named, with the endpoints in scope. The common anti-pattern is ownership by the project that built it, which dissolves at go-live. Integrations break when either endpoint changes, so ownership has to sit where changes to those endpoints are known about in advance — usually the platform or architecture function rather than either application team alone.
What is the most underestimated cost in an integration project?
Operation. Building the exchange is a bounded piece of work; keeping it correct through schema changes, version upgrades, credential rotation and partner-side incidents is not. Budgeting the build without budgeting the years after it is the reliable way to arrive at an estate of integrations that technically exist and nobody trusts.