System integration · APACHE KAFKA · insurance
Insurance system integration with Apache Kafka for centralized message and log distribution
Inteca designed and implemented an integration platform that connects an insurer’s source and target systems through a central data exchange layer, with reliable distribution, processing and archiving of logs and messages.
Technologies and solution components
Java 17
Apache Kafka
Kafka Streams
Kafka Connect
JMS
S3
ELK
JDBC
Client
An insurance organization with a distributed system landscape
The client is an organization operating in the insurance sector, with source and target systems exchanging service calls, logs and other communication data.
The organization needed a central mechanism for collecting this information, processing it reliably and making historical data available for analysis, correlation and verification.
The solution also had to support audits, incident analysis and investigation of reported issues.
Client profile
Anonymized at the client's request
- Industry
- Insurance
- Market
- Poland
- Area
- System integration
- Project team
- 7 people
Due to the client's information policy, the organization's name cannot be disclosed. The project scope, architecture, technologies and results are described in full.
Context
Why insurance system integration becomes complex
Insurance system integration becomes complex because insurance environments often include multiple interconnected systems that process different parts of the same business event. These may include policy administration, claims, billing, customer portals, agent platforms, legacy applications and other business applications using different technologies, formats and processing models.
A single transaction may therefore pass through several systems before it is completed. If each application records only its own part of the interaction, reconstructing the full data flow later becomes difficult. This is particularly important when teams need a reliable audit trail for troubleshooting, incident analysis or operational reviews.
What technical teams need to establish
Starting point
Fragmented communication data across connected systems
Information about service calls and communication between the client's systems was not available through one central mechanism.
The client needed to collect this data in one place so that it could be analyzed, correlated and later verified during audits, incident analysis or investigation of reported issues.
Centralized data collection
One consistent mechanism for collecting communication data from source systems and making it available for analysis and verification.
Real-time processing
The ability to process incoming data continuously, including during periods of increased system load.
Reliable message delivery
Protection against losing data between a source system and its destination, including situations in which a downstream system is temporarily unavailable.
Access to historical data
A centralized history supporting analysis, event correlation, audits, incident investigation and support cases.
Persistent storage
Durable storage and message queuing mechanisms reducing the risk of records being lost while information is transferred between systems.
Constraint
How did the integration layer work with existing systems?
The new integration layer had to work with technologies already operating in the client's environment. Source communication included JMS-based systems, while messages were processed in XML and CSV formats.
The Kafka-based platform was integrated with systems already operating in the client's environment.
Apache Kafka provided a common layer for buffering, processing and distributing data, while Kafka Connect enabled integration with existing source and target technologies.
Integration context
Where does event-driven integration fit among common integration patterns?
Enterprise environments use different integration patterns depending on how quickly information must move, how many systems consume it and how tightly applications should depend on one another.
Point-to-point
Directly connects one system with another. Simple and efficient for a small number of applications.
Complexity grows as sources and destinations are added: each integration introduces another direct dependency, and a change in one application may require changes across several connections.
Batch
Transfers information at scheduled intervals, for example through files or periodic database processes.
Useful when data does not need to be available immediately. Less suitable when systems need continuous visibility or downstream applications should receive information as events occur.
Event-driven
Producers and consumers are decoupled. A source publishes to an intermediary platform, while downstream systems consume data independently.
This approach fits scenarios where data must reach multiple destinations, systems process information at different speeds or temporary unavailability of a consumer should not interrupt message processing.
In this project, Apache Kafka provided the event-driven layer used to centralize the distribution of logs and messages between source and target systems. Kafka supported persistent buffering, processing and reliable message delivery while integrating with systems already operating in the client's environment.
The solution
A central data exchange layer based on Apache Kafka
Inteca built a central data exchange layer based on Apache Kafka to buffer, process and distribute logs and messages between the insurer's source and target systems. The platform works with systems already operating in the client's environment and avoids the need to redesign existing applications.
The architecture is modular. New sources and destinations can be connected through reusable integration mechanisms and connector configuration instead of building each integration from scratch.
Apache Kafka broker
The Apache Kafka broker receives, stores, buffers and distributes messages between systems. Persistent storage allows messages to remain available even when a downstream system is temporarily unavailable, reducing the risk of data loss.
Kafka Streams
Kafka Streams handles real-time processing of messages inside the integration layer. It transforms, filters and routes data before it reaches the appropriate target systems.
Kafka Connect
Kafka Connect links Apache Kafka with the surrounding source and target systems. Four connectors were implemented in the project, reducing the need for dedicated integration code and supporting the modular architecture.
S3 backup
Processed messages are additionally archived in S3 object storage. This provides an independent copy of the data alongside Kafka's persistent storage and the target repositories.
Modular integration architecture
The integration platform can be extended with new sources, target systems, connectors and processing mechanisms without rebuilding the entire solution. For the insurer, this creates a more scalable integration layer that can evolve together with the existing application landscape.
Four Kafka Connect connectors were implemented: JMS Source Connector for receiving JMS messages, S3 Sink Connector for object storage, ELK Sink Connector for log analysis and visualization, and JDBC Sink Connector for relational databases.
Architecture
From source systems to target repositories
The integration architecture moves data through four stages: ingestion, persistent buffering, real-time processing, and delivery to target repositories. Apache Kafka provides the central layer between the insurer's source and target systems, while Kafka Streams and Kafka Connect handle processing and distribution.
Receiving data from source systems
The integration platform receives messages from existing source systems, including JMS-based applications, and processes data in XML and CSV formats. This allows the insurer to connect legacy systems to the new integration architecture without rebuilding the source applications.
Persistent buffering in Apache Kafka
Apache Kafka persistently stores and queues incoming messages before they are delivered to target systems. If a downstream consumer is temporarily unavailable, the message remains in Kafka until it can be processed. This reduces direct dependencies between source and target applications and lowers the risk of message loss.
Real-time stream processing
Kafka Streams transforms, filters and routes messages before they reach their target systems. This creates a consistent real-time data pipeline in which processing logic is handled centrally and data can be adapted to the requirements of individual consumers.
Delivery and archiving
Kafka Connect delivers processed data to three target environments: S3 object storage, a relational database through JDBC, ELK. An additional S3 backup archives messages independently.
How does the shared integration layer simplify data flows?
The architecture provides a shared layer for receiving, buffering, processing, routing and archiving communication data. Source and target systems can exchange data through the Kafka-based integration platform, reducing the need for dedicated integration logic between individual systems.
Centralized communication data also improves operational visibility. Technical teams can access historical information when they need to analyze incidents, verify data delivery or investigate reported issues.
Result
One view of communication across systems
The client gained a consistent view of what happens between systems, available without manually collecting and correlating information from several different sources.
Regulated context
Why does traceability matter in insurance system integration?
Insurance processes may involve multiple applications, making historical communication data useful when teams need to reconstruct how systems interacted during a business or technical event.
In this project, centralized data supports analysis, correlation and later verification during audits, incident investigations and investigation of reported issues.
For insurance organizations, this type of traceability provides a clearer technical record of communication between connected systems.
Takeaways
What does this project demonstrate about insurance system integration?
Insurance system integration can introduce a central data exchange layer while continuing to work with technologies already operating in the organization. In this project, Apache Kafka was integrated with JMS-based source systems and XML and CSV message formats.
Kafka Connect linked the platform with source and target technologies, while Kafka Streams handled real-time transformation, filtering and routing.
The result is a modular integration architecture that centralizes communication data, improves delivery reliability and makes historical information easier to access for analysis, audits and incident investigation.
Need to integrate systems across your insurance environment?
Inteca designs and implements integration architectures for complex enterprise environments, including insurance and other regulated industries. We connect existing and legacy applications with modern event-driven platforms, design real-time data pipelines, implement Apache Kafka, Kafka Connect and Kafka Streams, and build integration layers that evolve together with the application landscape.
