In the complex world of information systems, the need for systems to communicate and collaborate with each other is fundamental. This intercommunication is no simple task as it involves transferring data between systems that may be built on entirely different architectures, formats, or technologies. The integration team will be confronted with a choice to be made between various strategies. Choosing the optimal strategy for a particular integration challenge will be key to achieving both immediate and long-term success. These various strategies are referred to as integration patterns, each offering a particular solution that facilitates effective communication between different systems.
Integration patterns are a set of design practices, honed through the collective experience of countless developers and architects, which provide standardized, reusable solutions to recurring problems in the field of system integration. They can be seen as the blueprints for solving common integration challenges, thereby ensuring that developers do not have to reinvent the wheel every time they encounter a similar problem.
A good integration pattern not only aids in the communication between systems but also promotes the principles of loose coupling, high cohesion, and scalability. It further ensures that the integrity of data is maintained, system performance is optimal, and any potential risks or failures are handled appropriately.
These patterns cover a wide variety of scenarios, from the transformation of data formats to routing, message construction, and system orchestration. Some well-known patterns include Message Channel, Message Translator, Publish-Subscribe Channel, and many more, each catering to different aspects of system integration.
In this era of digital transformation, and with the proliferation of microservices and cloud-based solutions, the significance of integration patterns has grown exponentially. They have become a key consideration for organizations looking to enable seamless cooperation across an inventory of existing and planned systems. Understanding and implementing these patterns can significantly enhance the robustness, efficiency, and the maintainability of an organization’s IT assets over time.
Even though the landscape of technology keeps evolving, these integration patterns, time-tested and refined, remain as relevant and essential as ever. It’s not just about connecting systems; it’s about doing so in a manner that is scalable, resilient, and effective. Integration patterns deliver the very foundation upon which systems can evolve in a stable manner.
For the purpose of this discussion, we have grouped the various integration patterns into Data and API centric patterns and discuss each separately. We then examine the pros and cons of a new pattern, a new approach: Event-Driven Integration.
1.1 Data-Centric Integration
Data integration refers to the process of combining data from various sources into a unified view, allowing organizations to analyze and make informed decisions based on a comprehensive data set. There are several approaches to data integration, each with its own advantages and considerations. Here are some common data integration approaches:
· ETL (Extract, Transform, Load): ETL is a traditional approach widely used for data integration. It involves extracting data from multiple sources, transforming it to meet the desired structure or format, and loading it into a target system such as a data warehouse. ETL typically requires predefined schemas and is suitable for batch processing of large volumes of data.
· ELT (Extract, Load, Transform): ELT is a variant of ETL where data is first extracted and loaded into a target system, such as a data lake, without significant transformation. The transformation is then performed within the target system using tools like SQL or big data processing frameworks. ELT leverages the processing power of the target system and enables organizations to work with raw data before transforming it;
· Data Replication: In this approach, data is replicated from source systems to a central repository or data warehouse in near real-time or scheduled intervals. Replication ensures that the data is always up to date and readily available for analysis. It can be achieved through database replication techniques, log-based change data capture (CDC), or messaging systems;
· Data Mesh/Virtualization: Data virtualization allows organizations to access and query data from multiple sources without physically moving or integrating it. It provides a logical layer that abstracts the complexities of underlying data sources, making it easier to access and integrate data on the fly. Data virtualization reduces the need for data movement and can provide real-time access to distributed data sources;
· Data Fabric/Federations: Data federation involves accessing and querying data resident in its original source system on-demand, without physically consolidating or moving it. It creates a logical abstraction layer that allows organizations to treat distributed data sources as a single virtual database. Data federation is useful when data latency or privacy concerns make it impractical to centralize all data.
These approaches can be combined or customized based on the specific requirements of an organization’s data integration project. The choice of approach depends on factors such as data volume, latency requirements, system compatibility, data quality, and the desired level of integration complexity.
Data Driven integration is well suited to use cases such as AI/ML training where a large set of data is required to train the initial models. The data does not need to be kept up to date, or the training occurs relatively infrequently. Other scenarios such as analytics and reporting where the data does not need to be up to date or is analyzing large amount of historic data and is not time critical. The following outlines a few sample user cases where data integration approaches may be appropriate integration patterns.
· Healthcare: Integration of data from different sources like electronic medical records (EMRs), lab results, and health monitoring devices can help provide comprehensive patient care. For example, predictive analytics can be used to anticipate health issues and suggest preventive measures.
· Retail: In the retail sector, integrating data from sales, inventory, and customer behavior can help in better understanding demand and supply. Predictive analytics can be used to forecast sales, optimize inventory, and personalize marketing efforts to enhance customer experience.
· Supply Chain Management: By integrating data from various sources such as production, inventory, transportation, and suppliers, companies making data-driven decisions to improve efficiency and reduce costs.
· Banking and Finance: Integration of data from various systems (like customer data, transaction data, market data risk assessment, and personalized banking services. For example, machine learning algorithms can be used to analyze spending patterns and flag suspicious transactions.
· Manufacturing: Integration of data from production, sales, and supply chain can help optimize production planning, predict maintenance of machines (Predictive maintenance), and improve product quality.
What all of these use cases have in common is the use of large amounts of data for the purposes of reporting or building models of behavior.
1.1.1 Disadvantages of Data-Driven Integration
Data-Driven Integration can be highly effective in certain scenarios, but it does have potential disadvantages that need to be considered. Here are some of them:
· No support for real-time: Data Driven Integration typically operates in a batch mode or on a scheduled basis, and therefore it may not reflect the real-time status of data. This can be a problem for systems that need up-to-the-minute or real-time data integration.
· System Coupling: Data Driven Integration usually requires more tightly coupled systems. If one system changes its data format or schema, the other systems that are integrated based on this data may need to be modified. This can lead to increased costs and effort in maintenance and upgrades.
· Performance Impact: Data synchronization or transfer processes can be resource-intensive, especially for large volumes of data. This can impact the performance of the involved systems, particularly if data transfers are not carefully scheduled and managed.
· Data Inconsistencies: If not properly managed, Data Driven Integration can lead to data inconsistencies. For instance, if two systems are updated almost simultaneously, it can be difficult to ensure that both have the same, most up-to-date data.
· Complexity in Error Handling: Errors during data transfer or synchronization can be difficult to manage and rectify. They may require complex transaction handling and rollback procedures to ensure data integrity.
· Security Risks: Exposing data for integration may increase the potential risk of data breaches if proper security measures are not in place. The more systems that have access to the data, the higher the risk.
Overall, while Data-Driven Integration can be a valuable approach for integrating systems, these potential disadvantages need to be considered during the design and implementation stages of an integration project.
1.2 API Centric Integration
Integration approaches refer to the methods and techniques used to connect or combine different systems, applications, or components to work together seamlessly. There are several common integration approaches, each with its own advantages and considerations:
· Point-to-Point Integration: Point-to-point integration involves establishing direct connections between two systems or applications. In this approach, each system is connected individually to every other system it needs to communicate with. While it may be simple to set up initially, this approach becomes complex and difficult to manage as the number of systems increases.
· Enterprise Service Bus (ESB): An Enterprise Service Bus is a middleware infrastructure that provides a centralized, scalable, and flexible platform for integrating various systems. It allows systems to communicate through a common bus or messaging backbone. The ESB handles message routing, transformation, and mediation between systems, enabling seamless integration. ESBs often support multiple integration patterns, such as publish/subscribe and request/reply.
· Service-Oriented Architecture (SOA): Service-Oriented Architecture is an architectural approach that focuses on designing systems as a collection of services. Services are self-contained, modular units that expose well-defined interfaces and can be invoked by other systems or applications. SOA promotes loose coupling and reusability, making it easier to integrate systems both internally within an organization and externally with partners.
· Application Programming Interface (API) Integration: API integration involves leveraging APIs, which are sets of rules and protocols that allow different software applications to communicate with each other. APIs expose specific functionalities or data of an application, enabling other applications to access and interact with them. API integration is commonly used in web services and cloud-based integrations.
These are some of the common integration approaches used in modern systems and applications. Organizations choose the appropriate approach based on their specific requirements, complexity, scalability, and the technologies involved. It’s important to consider factors like security, performance, maintainability, and future extensibility when deciding on an integration approach. The following outlines some sample Application Integration examples.
· E-Commerce Platform and Logistics Provider: An e-commerce platform can integrate with a logistics provider via an API, allowing users to get updates on order status, shipping details, and tracking information. This integration can facilitate smoother business operations and improve customer experience.
· Customer Relationship Management (CRM) and Email Marketing: APIs can connect CRM systems like Salesforce with email marketing platforms such as Mailchimp. This integration enables companies to leverage customer data for more targeted and personalized email marketing campaigns.
· Financial Institutions and FinTech: Banks can integrate with FinTech solutions through APIs to offer services like up-to-date account balance checks, fund transfers, and payment services. This provides customers with a more comprehensive and seamless banking experience.
· Social Media and Business Websites: Businesses can integrate their websites with social media platforms using APIs. This allows for features like social login, display of social feeds on the website, and direct sharing of content to social media platforms.
· Healthcare Systems Integration: Healthcare providers can use APIs to integrate electronic health record (EHR) systems with other healthcare applications. This could allow for sharing of patient data, scheduling appointments, and e-prescription services.
· HR Systems and Payroll: APIs can connect HR systems to payroll and benefits providers, helping to streamline HR processes. This can reduce the administrative burden on HR teams and increase data accuracy.
All of these are traditional API integration use cases that are often human driven. We will look at the Event-Driven uses cases and the Event-Driven integration patterns below.
1.3 Disadvantages of API Driven Integration
API-driven integration involves using Application Programming Interfaces (APIs) to enable communication and data exchange between different software applications. While this approach has many advantages, including flexibility, scalability, and the ability to support real-time data exchange, it also has some potential disadvantages:
· API Complexity: APIs can vary greatly in their complexity. Some APIs are simple and straightforward to use, while others may be quite complex, requiring a deep understanding of the specific API and its associated software application.
· System Coupling: API Driven Integration requires tight coupling between systems. Changes at the API, Data format or schema requires changes to the system consuming the API.
· API Versioning: APIs are often updated or changed by their providers. These changes can break existing integrations if not managed carefully. API consumers must be aware of the versioning and update their systems accordingly to avoid disruptions.
· Rate Limiting: Many API providers impose limits on how many API calls can be made within a certain timeframe (often to prevent abuse or to manage load on their systems). This can limit the performance and scalability of systems that rely on these APIs for integration. This also forces the consumer of the API to implement complex logic to manage the rate or use other complex technologies to manage these rate issue.
· Dependence on External Systems: If you’re using third-party APIs, you are dependent on those systems and their reliability. If an API you depend on goes down, is deprecated, or changes its functionality, your system may be adversely affected.
· Security and Privacy Concerns: APIs can be potential targets for hackers. Also, data shared through APIs could potentially be exploited if not properly secured. Furthermore, if you’re dealing with sensitive data, API integration needs to be designed with privacy regulations and best practices in mind.
· Integration Complexity: It can be challenging to manage integrations with a large number of APIs, each of which may have different conventions, data formats, error handling mechanisms, and other characteristics.
Some of the complexities of API-Driven Integration can be alleviated with other technologies, such as ESB’s and API Gateways. But these products do introduce yet more technology, management and maintenance. So although they can at first appear to be a good solution, over time, they will often become a burden and will need to be centrally managed.
Despite these potential disadvantages, API-driven integration remains a powerful and often essential approach for enabling communication between different software systems. Careful planning, robust error handling, and thorough security practices can mitigate many of these potential issues.
Event-driven integration is an approach to system integration where different software components or systems/applications communicate and interact with each other through events. An event represents a state change within one system or application that is of interest to other systems or applications. An event could be an IoT style event (such as a new reading from a sensor), changes to a record in a database or application-level state changes such as a new order. This integration style allows systems or application to react and respond to events in a decoupled and asynchronous manner, enabling scalability, flexibility, and loose coupling between systems and application. Event Driven Integration consists of Event Producers (publishers) and Event Consumers (subscribers), there can be multiple subscribers to a given publisher.
Event Driven Integration typically consists of the following actors and facilities.
· Events: These are notifications that indicate a change in state or a significant occurrence within a system. An event could be anything from a user clicking a button, to a sensor reading exceeding a certain threshold, to a new record being added to a database.
· Event Producers/Publishers: These are the sources of events. They generate or produce events but don’t dictate what should be done in response to them.
· Event Consumers/Subscribers: These are the components or services that listen/subscribe to and react to events. They act based on the events they receive. Typically, there could be many Consumers/Subscribers to Event Rpoducer/Publisher the publisher of an event is generally unaware of the Consumers/Subscribers
· Event Channels or Brokers: In many event-driven systems, especially those that operate at scale, there’s a need for an intermediary that can efficiently route events from producers to consumers. These intermediaries, often called event brokers or event channels, can provide additional capabilities like event storage, filtering, and transformation.
Event-Driven Integration can apply to many domains and industries. The following are some sample uses cases where Event-Driven Integration patterns could be applied.
· Ground Based Air Defense System: Ground Based Air Defense System consist of multiple sensor platforms, Radar, Drones, Satellite Imagery events from these systems need to be filters, aggregate and detect situations of interest and distributed the events to the relevant fire control systems in a coordinated and real time manor. This will reduce the reduce the time from detection to action and also reduce the amount of irrelevant information that an operator of a fire control system may have to deal with.
· E-commerce platforms: In an e-commerce platform, actions like order placement, payment, and shipment are all significant events. When a customer places an order, it can trigger an event that initiates payment processing, inventory updates, and shipment processes. This approach allows for a more seamless and efficient integration of different business processes.
· Internet of Things (IoT) applications: IoT devices generate an enormous amount of data, and event-driven integration plays a crucial role in handling this data. For instance, a temperature sensor in an IoT network could generate an event whenever the temperature crosses a certain threshold. This event could trigger other actions, like adjusting a thermostat or sending a notification to a user.
· Healthcare systems: In healthcare, patient data is constantly updated and used across different systems. For example, if a patient’s lab results are updated, an event can be triggered that notifies the patient’s healthcare provider. Event-driven integration makes it easier to keep all involved parties updated and coordinate care.
· Banking and finance systems: Many banking transactions like money transfers, withdrawals, deposits, etc., can trigger events. For instance, when a customer makes a transaction exceeding a certain limit, an event could be triggered to inform the bank’s fraud detection system for additional verification.
· Supply chain and logistics: In supply chain systems, events like a product moving from one location to another or changes in inventory levels can trigger other processes. For example, when inventory levels drop below a certain point, an event could be triggered to order more products from suppliers.
· Real-time analytics: Many businesses use real-time analytics to gain insights and make quick decisions. Event-driven integration is crucial in this case as it allows data from different sources to be aggregated and analyzed as soon as it’s generated. For instance, a sudden increase in website traffic can trigger an event to alert the marketing and IT teams.
· Social media platforms: On social media platforms, user actions like liking a post, sharing content, or making a comment are events. These events can trigger notifications, updates in feeds, and other processes.
Event-Driven Integration patterns are perfectly suited to scenarios and use cases the exhibit the following behaviors.
· Real-time Responsiveness and the need for information to be up to date in real-time, for instance fraud detection in banking or alerting systems in healthcare.
· Event-Driven integration allows systems to be loosely coupled. Systems are not directly connected to one another, which enhances maintainability and reduces the risk associated with changes or failures in any one system. So, situations where the systems are evolving at their own pace and multiple systems need to be integrated is often a good fit for Event-Driven Integration.
2.1 Advantages of Event-Driven Integration
Event-driven integration offers several advantages in the context of software development and system integration. Here are some key advantages:
· Loose coupling: Event-driven integration promotes loose coupling between components or services within a system. Instead of direct and tightly coupled interactions, components communicate indirectly through events. This decoupling allows for greater flexibility, scalability, and maintainability of the system over time.
· Asynchronous communication: Events enable asynchronous communication between components. When an event occurs, it can be published and consumed by interested parties without requiring immediate or synchronous responses. This asynchronous nature enables components to operate independently and efficiently, leading to better performance and responsiveness.
· Scalability: Event-driven integration facilitates scalability by distributing workloads across multiple components or services. As events are published, they can be processed by multiple subscribers in parallel. This distributed processing capability enables systems to handle high volumes of events and efficiently scale horizontally as the demand increases.
· Extensibility: Event-driven architecture allows for easy extensibility. New components or services can be added to the system by subscribing to relevant events without modifying existing components. This modularity and extensibility make it easier to introduce new functionalities or integrate with external systems.
· Event-driven patterns: Event-driven integration enables the implementation of powerful design patterns such as publish/subscribe, event sourcing, and message queuing. These patterns provide enhanced reliability, fault tolerance, and event-driven analytics, which can be leveraged to build robust and resilient systems.
· Integration of heterogeneous systems: Event-driven integration is particularly well-suited for integrating heterogeneous systems or services developed on different platforms or using different technologies. By abstracting communication through events, systems can exchange data and trigger actions regardless of their underlying implementation details.
· Real-time processing: Event-driven architectures are well-suited for real-time processing scenarios. Events can be processed as they occur, allowing for immediate reactions or updates. This is especially useful in domains such as real-time analytics, IoT applications, and event-driven workflows.
Overall, event-driven integration provides a flexible, scalable, and loosely coupled approach to system integration, enabling efficient communication, extensibility, and integration of heterogeneous systems.
2.2 Fit for purpose!
Deciding which approach to use when building an integration strategy is very complex and several factors need to be considered. Also, an integration strategy may involve multiple integration strategies.
· Does the data need to be up to date?: Often, reporting systems do not require up-to-date information. The use of data for the purposes of training AI/ML models does not need 100% current information, for example. Therefore, in these scenarios, more traditional approaches to data integration work well;
· Existing synchronous API: Often, a given system will only expose a synchronous API and, therefore, API level integration is the only approach possible. Although Event-Driven integration may still be possible – see below;
· Low data change rates: When data is not changing rapidly, or the rate of integration is low, API driven integration is a valid and possible approach.
2.2.1 Combining Approaches
Event-Driven Integration can be combined with other integration approaches to implement any integration strategy:
· Data Integration and Event-Driven Integration: Data integration patterns such as ETL can be combined with Event-Driven Integration. For instance, an ETL approach may be used to load the initial data from one system to another and then an Event-Driven Integration approach could be used to keep the two sources synchronized;
· API Integration and Event-Drive Integration: Often applications may only provide an API to integrate. Event-Driven Integration patterns can still be used by using Event-Driven Integration patterns to ingest the inbound events and push these events into a system via an API. Care must be taken with this approach to handle load on the API as they will often be rate limited, but often this can be handled by buffering information before sending onto the API.
VANTIQ is a platform designed for event-driven integration, enabling organizations to develop real-time, event-driven applications and systems. It provides a set of tools and capabilities that facilitate the seamless integration of diverse data sources, services, and applications.
Here’s how VANTIQ supports event-driven integration:
· Event Sources: VANTIQ supports a wide variety of off-the-shelf event sources and a flexible mechanism of integrating additional event sources.
· Publisher/Subscriber Model: The VANTIQ platform and any application built within VANTIQ makes use of an internal pub/sub mechanism. Therefore, everything deployed within VANTIQ, whether a new use-case or an integration pattern, is event-driven in nature.
· Asynchronous Services: Services are a common approach to application development. VANTIQ services support both synchronous and asynchronous interfaces, allowing developers to build applications from a series of asynchronous services.
· Reliable Events: Both external event sources and internal event processing can be reliable and use an ‘At Least Once’ delivery model.
· Distributed Capabilities: Publisher and Subscribers can be distributed across multiple tenants, VANTIQ instances and geographies. Subscribers can be added dynamically and move location without affecting the publishers or other subscribers.
· Event Processing: Asynchronous Services process event using Visual and textual Event Handlers. Event Handler both Visual and textual are reliable and support ‘At Least Once’ processing. Each separate step within a Visual Event Handler will maintain a check point and if an event is redelivered the check pointing with skip steps that have already been processed.
· Distributed Event Broker: The VANTIQ Event Broker uses a distributed/mesh-based architecture verses the classic centralized approach that most event brokers/MOM’s use. This allows for great scalability and reliability as it does not rely on a complex centralized broker.
· Catalog: A centralized catalog of events and synchronous services allows the developer to browse the available events and services and publish or subscribe to events and asynchronous services registered in the catalog. The catalog is only used to register an event and to subscribe to events, it is not involved in the event distribution – this is handled by the distributed event broker. The catalog also provides security mechanisms to control who can see certain events and services that are registered.
· Scalability and Deployment: VANTIQ is designed to handle high-volume event streams and scale horizontally as per demand. It supports deployment options both on-premises and in the cloud, allowing you to choose the infrastructure that suits your needs.
· Low Code: Integration in VANTIQ makes use of various low code/visual modeling and development tools to greatly increase developer productivity and scalability.
Overall, VANTIQ’s event-driven integration capabilities empower organizations to build real-time, responsive integrations between existing and new applications by seamlessly integrating disparate data sources and systems. Vantiq allows the creation of Integration Applications/Services as well as new Business Applications and allow the new Business Application to integrate with the existing application in an event driven approach as well as the new Business Application generating new Events that could be integrated into new and existing applications. It enables integration architects to harness the power of event-driven architecture to drive agility, responsiveness, and innovation in their organization’s digital transformation efforts.
3.1 Event and Event Sources
Events are generally obtained as streaming data. Streaming data is data (aka events) that arrive continuously. There is no notion of an end to the stream as events will be delivered periodically until the application or the event producer is terminated. Events may be delivered at high rates, with thousands of events arriving each second, or low rates, with the interval between events measured in minutes, hours or days. If the arrival rate is high, it is advisable to use the most efficient integration technologies to obtain the events. If the rates are low, less efficient technologies are perfectly adequate.
There are a number of key integration technologies available for obtaining events:
· Messaging system integrations such as MQTT, AMQP, Kafka and JMS, Hyper Scaler Messaging systems such as Azure Event Hub, AWS Kinesis
· IoT Platforms
· VANTIQ REST/WS interface
· Custom integrations of proprietary APIs
· Vision Analytics and AI Platforms
All these sources are naturally event driver and streaming but many integrations and systems do not support events and streaming integrations. VANTIQ also provides mechanisms of consuming non-stream data/event sources. The most efficient approach to integrating these sources is to convert the non-streaming nature of them into streams of event at the periphery of the system.
· REST/API sources
· Databases – JDBC, Big Data Databases
· Files – CSV files
· Legacy Systems
These systems are not streaming sources and will not deliver events continuously and therefore mechanisms are built into VANTIQ to automatically smooth the delivery of bulk/burst events into more manageable streams. Back pressure can also be applied to streaming sources when the event rate exceeds the system ability to process these events, further increasing the reliability of the system.
3.2 Decentralized Event Broker and Catalog
VANTIQ’s Event Broker is designed to distribute events across multiple systems/applications and environments.
The Event Broker uses a publish/subscribe model. Any application, developed in any technology, may publish events to the broker. Any application, developed in any technology, may subscribe to any event published by the broker. In this way, business events produced by one application can be consumed by other applications to create an event-driven system.
Events are made available by registering them in the catalog. The event consists of a topic name and a schema for the event’s payload. These events are registered in the broker event catalog. An application wishing to publish the event registers themselves as a publisher. An application wishing to receive events published to the registered topic, register themselves as a subscriber. At runtime, when an event is published, it is delivered to all subscribers.
The Broker supports a distributed pub/sub model supporting publishing applications and subscribing applications in different geographic regions.
VANTIQ applications can automatically publish events to the Broker and subscribe to events from the Broker with a simple declaration making VANTIQ applications the easiest to combine into an event-driven system. Once an event stream is selected for inclusion in an application, the VANTIQ runtime is automatically provisioned to deliver PUBLISHED messages to their SUBSCRIBERS.
Since there is no centralized broker it is much easier to scale the system. Each publisher or subscriber can be scaled separately to handle their expected workload rather than having to scale a single broker to handle the combined workload.
3.3 None Streaming Sources
One of the challenges of Event-Driven Integration is the that not all applications and systems will support a streaming/event-driven interface. Although it would be possible to update the applications to add a streaming interface, this is not a practical solution. Each Publisher or Subscriber within the VANTIQ Event Broker is a separate event-driven application. Publishers and Subscribers can act as gateways between applications and systems that do not support a streaming interface and the pub/sub model used in Event-Driven Integration. A Subscriber could receive events and send these events via an API to an application or integrate directly with a database to insert these events into a database. A publisher could use polling, file uploads etc. to receive non streaming data from an external system and convert these into a stream of events that are published to the various Subscribers.
3.4 Reliability
VANTIQ supports an ‘At Least Once’ reliability model, any event marker as reliable will use a store and forward mechanisms for event delivery. This reliability mechanism works across event sources that also support ‘At Least Once’ delivery. For event sources that do not support ‘At Least Once’ delivery mechanisms the event can become reliable once it hits the VANTIQ platform. The internal pub/sub and the Event Broker distributed pub/sub mechanisms support reliability. Within the Visual Event handlers – event processors – reliability is supported through an in-memory check pointing mechanisms such that when events are redelivered to the Visual Event Handler the check pointing logic ascertains which tasks have been executed and start processing from the next step. The checkpointing state is replicated across the cluster to support high availability of in memory state.
3.5 Event Processing
Integration is not just about connecting system A with system B. As events are flowing through the system they need to be checked, enriched and transformed and in some cases filtered. VANTIQ Event Handlers and specifically the low-code Visual Events Handler allow developers to implement this business logic in a low code approach. Visual Event Handler can be used on both the publisher and subscriber side of any integration. Publishers could use the Visual Event Handler to perform checks and transformations etc. to verify any event data is valid and to transform the events into a standard format and perform functions such as property level conversions. They can also enrich the event data with additional information, for instance an AI platform that is detecting situations of interest in a camera stream does not know the location of the camera, this may be useful information for the subscribers so this additional meta data can be added to the events before being published. Subscribers may wish to only send certain events to some external system or databases so may perform filtering or thresholding to reduce the load on the external systems.
3.5.1 Event Processing and Meta Data
Certain events that are ingested may contain sensitive or incompatible data that other systems are unable to process. Frequently, data sets may be sufficiently large that moving this information around would cause a performance issue or swap available bandwidth. One such example would be satellite image data and the AI analytics performed on these images. The image itself does not immediately need to be published, but only the meta data and situational awareness data associated with the image. The image location and access to the raw image can be managed by the publisher and the publisher only ‘publishes’ the meta data about the image. Sensitive information could be obfuscated before publishing the event data, this could be for security or privacy purposes such as GDPR.
3.6 Multiple Brokers/Security Domains
It is possible for a published or subscriber to be connected to multiple brokers. This allows events to flow between different security domains or environments. The individual publishers could perform obfuscation etc. on the events depending on the domain it is published to.
In conclusion, VANTIQ’s event-driven integration capabilities provide organizations with a powerful platform for developing real-time, event-driven applications and systems. By seamlessly integrating diverse data sources, services, and applications, VANTIQ enables organizations to build responsive and agile integrations that drive innovation and digital transformation.
VANTIQ offers a range of features that support event-driven integration. The platform supports a wide variety of event sources, both off-the-shelf and customizable, allowing organizations to efficiently obtain events from different systems and technologies. The publisher/subscriber model used by VANTIQ ensures that all applications and use cases deployed within the platform are inherently event-driven, promoting a seamless flow of events throughout the system.
With VANTIQ, developers can leverage asynchronous services and reliable event delivery mechanisms, such as the ‘At Least Once’ model, to ensure the robustness and scalability of their integrations. The distributed capabilities of the platform allow for flexible deployment across multiple tenants, instances, and geographic regions, enabling organizations to scale their event-driven systems as per their needs.
The VANTIQ Event Broker, built on a decentralized and distributed mesh architecture, serves as the backbone of event distribution, supporting a publish/subscribe model and facilitating the seamless flow of events between applications. The centralized catalog provides a convenient way to browse and manage available events and services, ensuring secure access and control.
VANTIQ’s event processing capabilities, including visual and textual event handlers, enable developers to implement business logic, data validation, transformation, enrichment, and filtering within a low-code environment. This empowers organizations to effectively manage and process incoming events to meet their specific integration requirements.
Moreover, VANTIQ recognizes the challenge of integrating non-streaming sources into an event-driven system. The platform offers mechanisms to convert non-streaming data sources into manageable event streams, allowing seamless integration with legacy systems, REST/API sources, databases, and files.
Overall, VANTIQ’s event-driven integration capabilities provide organizations with a comprehensive solution for building real-time, responsive, and scalable integrations. By leveraging the power of event-driven architecture, organizations can drive agility, innovation, and success in their digital transformation endeavors.