Vantiq Catalog Reference Guide

Overview

The Vantiq Catalog connects Namespaces across a distributed Application to quickly and easily execute operations and copy resources between Namespaces. It has three sub-catalogs that focus on different features: the Event Catalog, the Service Catalog, and the Assembly Catalog.

The Event Catalog offers a sophisticated event ledger for recording events as they are published and/or delivered to subscribers. The event ledger takes advantage of the powerful transformation and augmentation facilities available in the Vantiq system to record precisely the desired information in the ledger.

The Service Catalog allows a Namespace to publish a set of high-level Procedures and Service Events to the Catalog. Subscribers may treat the Service as an interface, calling Catalog Service Procedures as if they exist in their own Namespace. The Service Catalog provides logical and functional connectivity between Namespaces operating on analogous Applications.

The Assembly Catalog lets users install resources from a central location, and keep those resources up-to-date with any fixes or changes. Subscribers have a full, read-only copy of all the resources published by the publisher namespace.

The Vantiq Catalog provides a full set of security features for fine-grained control of publishers and subscribers.

For a more structured walk-through of building an application with the Vantiq Catalog, please review the Vantiq Catalog Tutorial.

Terminology

There are some core concepts in the Catalog that need to be defined to avoid confusion.

  • Namespace – a Namespace is the unit of isolation within Vantiq. Each Catalog resides in a single Namespace that is isolated from all publishers and subscribers. Namespaces connected to the Catalog can act as publishers and subscribers. Applications that wish to publish and/or subscribe to Event Types or Services authenticate into a Namespace that is connected to a Catalog.
  • Event Catalog – The list of Event Types that are managed by the Catalog. Event Types are identified by a name which is also known as a Topic in the Catalog.
  • Service Catalog – The list of Services that are managed by the Catalog. Services are identified by their name, which is also the prefix for the Procedures used.
  • Catalog Namespace – A Namespace that hosts a Catalog. This stores information on every entry in the Catalog, such as Event Types or Services.
  • Event Type – An entry in the Event Catalog that defines an identifier for the Event Type, a schema which the events should conform to, and metadata for identifying the Event Type in the Catalog.
  • Service – An entry in the Service Catalog that holds information on the Service’s publisher and the Procedures that are part of the Catalog.
  • Publisher – A Namespace connected to a Catalog that produces events or holds the Procedures for a Service.
  • Subscriber – A Namespace that has requested receipt of all events of an Event Type or use of a Service in the Catalog.
  • User – a user authorized to access the Catalog. A user authenticates with their provisioned access credentials or with an access token supplied to the user by a user with sufficient privileges to create an access token. An application that publishes or subscribes to Catalog events must be associated with a User either through an access token or by having the user using the application supply their credentials to authenticate with the Catalog.
  • Manager Namespace – A Namespace with sufficient authorizations in the Catalog Namespace to administer the Event Catalog.
  • Member Namespace – A user with sufficient authorizations in the Catalog Namespace to use the Catalog to subscribe, publish or create new Event Types and Services.

Configuring the Vantiq Catalog

Provisioning Vantiq Catalog Users

Catalog users are created and managed using the Vantiq User Editor

Provisioning Catalogs

The Catalog is a central location where Event Types are defined and Services published. The Catalog is hosted in the Catalog Namespace, but can be viewed by any Namespace connected to the Catalog Namespace. An Organization can host many Catalogs, but each Namespace can only host at most one Catalog. Individual Namespaces can be connected to many Catalogs.

Vantiq recommends starting with a single Catalog so that all Catalog users will have access to a complete, global view of the available Catalog entries within the organization. Once additional experience is gained, it may make sense to define more specialized Catalogs.

At least one Catalog must be created. The Catalog is created by first creating a Namespace to hold the Catalog. The Namespace name will serve as the Catalog name. A Catalog is created by using the Administer>Advanced>Catalog menu item. To create a Catalog in this Namespace, click the Create Catalog button.

Enabling Edge Connections

Catalogs can accept connections from edge nodes. This is not enabled when created through the UI, but the setting can be changed at any time by navigating to Administer>Advanced>Catalog and toggling the “Allow Edge” box for the catalog namespace. Disabling it will forcibly disconnect all edge members.

Note that this may force the catalog namespace to perform work on the connections. Since edge nodes cannot directly talk to each other, the catalog namespace will act as a bridge for any Services and Event Types that have edge members as both publishers and subscribers.

Connecting To A Catalog

A Namespace must be connected to the Catalog to be able to search the entries as well as to publish or subscribe to events and Services.

When a Namespace needs to be connected to an existing Catalog in a different Namespace, use the Show>Catalogs menu item to display the Catalogs pane. Click the New button to create a new Catalog connection.

To connect to the Catalog Namespace, you will need an Access Token that was created in the Catalog Namespace, which must be generated by the Catalog Namespace administrator. The Catalog administrator must deliver this token to the user desiring to gain access to the Catalog. This token will be used to make the initial connection between the Broker Member Namespace and Catalog Namespace. During this initial connection, new Access Tokens and Nodes are generated in both Namespaces in order to create a permanent connection that can be uniquely identified. The token used
to make the initial connection between Namespaces is not reused, so the Catalog Namespace administrator should make those tokens relatively short lived.

Connecting From an Edge Installation

If you are connecting from an edge installation, you must check the “Connect using VQS” box if using the UI or set the useVQS value to true when connecting to the catalog. Not all catalogs will allow edge connections. If you need to connect from an edge installation and the catalog does not allow it, you will need to contact the catalog owner and ask them to enable it.

Repairing Catalog Connections

There are some situations in which tokens or nodes used by the catalog can break or disappear. You can repair the connections using this operation.

This can be done through the UI as well. When selecting Show>Catalogs, you can right-click on a catalog and select Repair. This will reconnect you to the catalog and any other members as necessary, and will also update any services or event types the namespace is using. Both the URI and the token are optional, but if neither is provided then it requires the catalog connection to be working, and if only the token is provided then the catalog node must still exist and have the correct URI.

Public Catalogs

In some cases it may be more convenient to make a catalog available to an entire Organization or installation, instead of providing credentials to each namespace individually. In these cases, the Catalog administrator may provide a longer-lived access token to an Organization administrator. The Org admin can then, in the Organization namespace, go to Administer>Advanced>Catalog, click New, and fill in the Catalog token and url.

createPublicCatalog

Once this is done, an administrator in any of the Org’s namespaces can connect to the catalog by navigating to Show>Catalogs, clicking New, selecting “Public Catalog”, and picking the catalog.

connectPublicCatalog

Viewing The Catalog

To browse the contents of a Catalog, use the Show>Catalogs menu item, then click the name of the catalog in the list. Here’s an example view of what the Catalog looks like in the IDE:

eventCatalog

The screenshot illustrates a collection of entries in the Event Catalog for Acme Industries. Note the name in the title bar of the Catalog pane is “AcmeIndustries”. This is because the Catalog Namespace for this
Catalog is the AcmeIndustries Namespace.

Searching The Catalog

The example Catalog shown above uses a hierarchical naming scheme where events are organized by resource (e.g. employee, machine, and sensor). Hierarchical names make the Catalog easier to search along the dimensions represented by the hierarchy. This is just an example convention. Any hierarchical naming convention can be used.

Searches are specified using the search bar at the top of the Catalog. The search term is checked against the entry name, description, schema type name, and the tags for any partial matches.

Catalog Actions

Entries in the Catalog have the following Actions available.

View Ledger – looks like a closed book, and runs a query for all entries in the ledger. The results are displayed in a newly opened pane.

Subscribe – looks like a play button in a circle, subscribes the current Namespace to the entry.

  • Subscribing to an Event Type through the Catalog adds a topic to the current Namespace, which will receive all events of that Event Type. To view a live stream of the events, click on the topic in the project resource graph and click the play button in the topic pane’s title bar.
  • Subscribing to a Service adds Procedures to the current Namespace. To view information on the Procedures, select Add > Services in the top of the Development bar.

Unsubscribe – looks like a pause button, terminates the current subscription to the entry. The unsubscribe button will only show up under actions if there is an active subscription to the entry in the current Namespace.

Delete – removes the entry from the Catalog and removes all publishers and subscribers. * For Event Types they are no longer visible across the Catalog environment. The events may still be generated in the publisher Namespace, but no applications that consume the events through the AEB will see the event. Only admins in the Catalog Namespace can remove Event Types from the Catalog. Removing Event Types may impact running production apps and so permission to remove Event Types is restricted from every other Namespace. * For Services only the publisher or the Catalog Namespace admin can remove the entry, and doing so will remove the subscribed Procedures from each subscriber.

<entry name> – clicking the entry name navigates to the detail pane for the Event Type or Service.

Graph – Visualize publishers and subscribers of entries in a graph.
– If this action is performed in a Catalog member Namespace, only Event Types used by this Namespace are displayed inthe graph.
– If this action is performed in the Catalog Namespace, selected entries are displayed in the graph. All publishers and subscribers of each entry are also displayed in graph. If no entry is selected, users are given the option to graph all entries within the Catalog. For example, after completing the Vantiq Catalog Tutorial, the Event Type graph looks like the following:
eventGraph

Setting Permissions

Permissions can be set for each operation available in a catalog. If a member tries to perform an operation it does not have permission to do, the system will throw an error. The default permissions can be set in the host’s Catalog resource, and per-member permissions can be set in the relevant Member resource. The Catalog resource must have a value for each operation, but the Member resource only requires a value where it will override the Catalog’s default permissions.

In the Vantiq IDE, this is managed through the Manage Catalogs pane in the Catalog Namespace, accessed through Administer>Advanced>Catalogs. Select Edit Permissions for the catalog namespace to change the defaults or for a member namespace to change its specific permissions.

Manage Catalogs Pane

Format

The permissions are stored as an Object in the catalogPermissions field of Members and Catalogs. The value true gives permission for the operation, false denies permission. The format and default values are below.

{
    "service": {
        "publish": true,
        "subscribe": true,
        "createEntry": true,
        "removeEntry": true
    },
    "event": {
        "publish": true,
        "subscribe": true,
        "createEntry": true,
        "removeEntry": false
    },
    "assembly": {
        "publish": true,
        "install": true,
        "createEntry": true,
        "removeEntry": true
    }
}

Types in the Catalog

Both Event Types and Services may have types associated with them. If an entry uses a type, that type must exist in the catalog namespace as well as any publishing and subscribing namespaces. When such a type does not exist in the relevant namespace, it is created there with its role set to “schema”.

Types are created in the catalog namespace when:

  • An Event Type is created or updated.
  • A Service is published and has a typed parameter, return type, or event type using a user-defined type.
  • A published Service is updated and has a typed parameter, return type, or event type using a user-defined type.

Types are created in a member namespace when:

  • It publishes or subscribes to an Event Type.
  • It subscribes to a Service that has a typed parameter, return type, or event type using a user-defined type.
  • It updates a subscribed Service that has a typed parameter, return type, or event type using a user-defined type.

Type Compatibility

When a catalog action would create a new type, it may conflict with an existing type in the namespace. In this case the types are compared, and if they are compatible the existing type has its fields updated. Types are compatible if:

  • The new type has all the fields of the existing type, and the sub-properties of each field match.
  • Any fields that are in the new type but not the existing type are not required.

Put simply, types are compatible if the only changes are new, non-required fields.

Import and Export of Catalog Connections

Catalogs are a Vantiq resource and Catalog connections may be exported and imported to move them between Namespaces. When importing projects or Namespaces that include Catalog connections, those connections must be reestablished by providing a current connection Access Token and URL. Use the Show>Catalogs menu item to reestablish Catalog connections.

Event Catalog

Using the Event Catalog

Events

The Event Catalog manages a collection of Event Types, which describe what events will look like. For those familiar with pub/sub messaging systems, Event Types can be thought of as similar to topics to which messages are published and subscriptions are requested.

Event Types are created by users with sufficient permissions to create Event Types. When an Event Type is first created, there are no publishers or subscribers.

Publishers are registered to define the producers of instances of the Event Type.

Subscribers are registered to indicate their desire to receive instances of the Event Type published by the publishers. A subscriber that subscribes before any publisher is registered will receive no event instances since there is no publisher producing such events. Once a publisher is registered and starts publishing event instances, existing subscribers will receive the published event instances.

The connections for Event Types look like this:

Connections Between Event Publishers and their Subscribers

Adding Event Types

Create a new Event Type by clicking the plus icon in the title bar. A new pane opens in which you type the name, description and schema for the new event.

The name of an Event Type must begin with a / character. The names use a hierarchical naming convention with each level in the hierarchy delimited by a / (in a fashion similar to Linux filenames). This is necessary because Event Type names must be valid topic names.

When a Namespace subscribes to an Event Type in the Catalog all of the events will be delivered to the subscriber on a topic in the subscriber Namespace, and unless overridden, the topic in the subscriber Namespace will be the same as the Event Type name.

Reliable Events

When adding an Event Type to the Catalog, check the Is Reliable? box to indicate that publishers of the event must be reliable and that the Catalog must retry forwarding of events from a reliable publisher until successfully delivered to all subscribers. Note that in some failure scenarios this means a particular subscriber might see the same event more than once.

Registering Event Publishers

In the Event Catalog find the existing Event Type or create a new one if the desired Event Type does not exist. Click on the name of the Event Type to open the Event Type detail pane, resulting in a display similar to the following:

eventDetail

Click on the “Click to View” link next to “Publishers” to open the list of known publishers and a form to register as a publisher:

publisherList

The top half of the popup registers the local topic or source that represents the stream of events to bind to the Event Type. In the example there is an existing publisher, FactorySensors, of /machine/offline. The local events that are bound to /machine/offline are produced by the topic: /sensors/offline.

Note that if you choose an existing persistent topic to be a publisher of an Event Type, it must have a message type that matches the message type expected by the Catalog. If the message types are not compatible, the Catalog will reject the registration of the publisher.

Instances of Event Types in the Catalog are generally published by Vantiq sources or by applications using the Vantiq SDKs to publish event instances to the Advanced Event Broker. Use of the Vantiq SDKs is detailed in a separate API reference guide.

Removing Event Publishers

To remove the FactorySensors Namespace as a publisher of the Event Type, click the remove button in the right-most column. Only administrators of the Catalog Namespace and of the FactorySensors Namespace can remove FactorySensors as a publisher of the Event Type. Other connected Namespaces can see that FactorySensors is a publisher, but do not have the authority to remove the publisher.

Registering Subscribers

As described in the Catalog Actions Section, it is possible to subscribe to an Event Type in the Catalog using the play button in the Actions column of the Event Catalog. Doing so will use the Event Type name as the local topic on which published events will be delivered to the subscribed Namespace.

If the default topic will not work, it is possible to subscribe to an Event Type and specify a different local topic on which published events will be delivered. To do so, use the “Click to View” link next to “Subscribers” in the Event Type detail pane. The subscribers popup should look like this:

subscriberList

To register as a subscriber of the Event Type and use a different topic name, simply enter the topic in the input field and click the “Subscribe” button. Events will immediately start being delivered to the Namespace on the specified topic as soon as they are published.

Note that if you choose an existing persistent topic to be a subscriber of an Event Type, it must have a message type that matches the message type expected by the Catalog. If the message types are not compatible, the Catalog will reject the registration of the subscriber.

Removing Subscribers From Event Types

In the screenshot above, note that there is a single subscriber listed. The Namespace OperationalManagement has subscribed to the /machine/offline events, and has requested those events are delivered locally on the topic /m/offline. From this subscribers list, it’s possible to unsubscribe to events as well using the remove button in the right-most column. Similar to publishing, only the Catalog Namespace and the Subscribing Namespace can remove a subscriber.

Augmenting Brokered Events

App Builder Integrations

Many of the features of an Event Catalog are enabled through the App Builder. These integrations take a few different forms:

  • Subscribe to Event Types directly in app Event Stream tasks
  • Publish the output of an app task to an Event Type
  • Record the output of an app task in the Event Ledger

All of these integrations are demonstrated in the Catalog Tutorial.

Publishing Events

Instances of Event Types in the Catalog are generally published by Vantiq sources or by applications using the Vantiq SDKs to publish event instances to the Catalog. Use of the Vantiq SDKs is detailed in a separate API reference guide.

Receiving Events

Instances of Event Types in the Catalog are generally subscribed to and delivered to applications built using the Vantiq SDKs. Use of the Vantiq SDKs is detailed in a separate API reference guide.

Source Integrations

In addition to supporting applications that produce event instances through the Vantiq SDKs, the Catalog contains pre-integrated support for popular platforms that produce event streams. These platforms are represented in Vantiq as a Source. To avoid putting extra load on that external system, it’s a good idea to create only one source to receive the inbound stream of events from the external system. The source can then publish thoseevents to the Event Catalog so they can be consumed by any number of subscribers.

Once you’ve created a source, there are a few easy steps you can take to create an Event Type in the Catalog and begin publishing source events directly to that Event Type. The first step is to test the source and receive a sample event, which can be inspected to determine the schema of the inbound events. From the source pane, click the Test Data Receipt button.

clickSubscribeSource

Clicking the Test Data Receipt button opens a live subscription pane that shows the events streaming in from the source in real time. Once an event arrives, you can click on that event and then use the Create Data Type button to generate a schema type using the format of the received event as a model.

clickCreateType

This should open a popup to confirm the type name and role, and after clicking OK the schema type is created.

Once the schema type has been created, go back to the source pane, scroll to the bottom and click Publish To Event Catalog, which opens the following popup:

publishFromSourcePopup

It’s possible to publish to an existing Event Type chosen from the drop list, or to create a new Event Type and register as a publisher using the Create New Event button, which opens a new dialog box to complete the Event Type definition like this:

createNewEventTypeFromSource

After that, all events produced by the external system will be published to the Event Catalog.

Service Catalog

Using the Service Catalog

Services

The Service Catalog handles Services, which are collections of Procedures that the Service Catalog allows to be run from any subscriber.

Services do not exist in the Catalog until published by a Catalog member. Once published, no other Namespace can publish a Service with the same name to the same Catalog until the publisher removes the Service.

Subscribers register for use of a Service once it has been published. Local Procedures are generated in the subscriber Namespace that will call the Procedures of the same name in the publisher.

Note that the Service Procedures are run in the publishing Namespace, so they cannot directly affect anything in the subscriber’s Namespace. Any development that occurs after publishing the Service will affect subscribers’ use of the Service. If significant changes are expected to occur while a Service is published, it may be useful to note that in the Service’s description.

The connections for Services look like this:

Connections Between Service Publishers and their Subscribers

Subscriber Permissions

Subscribers will only have user-level permissions in the publisher Namespace. If any Procedures in a published Service require higher permissions than the default user-level permissions, you will need to define
explicit profiles on the Procedure for the Service to function properly.

Creating and Editing Service Procedures

Use the Add>Service menu item to create and edit Services. Users may create and view Service Procedure Signatures through the Service pane.

The Interface tab of the Service pane shows all of the Procedure signatures contained by the Service. Users may add, edit, and remove Procedures from this pane. When the Service is saved, the changes made in the Service Pane must be applied to the Procedures in the Implement tab, otherwise the Service will be in error.

Interface tab

Procedures in the Interface are added to the Catalog when the Service is published. Private Procedures are never published to the Catalog. When creating or editing a Procedure, the user may define the Procedure name and parameters. Procedures in the Interface may also specify the Procedure description and return type. The return type may be set to either a Built-in Type (ex: String, Boolean, Integer…) or to a Schema Type defined in the Namespace.

By clicking on the pencil icon next to a parameter, you may define a parameter description as well as set the type of the parameter.

Procedures tab

Publishing Services

All local Services are publishable. To publish a service through the IDE, go to Add > Services to open the Services pane which looks as below.

ServicesPane

Find the desired Service and click on it to open its Service pane. Navigate to the Interface tab. Here is where you can see and set the interface for procedures. You can give a description for the Procedure, add a return type, and fill in some details for the parameters. See above for more details.

Click the up arrow in the Publish column for the Catalog you want to publish to. This will attempt to publish the Service to that Catalog and, if successful, the Publish symbol will turn into an x for that Catalog.

Service pane

Services may also be published directly from the Catalog Pane. Clicking on the “+ New” button will give the user the option to Publish a new Service to the Catalog.