Remote Source Integration

PostedAugust 25, 2023

UpdatedNovember 13, 2023

ByVantiq

0 out of 5 stars

5 Stars		0%
4 Stars		0%
3 Stars		0%
2 Stars		0%
1 Stars		0%

A remote source represents a REST service. A remote source may supply data or be a recipient of requests produced by Vantiq or both supply data and be a recipient of notifications.

A remote source implements one or more of the following capabilities:

Stream data from the REST service. A request is issued to the REST service periodically with the results producing an event which will be delivered to any subscribing rules or clients (thus simulating a stream).
Query the REST service for data. The REST service receives a standard REST query as documented in the RST API Reference Guide and returns a set of objects in one of the supported content types.
Send a request to the REST service.

Defining a Remote Source

A remote source is usually defined in the Vantiq IDE by using the Add button to select Source…. A remote source may also be defined by creating a source resource that represents the definition of the remote source and submitting the definition to the Vantiq server for registration. The source object is submitted using a POST request.

An example:

POST https://dev.vantiq.com/api/v1/resources/sources
{ 
    "name": "RESTServiceName",
    "type": "REMOTE",
    "config": {
        "uri": "http://sub.domain.vantiq:9900/aDemo/add/demoData",
        "username": "vantiq",
        "password": "vantiq",
    }
}

Creates a REST source that can be queried to obtain data from the REST service or notified to send data to the REST service. Notifications are issued via the PUBLISH action in a rule. The above example does not define a stream for automatically querying the REST service periodically for data.

Once registered the remote source will immediately begin reading and processing messages if a stream is configured and will also immediately be available for query and notify operations.

The relevant source properties are as follows:

name The name given the remote source by the user. The name must be unique among all sources.
type The string REMOTE to indicate this is a REST source.
config A JSON object containing additional REMOTE configuration parameters:
- uri The endpoint for the REST service. This is a String containing the URI supported by the REST service. The form of this URI is important when considering how it may be resolved against the path of any requests. See URI.resolve for more complete details. For example, during resolution any trailing path element (such as demoData in the above example) will be removed. To prevent this you would need to use the URI "http://sub.domain.vantiq:9900/aDemo/add/demoData/" (note the terminating slash (/)).
- username If basic auth is supported by the REST service, the username with which the REST service will be accessed.
- password If basic auth is supported by the REST service, the password with which the REST service is accessed.
- accessToken If the user is authenticated using an accessToken rather than basic auth, the accessToken to be presented to the service. See also the Fetching Access Tokens on Demand section.
- realm The authentication realm to use if an accessToken is specified. Defaults to Bearer.
- requestDefaults A request document which provides default values to use when issuing a request to the source (including a polling request). See request document schema for details on the expected contents.
- pollingInterval The interval at which the REST service is polled, in seconds. If a value of 0 is provided then no polling occurs.
- query A request document which describes the query to perform when polling (see request document schema for details on the expected contents). If a query isn’t provided then the source defaults to performing a GET operation on the base URI with no authentication and assumes content type JSON.
- clientOptions The JSON object form of Vert.x HTTP Client Options used to configure the client when connecting to the REST service.

Fetching Access Tokens on Demand

For some sources, the accessToken outlined above can be part of the configuration. This works well when that access token is long-lived. However, there are times when that token has a short lifetime – a few minutes to a matter of hours. Rather than reconfigure the source constantly, you can configure the source so that it can fetch a new token on demand. To do so, you will need to have another REMOTE source (aka the OAuth source) defined that represents an OAuth (Open Authentication) authorization server.

The configuration properties described below instruct the Vantiq system to perform the following actions when a request is made on a REMOTE source so configured.

Check to see that a valid access token for the source is known. Valid here means that the access token as been obtained, and that it has not expired.
If not, based on the configuration described below, make a request to the OAuth authentication server to obtain a new access token.
When the access token is obtained, associate it with the source, and use that access token to interact with the REMOTE source.

The properties defined below can be part of the REMOTE source’s configuration. If it is more convenient (say, because there are a number of sources that may use the same property values to obtain an access token), these configuration properties may be provided as part of the configuration of the source used to obtain the access token. Any of the configuration properties described EXCEPT for the oauthSourceName property can be included in the OAuth source’s configuration. Since this property tells the remote source which other source to employ to fetch the access token, this must be provided as part of the REMOTE source’s configuration. If one of these properties is found in both the REMOTE source’s configuration and the OAuth source’s configuration, the value from the REMOTE source will be used.

The following outlines the configuration properties required to dyamically obtain the access token.
These properties are under the config property outlined in the section above.

config The following properties are in addition to those defined above.
- oauthSourceName The name of the source to use to fetch the access token
- oauthGrantType The type of request flow to make to get the token. Choices are client_credentials or refresh_token.
- oauthClientId The client_id supplied to the user (if any)
- oauthClientSecret The client_secret supplied to the user (if any)
- oauthRefreshToken The refresh token supplied to the user. This is used when the oauthGrantType is refresh_token. In this case, the oauthClientId and oauthClientSecret may be required as well.
- oauthScope The scope defined for the token. This is usually not required
- oauthAudience The audience for the token. While non-standard, this is sometimes used by authorization servers.
- oauthUseBasicAuth Tells Vantiq to package the client_id & client_secret into a BasicAuth credential, and use that to access the authorization server.
- oauthContentType This allows the user to override the content type used in the authorization server request. The default value is application/x-www-form-urlencoded; this should be overridden only when required.

The oauthSourceName indicates that another source, representing the authorization server, is to be used to get an access token. This source needs to exist and be active at the time a request is made.

OAuth defines a number of grant type flows, two of which are appropriate for our case. The client_credentials flow, indicated by setting oauthGrantType to client_credentials, uses the oauthClientId and oauthClientSecret (often in conjunction with the oauthScope and/or oauthAudience) to request an access token from the authorization server.

The refresh token flow, indicated by setting oauthGrantType to refresh_token, uses the oauthRefreshToken in association with oauthClientId and oauthClientSecret (and often in conjunction with the oauthScope and/or oauthAudience) to request an access token from the authorization server. The oauthRefreshToken is usually a long-lived token that can be used, with the other properties, to obtain an access token.

The oauthUseBasicAuth and oauthContentType properties do not affect the content of the request; rather, they affect the form the requests take. Some authorization servers will want the oauthClientId and oauthClientSecret packaged in a Basic Auth and placed in the Authorization HTTP header. If your server requires that, set the oauthUseBasicAuth property to true.

The oauthContentType property allows you to use application/json instead of the default application/x-www-form-urlencoded. This is rarely required.

Note that oauthClientSecret and oauthRefreshToken properties should be considered sensitive data and, thus, should probably be stored as Vantiq Secrets. This should be done by referencing the secret in the configuration.

Making a Request to a Remote Source

When making a request to a remote source (via polling, SELECT, or PUBLISH) you provide a JSON document which fully describes the request. This is known as a request document and has the following properties:

method The HTTP method to use when issuing the query. The default is GET for query and POST for publish.
path The path to use when issuing the query. If none is provided then the query is performed using just the base URI. The path provided will be resolved against the source’s base URI. This means that if it is a relative path it may be appended to that URI whereas if it is an absolute path it may replace portions of the URI (depending on whether or not the base URI includes a path segment).
query An object containing name/value pairs which are added to the request as URL query parameters. Supports the @secrets notation.
fragment The fragment property, if defined, contains a URI fragment to be added to the URI for the remote source.
headers An object containing a set of name/value pairs which will be added to the request as HTTP headers. Supports the @secrets notation.
body or parts The content to send. Either body or parts is permitted, not both.
- body An object containing the body content to provide with the request (if any).
- parts A list/array of objects where each item in the list (part description) consists of the following properties. This is used to send a multipart/form-data message.
  - content or ref The content to be included in this part. Either content OR ref must be provided, but not both.
    - content The content to be included in this part.
    - ref A resource reference to a document, image, or video.
  - filename [Optional] The name to be sent as the filename parameter. If absent, filename will be set to the name of the document/image/video referenced. If no such name is available, the filename will not be set.
  - name The name of the part. If content is provided, this is required. If ref is provided, this will default to the name of the referenced entity.
  - contentType The MIME type for the part. If content is provided, this will default to text/plain. If ref is provided, this will default to the fileType of the referenced entity.
contentType The MIME content type used to process both the body and, if the responseType property is not defined, the response if it does not contain a content type header. Default is application/json if body is provided, multipart/form-data if parts is provided.
acceptType The MIME content type sent as the ACCEPT header in the HTTP request. Either acceptType or responseType may be specified, but not both.
responseType The MIME content type used to process the response. The property defaults to the content type supplied by the remote source in its response, so most of the time it will not need to be set. However, if that type is not one of the ones recognized by Vantiq, then it may be necessary to provide an explicit value. This value is also sent as the ACCEPT header in the HTTP request otherwise the value of the contentType property is sent, although not all remote sources honor this header. The supported response type values are:
- */json – parse the content as JSON.
- */xml – parse the content as XML. Given precedence this will match text/xml before it gets treated as plain text.
- text/* – convert the content to a string, but otherwise leave unprocessed.
requestTimeout The amount of time the request has to receive any data. If no data is received in the specified amount of time, a timeout occurs and the request is closed. The value must be specified as a VAIL duration, not exceeding the request execution time which defaults to 2 minutes. If this limit is exceeded, the request times out.

As noted above, when the path of a REST request is used, it is resolved against the source’s base URI. If the source URI contains only a schema, host, and port (such as https://myRestService) then this distinction won’t matter. However, when the base URI includes a path, it becomes very important. For example, assume we have a base server URI of http://some.server.com:8777/api/v1/. Resolving the path info/getStatus will result in a final URI of http://some.server.com:8777/api/v1/info/getStatus. However, resolving the path /info/getStatus results in http://some.server.com:8777/info/getStatus. As you can see, when resolved the relative path appends to the existing path, while the absolute path replaces it. For a more detailed description of the resolution process see the Java URI documentation.

If your query or header values contain sensitive data, you can store them as Vantiq Secrets and reference them in your code using the @secrets syntax.

The following REST requests are used to manage remote sources:

Create a REMOTE Source

POST https://dev.vantiq.com/api/v1/resources/sources
{ 
    "name": "RESTServiceName",
    "type": "REMOTE",
    "config": {
        "uri": "http://sub.domain.vantiq:9900/aDemo/add/demoData",
        "username": "vantiq",
        "password": "vantiq",
         "requestDefaults": {
            "requestTimeout": "30 seconds"
         }
    }
}

Creates a REST source with the remote service responding at the URI specified. Alternatively, to use a secret password named “MySecret”, change the password property to a reference and specify ‘secret’ as the passwordType like this:

POST https://dev.vantiq.com/api/v1/resources/sources
{ 
    "name": "RESTServiceName",
    "type": "REMOTE",
    "config": {
        "uri": "http://sub.domain.vantiq:9900/aDemo/add/demoData",
        "username": "vantiq",
        "password": "/system.secrets/MySecret",
        "passwordType": "secret"
    }
}

It’s also possible to use a secret to reference an accessToken instead of configuring the source with a username and password:

POST https://dev.vantiq.com/api/v1/resources/sources
{ 
    "name": "RESTServiceName",
    "type": "REMOTE",
    "config": {
        "uri": "http://sub.domain.vantiq:9900/aDemo/add/demoData",
        "accessToken": "/system.secrets/MySecret",
        "accessTokenType": "secret"
    }
}

To configure the client connection used to communicate with the REST service, you can include a clientOptions document. Common uses of this are:

Disable the SSL trust check:

"config": {
    "uri": "https://sub.domain.vantiq:4443",
    "clientOptions": {
        "trustAll": true
    }
}

Enable the use of an HTTP proxy:

"config": {
    "uri": "http://sub.domain.vantiq:9900",
    "clientOptions": {
        "proxyOptions" : {
            "host": "hostABC",
            "port": 8080,
            "password": "pwd123",
            "username": "uuuu"
        }
    }
}

Create a REMOTE Source that uses OAuth to obtain an Access Token

POST https://dev.vantiq.com/api/v1/resources/sources
{ 
    "name": "OAuthAuthenticatedRESTServiceName",
    "type": "REMOTE",
    "config": {
        "uri": "http://sub.domain.vantiq:9900/aDemo/add/demoData",
         "oauthSourceName": "authSource",
         "oauthGrantType": "client_credentials",
         "oauthClientId": "myClientIdentifier",
         "oauthClientSecret": "@secret(myClientSecret)",
         "requestDefaults": {
            "requestTimeout": "30 seconds"
         }
    }
}

This request creates a REMOTE source named OAuthAuthenticatedRESTServiceName. This source uses another source named authSource to obtain an access token using the client_credentials flow, and using the myClientIdentifer as the client id, and the value contained in the Vantiq secret myClientSecret as the client secret.

The source AuthSource would be defined as follows:

POST https://dev.vantiq.com/api/v1/resources/sources
{ 
    "name": "authSource",
    "type": "REMOTE",
    "config": {
        "uri": "http://some.oauthserver.com/
    }
}

Delete a REMOTE Source

DELETE https://dev.vantiq.com/api/v1/resources/sources/RESTServiceName

Deletes the source created in the previous example.

Stream Data from REST Service

A stream sourced from the REST service is defined by setting the pollingInterval. A request will be issued every pollingInterval seconds. The response to the request generates an event which is delivered to any subscribing rules.

An example definition of a streaming REST source:

{
     "name": "RESTServiceName",
     "type": "REMOTE",
     "config": {
        "uri": "http://sub.domain.vantiq/streaming/source",
        "username": "vantiq",
        "password": "vantiq",
        "pollingInterval": 60
    }
}

This source will issue a GET (the default HTTP method) on the URI every 60 seconds. An example definition with optional clientOptions:

{
     "name": "RESTServiceName",
     "type": "REMOTE",
     "config": {
        "uri": "https://sub.domain.vantiq/streaming/source",
        "username": "vantiq",
        "password": "vantiq",
        "pollingInterval": 60
        "clientOptions": {
            "trustAll": true,
            "connectTimeout": 120,
            "proxyOptions" :    {
               "host": "hostABC",
               "port": 8080,
               "password": "pwd123",
               "username": "uuuu"
            }
        }
    }
}

Read from REST service

The REST service can respond to queries when SELECT is issued on the remote source. The WITH clause of the SELECT is used to specify the request document to use when the select is performed. The request will also incorporate any defaults provided as part of the source definition.

The body of the response carries the results of the query that will be delivered to the caller of SELECT. For example:

var someData = SELECT FROM SOURCE someSource WITH path = "/api/getData", method = "GET"

Publish Notifications to REST Service

The PUBLISH request for remote sources takes three parameters: the source to which the publish is sent and the body and publish parameters object. The message body and any publish parameters are combined with any request defaults provided by the source to form the final request document which is used to perform the PUBLISH.

For example:

PUBLISH { body: { data: "somedata" } } TO SOURCE someRemoteSource USING { path: "/submit" }

Form encoding

If the content type of a request is x-www-form-urlencoded, and the request body consists of key-value pairs represented as a VAIL object, each key and value is URL-encoded. The resulting key-value pairs are then combined into a single string and sent as the request body. If the body is already a string, it is sent as-is without additional encoding.

JSON Results

If the remote source declares a content type of json, results of polling and query operations are returned as JSON objects and JSON arrays. The results can be manipulated in rules using the standard notations of accessing lists, objects, properties.

XML Results

If a remote source declares a content type of xml, results of polling and query operations are returned using an internal representation of the XML element structure. If the result is used in a rule, the rule author must take care to properly navigate the XML structure. The following summarizes the most important access rules using the example XML:

<houses>
    <house bedrooms="2", bathrooms="2">
        <address>25 Main Street, Anywhere CA</address>
        <color>green</color>
    </house>
     <house bedrooms="3", bathrooms="3">
        <address>50 Main Street, Anywhere CA</address>
        <color>blue</color>
    </house>
</houses>

An XML document may be received by invoking a query using the SELECT operation to query the remote source for the list of houses:

var query = SELECT FROM SOURCE "someRemoteSource" WITH parm = 1

When the query returns the XML document is assigned to the variable query. The XML document can then be examined by navigating using element names and attribute names. The set of houses can be obtained via the navigation expression:

houseList = query.house

Note that house returns the array of houses that are children of the houses element. The array of houses is assigned to the variable houseList. A single house within the list of houses can be reference by the house’s index in the house array:

firstHouse = query.house[0]

returns the first house in the array of houses while:

secondHouse = query.house[1]

returns the second house in the array of houses. An attribute can be referenced using the XPath syntax [“@“]. Note the use of double quotes to identify the attribute name:

bedrooms = query.house[0]["@bedrooms"]

will return an XML element that contains the number of bedrooms. However, the returned value is an element. In order to obtain the actual value of the element (the value: “2”) the reference must explicitly request the text value of the attribute as follows:

bedroomsAsText = query.house[0]["@bedrooms"].text()

Nested elements can be referenced using the dot (.) notation to navigate the hierarchical structure of the XML document. For example, to obtain the address of the second house in the list:

query.house[1].address.text()

will return the value:

50 Main Street, Anywhere CA

In the example above all the sub-elements of houses are elements named house. In XML the sub-elements may be of multiple types. The sub-elements can be directly referenced by their element names. A more general example:

<places>
    <house bedrooms="2", bathrooms="2">
        <address>25 Main Street, Anywhere CA</address>
        <color>green</color>
    </house>
     <house bedrooms="3", bathrooms="3">
        <address>50 Main Street, Anywhere CA</address>
        <color>blue</color>
    </house>
    <office floor="2", size="2000">
        <address>75 Main Street, Anywhere CA</address>
        <color>green</color>
    </office>
     <office floor="3", size="10000">
        <address>100 Main Street, Anywhere CA</address>
        <color>blue</color>
    </house>
</places>

In this example the first house can be referenced with the following expression:

var query = SELECT FROM SOURCE "someRemoteSource" WITH parm=1
firstHouse = query.house[0]

This is exactly the same syntax as in the previous example. In detail it asks the system to find the first element of type house that is a sub-element of the root of the XML document (the places element).

The second office in the list can be obtained in a similar fashion.

secondOffice = query.office[1]

interpreted as find the second element of type office that is a sub-element of the root of the XML document.

Document, Image, or Video Operations

A REMOTE source can be used to upload or download (i.e., send or receive, respectively) Vantiq Documents, Images, or Videos.

Receiving a Document, Image, or Video

To receive a Vantiq Document, Image, or Video from a REMOTE source, provide a saveAs parameter to the WITH clause of the SELECT statement. The value of the saveAs parameter should be a ResourceReference identifying the where to save the result.

For example, the following VAIL code will fetch an image from a REMOTE source (here, called imageSource), saving it into a VAIL document named myImage.

var imageRef = "system.images/myImage"
var fetchedImg = SELECT ONE FROM SOURCE imageSource WITH saveAs = imageRef

The return value of the SELECT statement (here, stored in the fetchedImg variable), will be the resource reference into which the image was saved.

Sending a Document, Image, or Video Using PUBLISH

Sending a Vantiq Document, Image, or Video is done using the PUBLISH statement. Vantiq supports sending such data as the directly embedded content of the PUBLISH message or via multipart/form-data. These two types are the most common upload formats for large data elements.

Both forms of upload involve the use of a packaged reference. A packaged reference allows us to optionally encode the data as Base64 when required. When uploading data to a REMOTE source directly, this will rarely be required (i.e., only if the REMOTE source itself requires it). A
packaged reference is generated using the VAIL procedure Utils.packageReference(rRef) where rRef is a ResourceReference for the Vantiq document, image, or video to upload.

Embedding the Data Directly

To upload a Vantiq document, image, or video directly, PUBLISH to the source in question, setting the body of the message to the packaged reference of the data required. For example, to upload a Vantiq document named aBeautifulPicture, we would use the following VAIL code

    var path = ...
    var rRef = "system.documents/aBeautifulPicture"

    PUBLISH { body: Utils.packageReference(rRef) } TO SOURCE imageSource USING {
                    path: path
    }

where path is set to the path required by the server to which you are uploading. The method parameter can be set to override the default of POST. For example, to upload the same document using the HTTP PUT method, use

    var path = ...
    var rRef = "system.documents/aBeautifulPicture"

    PUBLISH { body: Utils.packageReference(rRef) } TO SOURCE imageSource USING {
                    method: "PUT",
                    path: path
    }

The contentType will be set to the fileType of the Vantiq Document, Image, or Video if not specified. If a value is provided, it will be used (except for the special case of multipart/form-data outlined below).

Using Content-Type multipart/form-data

Sending a single entity

A common case for uploading large binary data is to send it using the HTTP content type multipart/form-data. The Vantiq REMOTE source is capable of utilizing this format.

To upload data in such a way, you will set the contentType parameter to multipart/form-data; boundary=... as shown below. The value of boundary must be a value known not to be found in the data being uploaded. A common case is some random string of characters, but you can use any value you find appropriate. The boundary value is used in the HTTP message to delimit the uploaded content.

    var rRef = "system.videos/testVideo.mp4"
    var boundary = "lkadifuloiuwrnkcnpow983lsl8sla"

    PUBLISH { body: Utils.packageReference(rRef) } TO SOURCE imageSource USING {
        path: path,
        contentType: "multipart/form-data; boundary=" + boundary
    }

Sending multiple parts

Sometimes, it is desirable to send a multipart message where the sender and receiver are aware of the parts. To do this, we use the parts property to specify the content. The parts property is used instead of the body property.

For example, to send a multipart message containing some object in a part named specification and an image showing the part, we could use

PUBLISH { parts: [ { name: "specification",
                     contentType: "application/json",
                     content: { specName: "partSpec", 
                                specId: "someSpecificationId",
                                partName: "widget 42",
                                specDescription: "This describes widget 42." }},
                   { ref: "system.documents/widget42Image.jpg" }
                 ] } TO SOURCE mySource

Alternatively, if we expect some response, we can use the SELECT form:

SELECT ONE FROM SOURCE mySource WITH
                method: "POST",
                parts: [ { name: "specification",
                           contentType: "application/json",
                           content: { specName: "partSpec", 
                                      specId: "someSpecificationId",
                                      partName: "widget 42",
                                      specDescription: "This describes widget 42." }},
                         { ref: "system.documents/widget42Image.jpg" }
                       ]

As with the direct upload above, the method parameter can be set to the appropriate HTTP Method name, overriding the default value of POST.

Note that this form can be used to send a single entity as a single part as well.

Sending Document, Image, or Video via JSON

It is likely to be quite rare, but there may be cases when you need to embed document, image, or video data in a JSON message. This may be to send data to some specialized service that includes other information in the JSON body. To do this, we again make use of the packaged references outlined above.

Specifically, we can provide the packaged reference as the value for a property in our VAIL message body. For example, to perform a query on our imageSource, sending the data from the Document aBeautifulPicture, we would use the following code.

    var rRef = "system.documents/aBeautifulPicture"

    var fetchEnt = SELECT ONE FROM SOURCE imageSource WITH
            method = "POST",
            body = { name: "myPicture",
                     picture: Utils.packageReference(rRef) }

This will send a JSON message with a name property set to myPicture and a picture property set to the value of the document aBeautifulPicture. If the fileType of the referenced document is not text, the value of picture will be base64 encoded. If that document’s fileType is text, the picture value will not be base64 encoded.

If you need to perform base64 encoding regardless of the target’s fileType, set the forceBase64 parameter to Utils.packageReference() to true.

    var rRef = "system.documents/aBeautifulPicture"

    var fetchEnt = SELECT ONE FROM SOURCE imageSource WITH
            method = "POST",
            body = { name: "myPicture",
                     picture: Utils.packageReference(rRef, true) }

In this case, the picture property will be base64 encoded regardless of the fileType of the referenced document, image, or video. Similarly, if you wish to PUBLISH data to a REMOTE source with document, image, or video data in the VAIL message, you would use something like the following.

    var rRef = "system.documents/aBeautifulPicture"

    PUBLISH { body: { name: "myPicture",
                      picture: { subject: "sunset",
                                 imageContent: Utils.packageReference(rRef)
                               }
                    }
            }

This example publishes a message whose body consists of a property name with the value myPicture, and a property picture whose value is a nested object with properties subject with value sunset and imageContent containing the value of the document aBeautifulPicture. As with the previous example, imageContent’s value will be base64 encoded if the fileType of aBeautifulPicture is not text. To override this behavior, provide the forceBase64 parameter to Utils.packageReference().

Size Limits When Operating on Documents, Images, or Videos

In all of these cases, the documents, images, or videos referenced are expanded in the Vantiq server as required. This expansion is controlled by organization quotas that limit how much memory these document, image, or video expansions can concurrently consume. Should these limits be exceeded, operations will wait until there is sufficient quota available. However, if a particular request would exceed the quota by itself, the request will result in an error.

Receiving the full response

When performing a SELECT the caller may request that the source return the entire response document instead of just the content in the body of the response. This is done by setting the asFullResponse property to true in the WITH clause of the select. When this is done the result will be a JSON object with the following properties:

status The HTTP status code of the response.
headers The HTTP headers returned in the response. If a header has multiple values then it is returned as an array of strings.
cookies The cookies contained in the response as an array of strings.
body The body of the response (what would normally be returned by the SELECT).

Response error

If a request response status code is an error, an exception is thrown. In that case, the exception object params property
contains the following information:

source name
status message
response message
status code

SSL Setup

A Remote Source can be configured for either one-way or two-way SSL communication using the optional clientOptions configuration property. This property is the JSON object form of the
Vert.x HTTP Client Options. Below are some usage examples of this configuration option.

One-way SSL

This section’s examples assume that a remote site certificate has been signed by a CA, CA certificate added to a trust store named sourceTrustStore.jks. A Remote Source is configured to access the remote site and the user configuring the Source has access to the trust store.

If the trust store is accessible on a file system readable by the Vantiq server (e.g., an edge installation),

{
    "uri": "https://remoteSite:4443",
    "clientOptions": {
        "trustStoreOptions": {
            "path": "/path/to/sourceTrustStore.jks",
            "password": "my_store_password"
        }
    }
}

If the trust store is not accessible by the server in which the remote source is being defined, the trust store content can be specified as a base64 encoded value,

# Copy/paste the following output to the value property below
$ cat /path/to/sourceTrustStore.jks | base64

{
    "uri": "https://remoteSite:4443",
    "clientOptions": {
        "trustStoreOptions": {
            "value": "/u3+7QAAAAIAAAABAAAAAgAGY2F........SzpeAUc7WXDK1HOg==",
            "password": "my_store_password"
        }
    }
}

The value can also be stored as a Secret and referenced in the configuration,

# Copy/paste the following output to a Secret named SourceTrustStore
$ cat /path/to/sourceTrustStore.jks | base64

{
    "uri": "https://remoteSite:4443",
    "clientOptions": {
        "trustStoreOptions": {
            "value": "@secrets(SourceTrustStore)",
            "password": "my_store_password"
        }
    }
}

Note that the store password can also be defined as a Secret using the same @secrets() syntax.

For example,

# Also define a Secret named SourceTrustStorePassword containing the trust store password

{
    "uri": "https://remoteSite:4443",
    "clientOptions": {
        "trustStoreOptions": {
            "value": "@secrets(SourceTrustStore)",
            "password": "@secrets(SourceTrustStorePassword)"
        }
    }
}

Assuming that the CA certificate is also accessible from a file named ca-cert in PEM format,

{
    "uri": "https://remoteSite:4443",
    "clientOptions": {
        "pemTrustOptions": {
            "certPaths": ["/path/to/ca-cert"],
            "password": "my_store_password"
        }
    }
}

This configuration can also be expressed using a base64 value,

# Copy/paste the following output as one String entry in the certValues array below
$ cat /path/to/ca-cert | base64

{
    "uri": "https://remoteSite:4443",
    "clientOptions": {
        "pemTrustOptions": {
            "certValues": ["LS0tLS1CRU.........0FURS0tLS0tCg=="]
        }
    }
}

The above example could also be defined using a Secret. Assuming two certificates ca-cert-1 and ca-cert-2
both in PEM format,

# Copy/paste the following output to a Secret named CertAuthority1
$ cat /path/to/ca-cert-1 | base64
# Copy/paste the following output to a Secret named CertAuthority2
$ cat /path/to/ca-cert-2 | base64

{
    "uri": "https://remoteSite:4443",
    "clientOptions": {
        "pemTrustOptions": {
            "certValues": ["@secrets(CertAuthority1)", "@secrets(CertAuthority2)"]
        }
    }
}

SSL Client Authentication

In addition to specifying a trust store, a source configuration can also specify a client certificate. This is necessary if the server requires the client to authenticate with a certificate.

The examples below assume a key store named sourceKeyStore.jks containing the client certificate signed by the CA. If the key store is accessible on a file system readable by the Vantiq server,

{
    "uri": "https://remoteSite:4443",
    "clientOptions": {
        "trustStoreOptions": {
            "path": "/path/to/sourceTrustStore.jks",
            "password": "my_store_password"
        },
        "keyStoreOptions": {
            "path": "/path/to/sourceKeyStore.jks",
            "password": "my_keystore_password"
        }
    }
}

Similarly to the trust store examples, any keystore specified using the path syntax can be specified as a base64 encoded value. For example using Secret definitions,

# Copy/paste the following output to a Secret named SourceTrustStore
$ cat /path/to/sourceTrustStore.jks | base64
# Copy/paste the following output to a Secret named SourceKeyStore
$ cat /path/to/sourceKeyStore.jks | base64

{
    "uri": "https://remoteSite:4443",
    "clientOptions": {
        "trustStoreOptions": {
            "value": "@secrets(SourceTrustStore)",
            "password": "my_store_password"
        },
        "keyStoreOptions": {
            "value": "@secrets(SourceKeyStore)",
            "password": "my_keystore_password"
        }
    }
}

Note that if a keystore contains several keys you can specify which key to use by providing the alias store options property.

Refer to the Vert.x HTTP Client Options document for a complete list of configuration options. Note: in that document, reference to Buffer means that a base64 encoded value can be specified (e.g., trustStoreOptions). Any add method translates into an array (e.g., pemTrustOptions) and any set method translates into a single property setting (e.g., path or value).

Realm Authentication

Some REST APIs can use an API key as a means of authentication. An API Key is a Token and must always be specified with a realm value. Although the realm value for API keys is often Bearer, the value might differ based on the REST APIs that you are accessing. Make sure to check the REST APIs documentation for the proper realm value to specify.

The below example configures a Remote Source to send heartbeats to an Opsgenie heartbeat endpoint. The API Key is stored as a Secret named HeartbeatAccessKey and the REST documentation instructs to use the realm GenieKey.

{
    "name": "HeartbeatSource",
    "type": "REMOTE",
    "config": {
        "accessToken": "/system.secrets/HeartbeatAccessKey",
        "accessTokenType": "secret",
        "realm": "GenieKey",
        "uri": "https://api.opsgenie.com/v2/heartbeats/"
    }
}

With the following Procedure sending heartbeats,

PROCEDURE sendHeartbeat(heartbeatName String)
    var pingPath = heartbeatName + "/ping"
    select from source HeartbeatSource with path = pingPath

JWT Authentication

If a REST API requires to authenticate with a JWT token, you can use the JWT service procedures and include the token in the Authorization header.

For example, assuming a remote source named RemoteAPI and a signing key MyPrivateKey configured as a secret,

// Claims to be included in the JWT token
var claims = {
  "iat": 1686095226,
  "exp": 1686116826,
  ...
}

// Create the JWT token using the RS256 algorithm
var jwt_token = JWT.createTokenUsingResource("RS256", claims, "system.secrets", "MyPrivateKey")

// Access the Remote API authenticating with the JWT token
var response = select from source RemoteAPI with headers = {Authorization: jwt_token}

Was this article helpful?

0 out of 5 stars

5 Stars		0%
4 Stars		0%
3 Stars		0%
2 Stars		0%
1 Stars		0%

Remote Source Integration

0 out of 5 stars

Defining a Remote Source

Fetching Access Tokens on Demand

Making a Request to a Remote Source

Create a REMOTE Source

Create a REMOTE Source that uses OAuth to obtain an Access Token

Delete a REMOTE Source

Stream Data from REST Service

Read from REST service

Publish Notifications to REST Service

Form encoding

JSON Results

XML Results

Document, Image, or Video Operations

Receiving a Document, Image, or Video

Sending a Document, Image, or Video Using PUBLISH

Embedding the Data Directly

Using Content-Type multipart/form-data

Sending a single entity

Sending multiple parts

Sending Document, Image, or Video via JSON

Size Limits When Operating on Documents, Images, or Videos

Receiving the full response

Response error

SSL Setup

One-way SSL

SSL Client Authentication

Realm Authentication

JWT Authentication

0 out of 5 stars

Please Share Your Feedback

How Can We Improve This Article?

Getting Started

Introductory Tutorials

Feature Tutorials

Product Documentation

Reference Guides

Client Builder

Operations & Management

Design Modeler

Service Builder

Branding

App Builder

Assemblies

Catalog

Image Processing

External Sources

Storage Containers

Testing

Core Platform

Release Notes

Articles

Architecture

Best Practices

Performance