Messaging

Introduction

The ClearBlade Platform supports MQTT, a lightweight IoT messaging protocol. ClearBlade has built an OAuth 2 security model into the MQTT broker to provide out-of-the-box messaging security.

Click here to learn about sending messages.

Purpose

Lightweight messaging is key to moving data around an IoT system at scale.

Workflows

Workflow 1: Upon publishing, a message is processed with code services, which publishes the output.

Workflow 2: Code service is invoked, computation is performed, and output is published.

MQTT

The ClearBlade Platform implements the full specification of MQTT 3.1.1 to allow for multiple quality of service (QoS) settings that best fit a developer's use case.

QoS

Quality of service	Definition	Description

Quality of service	Definition	Description
0	At most once	The message is only sent once whether it makes it to the subscriber or not.
1	At least once	The message will be sent one or more times to the subscriber until it is received.
2	Exactly once	The message will be received by the subscriber exactly once.

MQTT ports and their use

Port	TLS	Information	Requirements

Port	TLS	Information	Requirements
1883	False	MQTT Pub/Sub	user-token
1884	True	MQTT Pub/Sub	user-token
8903	False	WebSockets, usually used by browsers/applications (cannot communicate directly over MQTT)	dev-token
8904	True	WebSockets	dev-token
8905	False	Auth over MQTT	SystemKey, Secret, Username, Password, ClientId
8906	True	Auth over MQTT
8907	False	Auth over MQTT via WebSockets	TODO
8908	True	Auth over MQTT via WebSockets	TODO

Working with MQTT

These steps are involved with MQTT:

1. Prerequisites

2. Broker connection

3. MQTT authentication

Prerequisites

You must have a user or device with valid permissions to publish messages.

Please follow the instructions to do so:

1. Log in or create a developer account on a ClearBlade Platform instance.

2. View or create a system.

3. View or create a user.

4. View or assign a role to the user.

5. View the Roles page.

6. Add a topic to the message topics list enabled for that role (for example, mytopic).

Broker connection

The MQTT protocol allows for the connect action to provide a username and password. We will modify the use of those fields to accommodate our OAuth-styled token model.

Key	Value	Example

Key	Value	Example
URL	URL_OF_BROKER	ClearBlade
PORT	PORT	1883
Username	USER_TOKEN	abcdefabcdef01234567890
Password	SYSTEM_KEY	f0cbf0cbf0cbf0cbf0cbf0cbf0cb
ClientID	UNIQUE_CLIENT_ID	sjdbfkasdbf

Duplicate client ID behavior

If two clients connect to a broker with the same client ID, the broker behavior can be configured to be one of the following:

1. The new client can connect, and the existing client is kicked off. Depending on the client firmware, this could result in the two clients alternately connecting and disconnecting repeatedly.

2. The existing client connection is maintained, and the new client is rejected.

The chosen configuration is set when an environment is deployed and applies to all systems (brokers) in the given environment. Configuration #1 conforms to the MQTT specification.

MQTT authentication

A ClearBlade user token is required to communicate with a broker. A token can be obtained via a REST endpoint call or MQTT.

Connect

Required keys	Description	Example values

Required keys	Description	Example values
URL	<PLATFORM_IP>	ClearBlade
Port	<PORT_NUMBER>	8905
Username	<SYSTEM_KEY>	bacb8fb60bb4d7c2c2c0e4bb9701
Password	<SYSTEM_SECRET>	BACB8FB60BFDDB7DB97D7A8BF01
ClientId	<USER_EMAIL>:<PASSWORD> for User, <DEVICE_NAME>:<ACTIVE_KEY> for devices	cbman@clearblade.com:cl34r8l4d3 or temperature-sensor:faqb8fb60bc2c2b1c0e4bb9701
TLS (optional)	<ALLOW_TLS>	true

Subscribe

To receive the new JWT token, subscribe to the auth topic (recommended).
To receive the legacy token, subscribe to the v/1/auth topic.

Extract token

ClearBlade publishes the user token on the auth message topic with bit-level encoding.
The payload is in this format:

The length block is a 2-byte unsigned 16-bit integer.
The data block is UTF-8 encoded bytes.

Block mapping in the above packet structure:

Block-Num	Description

Block-Num	Description
1	Token length
2	Token
3	User ID or device name length
4	User ID or device name
5	Messaging URL length
6	Messaging URL

Once the client is authenticated, the token can be extracted, and the connection can be established.

Error codes

Referring to MQTT 3.1.1 spec

Value	Return code response	Description

Value	Return code response	Description
0	0x00 Connection Accepted	Connection accepted
1	0x01 Connection Refused, unacceptable protocol version	The server does not support the MQTT protocol level requested by the client
2	0x02 Connection Refused, identifier rejected	The client identifier is correct (UTF-8) but unallowed by the server. A duplicate `clientid` can cause this
3	0x03 Connection Refused, Server unavailable	The network connection has been made, but the MQTT service is unavailable
4	0x04 Connection Refused, bad user name or password	The data in the user name or password is malformed
5	0x05 Connection Refused, not authorized	The client is not authorized to connect. For example, entering the incorrect password
6-255		Reserved for future use

Message topics

Topics are used to separate messages and keep them organized based on what they are about.

There are three topic types:

Regular topics

Topics comprised of string literals: /devices

Wildcard topics

Wildcard subscription means they are subscribed to multiple topics simultaneously. There are two types:

Single level (+)

The single level wildcard (+) allows a subscription to match the topic path’s entire level.

/device/+/temp

Topic	Matches

Topic	Matches
/device/a/temp	✓
/device/b/temp	✓
/device/123/temp	✗
/device/temp	✗
/device/temp/a	✗

Multi-level (#)

The multi-level # wildcard matches with all values following the #.

The following subscribes to every topic:

#

See this example:

/device/#

Topic	Matches

Topic	Matches
/device/1/2/3	✓
/device/a	✓
/abc	✗
/abc/device/	✗

A subscription topic like /Foo/#/Bar is invalid. Once a hash is in the subscription, that hash is the last element in that subscription.

Message Relay

The ClearBlade Message Relay allows edge MQTT clients to communicate with MQTT clients and other edge MQTT clients. To achieve this, a few reserved MQTT topic paths exist to follow. The Message Relay uses the following six topic paths to route messages between edges and their parent platform. Message Relay applies to communication between edges and platforms within one single system.

Message type	Topic path

Message type	Topic path
Edge to Platform	<TOPIC_NAME>/_platform
Platform to Edge	<TOPIC_NAME>/_edge/<NAME_OF_EDGE>
Edge to Edge	<TOPIC_NAME>/_edge/<NAME_OF_EDGE>
Platform to All Broadcast	<TOPIC_NAME>/_broadcast
Edge to All Broadcast	<TOPIC_NAME>/_broadcast
Edge to Edge and Platform	<TOPIC_NAME>/_edgeAndPlatform/<NAME_OF_EDGE>

These topic paths are for sending messages. To receive messages, subscribe to the same topics, except for Edge to Edge Platform (which you need to subscribe to the same topics for platform and edge).

Event topics

Event topics allow for greater scalability when handling events by enabling the platform to send an MQTT message for each event. A system user can subscribe to events by subscribing to a specific topic for an event or using an MQTT wildcard subscription to subscribe to multiple events. This can be done in stream services by using subscribe() and waitForMessage(), as well as with shared subscriptions.

Click here to view the topic paths.

The message payload of each event message consists of the same JSON-encoded object data delivered to an event microservice. Each key in the object corresponds to a key set (by the event) in the req.params input to a microservice. Do the message payload’s JSON.parse(payload) before using the message. Below is an example of working with specific event types called triggers.

Example

function triggerStreamService(req, resp) {
  var deviceCreatedTopic = "$trigger/device/created";
  var deviceConnectedTopic = "$trigger/messaging/device/connected";

  ClearBlade.init({request:req});
  var messaging = ClearBlade.Messaging();

  messaging.subscribe(deviceCreatedTopic, function (err, errMsg) {
    if (err) {
      messaging.publish("service/error", "Subscribe failed");
      resp.error("Sub failed");
    }
  });

  messaging.subscribe(deviceConnectedTopic, function (err, errMsg) {
    if (err) {
      messaging.publish("service/error", "Subscribe failed");
      resp.error("Sub failed");
    }
  });

  while (true) {
    messaging.waitForMessage([deviceCreatedTopic, deviceConnectedTopic],     function (err, msg, topic) {
      if (err) {
        messaging.publish("service/error", "Wait for message failed");
        resp.error("failed to wait for message: " + err);
      } 
      else if (topic === deviceCreatedTopic) {
        // Process device created here. Do JSON.parse(msg) first
      } 
      else if (topic === deviceConnectedTopic) {
        // Process device connected here. Do JSON.parse(msg) first
      } 
      else {
        messaging.publish("service/error", "Bad topic: " + topic);
      }
    });
  }
}

Shared subscriptions

Shared subscriptions allow clients to share the same subscription on the MQTT broker. Clients are placed in the same subscription group and will receive the messages in a queue format. The message load of a single topic is distributed among all clients in a subscription group. Multiple shared groups on one topic are not supported in shared subscriptions.

There are three parts to the topic structure for sharing:

$share/<GroupID>/<Topic>

$share keyword: A static shared subscription identifier.
GroupID: Used to identify a group.
Topic: A standard MQTT topic (including wildcards).

Sample topic: $share/StreamServiceGroup/devices/+/+/event

Example

function SharedTopic(req, resp) {
  ClearBlade.init({ request: req });
  var messaging = ClearBlade.Messaging();
  
  const TOPIC ="topic/+" 
  var sharedTopic = "$share/GROUP_ID/" + TOPIC;

  messaging.subscribe(sharedTopic, function(err, errMsg) {
    if (err) {
    	//DEBUG MESSAGING
      messaging.publish("error", "Subscribe failed: " + errMsg);
      resp.error();
    }

   messaging.publish(TOPIC, "Publishing to a shared topic");
  
}

Sync topics

These topics allow users to monitor a deployment. They give the deployment status on edges and sync at runtime. The payload history can be viewed in Notification history under the username on the top right of the page.

Name	Topic	Description	Payload

Name	Topic	Description	Payload
Sync Status	Sync is successfully deployed, and the status is returned	`$notification/sync/status`	See Sync status
Sync Error	This error happens during a sync (i.e., the device failed to deploy)	`$notification/sync/error`	See Sync error
Sync Generic Error	This error happens before a sync takes place (i.e., edge not found)	`$notification/sync/generic_error`	See Sync generic error

Sync status

{
    "asset_class": <assetClass>,
    "asset_id":    <assetId>,
    "edge":        <edge>,
    "status":      <syncStatus>
}

Sync error

{
    "asset_class": <assetClass>,
    "asset_id":    <assetId>,
    "edge":        <edge>,
    "originator":  <originator>,
    "destination": <destination>,
    "message":     <errorMessage>,
    "event":       <crudEvent>,
    "timestamp":   <timestamp>
}

Sync generic error

{
    "table": <table>,
    "error": <errMsg>
}

FAQ

1. Why can’t I see messages from a topic with a new shared group?

If you have previously subscribed to a topic with one shared group, you cannot see the messages for the same topic with a different shared group. See Shared subscriptions.

2. How can I do load balancing with trigger topics?

Using the topic path $share/<TriggerGroup>/$trigger/#.

ClearBlade IoT Enterprise and Edge

Messaging

Analytics

Introduction

Purpose

Workflows

MQTT

QoS

MQTT ports and their use

Working with MQTT

Prerequisites

Broker connection

Duplicate client ID behavior

MQTT authentication

Connect

Subscribe

Extract token

Error codes

Message topics

Regular topics

Wildcard topics

Single level (+)

Multi-level (#)

Message Relay

Event topics

Example

Shared subscriptions

Example

Sync topics

Sync status

Sync error

Sync generic error

FAQ