Overview#

MQ Telemetry Transport is a Client – Server Publish/ Subscribe messaging transport protocol and was initially named after the MQ series of IBM where the protocol was first developed for proprietary embedded systems in the late 90’s.

Nowadays, MQTT is considered as the protocol name, as opposed to be an acronym previously, and is the de facto standard ↗ by OASIS and later ISO.

Throughout the development of MQTT, ease of use was always the key concern and thus the protocol are said to be easy to implement on the client side and excel at transferring data over the wire since it is lightweight, simple, and a widely adapted open standard.

Prerequisite#

1. A MQTT Client (for Desktop), e.g. MQTT.fx ↗, a cross-platform client (Windows, macOS, Linux) built under the Eclipse Paho Project. Another option is Mosquito MQTT ↗.

2. (Optional) Setup Mathwork free account for ThingSpeak ↗ & MATLAB Online ↗

NOTE: From this point onwards, the term MQTT is referring to the MQTT Version 3.1.1.

Understand how MQTT works#

The Pub/ Sub pattern of MQTT provides a decoupling mechanism between the Client and Server where the Broker acting as a medium between them (this can be extrapolated from the preview image).

• Space Decoupling: Publisher and Subscriber do not need to know each other, i.e. no exchange of IP address and port.

• Time Decoupling: Publisher and Subscriber do not need to run at the same time, e.g. sensor update at specific interval and only retrieved to end-user when needed.

• Synchronization Decoupling: Operations on both components do not need to be interrupted during sending or receiving.

The Fixed Header#

There are three different aspects of MQTT include basic concepts (Publish/ Subscribe, Client/ Broker), basic functionalities (Connect, Publish, Subscribe) of MQTT and basic features such as Quality of Service, Retained Messages, Persistent Session, Last Will and Testament, Keep Alive and more.

Fixed Header format

In general, the protocol has three different headers ↗ vary by the package type.

Structure of an MQTT Control Packet

The Variable Header#

A Variable Header includes a 2 byte Packet Identifier field and resides between the Fixed Header and the Payload. Although, not every Control Package requires the identifier field ↗.

An example of Variable Header of the CONNECT packet with Protocol Length, Protocol Name, Protocol Level, Connect Flag, and Keep Alive fields

For the above example, every bits encoded in hexadecimal format, e.g. Keep Alive field of 60 seconds ($60_{DEC} = 3C_{HEX}$).

The Payload#

A Payload is the final part of the packet and also optional depends on the Control Package type. In the case of the PUBLISH packet this is optional and commonly named as the Application Message, whereas it’s required in the SUBSCRIBE package and contain Topic Filter, Quality of Service.

An example of Payload of the CONNECT packet with Client Identifier fields

Quality of Service#

MQTT Protocol ensures high delivery guarantees with 3 levels of QoS serve to the constraint of embedded system communication in previous discussion.

• QoS 0: Guarantees a best effort delivery but not result.

QoS 0 delivers regardless of the result

• QoS 1: Guarantees that a message will be delivered at least once.

QoS 1 delivers with the result accepted at least once

• QoS 2: Guarantees that each message is received only once by the counterpart

QoS 2 delivers with the result accepted only once

Beside that, MQTT also provides other mechanism for quality control.

1. Last will means in case of unexpected disconnection of a client all subscribed clients will get a message from a broker.

2. Testament and Retained message means that a newly subscribed client will get an immediate status update.

Implement MQTT for practical uses#

Now that you have an idea of MQTT, let’s examine it in action using MQTT.fx to establish the connection and Wireshark to capture the packet.

In MQTT.fx Setting tab, select MQTT Broker under Profile section and fill in ThingSpeak Server info as above

Under tab Subscribe, use the below syntax to Subscribe any Public channel ↗

channels/<channel_ID>/subscribe/json

Once you receive the channel feed, it means that

1. The connection was requested & Server Acknowledged Request

2. Subscription requested & Server Acknowledged Subscription

3. Message Published from Server to MQTT.fx client

This can be verified in Wireshark by select “Start Capturing Packet” icon in the Top bar and re-do every steps.

MQTT packet captured while using MQTT.fx to the Broker on ThingSpeak server

Under network layer section, it can seen that MQTT indeed working on top of the TCP/IP layer. Beside that, Wireshark provides deeper look into the protocol and exactly how the packet was formed.

Analysis of MQTT CONNECT packet headers and flags setup

This can breakdown into several headers as discussed, i.e. Fixed Header with Control Packet Type, Variable Header with Protocol Length/ Name, Connect Flags, Keep Alive fields and lastly the Payload with Client ID and Authentication.

Also, these information can be interpreted in raw hexadecimal format too.

Same MQTT packet in raw HEX code with Headers highlighted and Fields separated

By using any DEC-HEX decoder, these code will yield similar result as expected.

Analysis of the same CONNECT packet in Microsoft Excel

Grab it here.

ThingSpeak & MATLAB Application#

The great thing about ThingSpeak & MATLAB combination is that it provides an easy-to-implement approach without any restriction of further improvement.

ThingSpeak panels provide basic data collection and visualization

Many tools can be found under Apps top menu on your ThingSpeak profile with two main categories: Analytics and Actions. Also, MATLAB also provides excellent MQTT Support Toolbox ↗ for both reading and writing data directly to the ThingSpeak database.

% thingSpeakRead(<channelID>, <Name>, <Value>)
[data,time] = thingSpeakRead(12397)

% Or specific Fields with Output Format flag
thingSpeakRead(12397,'Fields',[1 4],'NumPoints',10,'OutputFormat','table')

% Now try to write to your private channel
% NOTE: Example Key Value

% Update multiple fields at once 
thingSpeakWrite(<channelID>, 'WriteKey', <write-key>,...
		[field1_val, field2_val, field3_val]) 

% Or automated process for batch authentication from multiple channels
thingSpeakAuthenticate(<channelID>, <Name>, <Value>)

An Example of Bi-Histogram from ThingSpeak data imported for a Transportation Tracking application

Troubleshooting (Extra)#

• Authorization issues in MQTT.fx

Make sure that you type in User-ID and MQTT-API-key for fields <User Name> and <Password> of User Credentials Tab respectively.

Both can be found under Account $\rightarrow$ My Profile on your ThingSpeak profile.

• Decimal-Hexadecimal Conversion

In case you could not access the Excel file for reading the content of a MQTT packet, try to use directly in MATLAB Online ↗ with the Dec2Hex ↗ built-in function.