/An introduction to Messaging Protocols for constrained devices – Part 1: MQTT

An introduction to Messaging Protocols for constrained devices – Part 1: MQTT

Recently I have been studying sensor systems and applications (Internet of Things – IoT) at school and as a result I have taken a keen interest in communication protocols for constrained devices. In this post I will be using the terms IoT and constrained devices interchangeably.

It is fascinating to see these small pieces of electronics being able to communicate data between themselves or to the cloud, because unlike traditional communication on the web, which relies heavily on the resource-hungry TCP /IP stack, IoT devices typically have very limited resources in terms of memory and processing power in order to utilize the same protocol stack for communication. So how do they do it?

The IoT Protocol Stack

Before we delve into messaging layer protocols, let us take a look at what a typical IoT architecture can consist of. Take a look at this graphic from David E. Culler, which illustrates the different protocols that can be implemented at each layer of the IoT stack. While there is no single consensus on what an IoT architecture should look like, this diagram does a neat job at illustrating what it can look like.

Fig 1.1: The Internet of Every Thing – steps toward sustainability CWSN Keynote, Sept. 26, 2011

Unlike web 2.0, which uses the standardized TCP/IP stack, IoT networks have no standardized stack for communication. As a result, several organizations are developing their own stacks and have been pushing for their adoption in the IoT landscape. If you look at the physical layer of the graphic above, you can see how many protocols exist just for that layer. There may be even more protocol additions to these layers given fast IoT is gaining popularity.

For instance, a recent addition to the protocol stack at the physical level is the 6LoWPAN (IPv6 over Low-Power Wireless Personal Area Network) which is being developed by the Internet Engineering Task Force (IETF) for IoT and M2M, with the aim of enabling IP networking for devices with very limited small factor that currently support the IEEE 802.15.4 specification.

Messaging Protocols for IoT

Messaging protocols form part of the session layer of an IoT architecture. However they can also be considered as part of the application layer. These protocols allow constrained devices to communicate data acquired from sensors to edge devices or to the cloud. While there are several messaging protocols out there, we will take a look at the 3 most popular messaging protocols only, i.e, MQTT, CoAP and AMQP.

1. MQTT

MQTT (MQ Telemetry Transport) is a lightweight TCP-based M2M messaging protocol which has been developed by IBM and is widely used in constrained environments with issues such as unreliable communication.

Originally MQTT was an abbreviation for MQ Telemetry Transport, with MQ being the name of the product series that IBM developed to support the protocol (MQSeries). However when IBM submitted MQTT to the OASIS Standard Group, it was labelled as Message Queuing Telemetry Transport, even though MQTT is not directly related to message queues.

In contrast to the traditional request-response messaging pattern used by the client/server architecture when communication over HTTP, MQTT uses the Publish/Subscribe (Pub/Sub) messaging pattern to allow communication between constrained devices. This architecture involves 3 components, namely, publishers, subscribers and a message broker.

The Publisher can be any device which has acquired some data that can be distributed/used. For example, sensors.

The Message Broker is a device running a specialized software which takes data from publishers and classifies them in channels known as “Topics”.

The subscriber is any device which wants to view/use data collected by publishers. To receive specific data, a subscriber has to connect to a broker and subscribe to the specific topic which they are interested in. A subscriber can also subscribe to multiple topics.

This communication architecture can be illustrated with the following fictitious scenario. Suppose we have a temperature sensor on the first floor of our office and a motion sensor in our first server room.

Fig 1.2: High level overview of a typical MQTT architecture

To distribute the results obtained from the sensors, we tell the temperature sensor to send a publish event to our message broker with the topic cyberstorm/hq/office/floor/1/temp and its payload, which is 26 in this case. We do the same for the motion sensor except this time we want to publish its results to another topic, cyberstorm/hq/serverroom/1/status.

Now that these results are being published to their respective topics, we can access them by simply subscribing to cyberstorm/hq/office/floor/1/temp and cyberstorm/hq/serverroom/1/status. To do that we can either use an off the shelf application which has been manually configured to listen to publish events for a given topic on the message broker or implement a custom one.

In this case, we are subscribing to temperatures and motion activity in the 1st floor and in the 1st server room. But what if we wanted to monitor temperatures and motion activity for all floors and all server room? MQTT provides a convenient way to to so – by using wildcards. There are 2 types of wildcards, single-level(+) & multi-level(#) wildcards.

The single-level wildcard will match any string for a single topic level at that position in the topic structure. For example: cyberstorm/hq/office/floor/+/temp, will return the temperatures for all the floors found in our office, thus satisfying the use case we discussed above. Assuming our office spaces occupy 5 floors, each equipped with a temperature sensor, the resulting output will look like this:

cyberstorm/hq/office/floor/1/temp,
cyberstorm/hq/office/floor/2/temp,
cyberstorm/hq/office/floor/3/temp,
cyberstorm/hq/office/floor/4/temp,
cyberstorm/hq/office/floor/5/temp

The multi-level wildcard will match any string for zero or more topic levels at the end of the topic structure. Basically this means that this wildcard will return values for all topics registered based on how it is used with the topic structure. For example cyberstorm/hq/office/floor/#, will return all sensor readings for all floors. If we had humidity sensors, then this wildcard would return humidity and temperature sensor readings for all floors.

MQTT Control Packet and Message Flow Control

As specified by the OASIS Standard for MQTT v5.0 (07 March 2019), MQTT operates by exchanging a series of MQTT Control Packets in a defined way. This implies that for every command sent, a command acknowledgement is received.

An MQTT Control Packet consists of a fixed header and, optionally a variable header (which may contain the packet Identifier, depending on QoS) or a payload.

Fixed header, present in all MQTT Control Packets
Variable header, present in some MQTT Control Packets
Payload, present in some MQTT Control Packets

Structure of MQTT Control Packet

Control packets commands can be grouped into the following categories:

  1. Connection Management: CONNECT, CONNACK, DISCONNECT, PINGREQ, PINGRESP
  2. Subscription Management: SUBSCRIBE, SUBACK, UNSUBSCRIBE, UNSUBACK
  3. Message Delivery: PUBLISH, PUBACK, PUBREC, PUBREL, PUBCOMP
  4. Authentication: AUTH

Most of the above Control Packet commands are used in MQTT message flow, with PUBREC, PUBREL and PUBCOMP depending on the QoS level set.

There are 3 levels of Quality of Service (QoS):

  1. QoS 0: At Most Once
  2. QoS 1: At Least Once
  3. QoS 2: Exactly Once

QoS 0 Flow Control

Communicating with QoS 0 is considered the fastest way since is requires sending only 1 message – PUBLISH, without the need for it to be acknowledged. This is an advantage in terms of speed but a pitfall in terms of reliability.

Fig 1.3: Flow Control with QoS 0

QoS 1 Flow Control

With QoS set to 1, messages are guaranteed to be delivered at least once. However they may be delivered more than once, causing duplicate data to be sent. It requires 2 messages as when the sender sends a PUBLISH message, it expects a PUBACK to be received.

Fig 1.4: Flow Control with QoS 1

QoS 2 Flow Control

The “Only Once” QoS level guarantees that messages are delivered only once. For this level of guarantee, a much complex mechanism is required. This mechanism consists of 4 messages, thus making this QoS level the slowest.

  1. First the publisher sends a PUBLISH message and waits for a PUBREC.
  2. If the broker receives the messages, it sends a PUBREC to the publisher to acknowledge that it received the message.
  3. If it does not, then the publisher does not receive any message and re-sends its message with a DUP (Duplicate) flag set.
  4. When the publisher gets a PUBREC message, it sends a Publish Release (PUBREL) message to the broker telling it to forward the message to its subscribers.
  5. If the broker does not receive a PUBREL message from the publisher, it will re-send the PUBREC message.
  6. Once the PUBREC and PUBREL messages have been exchanged, the message is then deleted from the queue and forwarded to subscribers.
  7. The broker then sends a PUBCOMP message to indicate that the process has been completed.
  8. If the publisher does not receive a PUBCOMP message it will re-send a PUBREL message.
  9. With the PUBCOMP message being received on the publisher’s end, the message is deleted from the outbound queue along with the message state.
Fig 1.5: Flow Control with QoS 2

Limitations of MQTT

One of the major limitations of MQTT is security. Out of the box, MQTT offers a basic secure communication mechanism consisting simply of a username and password. By default MQTT does not use encryption, therefore usernames and passwords are communicated in plaintext! This is terrifying as anyone with basic networking knowledge can look up default MQTT open ports and try to takeover a device. Remember Mirai?

It is the responsibility of the developer to implement proper security measures for communication over MQTT as well as at all other levels of their IoT stack.

Another issue, although not too alarming is the fact that brokers tend to be centralized which acts as a single point of failure. However this can be mitigated by implementing distributed broker clusters which can be seen as a single logical broker by clients

Conclusion

MQTT can be a fast and secure communication protocols when implemented correctly. It has a range of features for message delivery as well as persistent sessions. It can also be scaled up easily to meet growing demands given its decoupled pub/sub nature, making it ideal for IoT projects.