Project

Introduction to the MQTT Protocol on NodeMCU

September 15, 2015 by Patrick Lloyd

MQ Telemetry Transport (MQTT) is a popular low-overhead messaging protocol used by many IoT devices to communicate. This tutorial will show the user how to to set up a basic MQTT network using Python on his or her computer and an ESP8266 running the NodeMCU firmware.

Recommended Level:

Intermediate

Introduction

Minimizing power consumption is a huge driving force in the Internet of Things since a vast majority of it relies on wireless, battery-powered embedded devices. In addition to creating chips and peripherals that use less and less power, it's also important to focus on wireless protocols that maximize reliability while also minimizing computational overhead and transceiver power. The data from an IoT sensor generally isn't presented to a user directly, so they rely on machine-to-machine (M2M) communication protocols to get the data to a point where it can be processed and used in a meaningful way. One very commonly used M2M protocol used by IoT devices is called MQTT (Message Queuing Telemetry Transport). This tutorial will demonstrate how to create an MQTT network between a computer running Python and an ESP8266 development board (in this specific case a NodeMCU-Devkit v0.9) running the NodeMCU firmware. I've covered some of the capabilities of NodeMCU and the NodeMCU-Devkit in previous articles. The NodeMCU-Devkit will be connected to a single RGB LED and the user will be able to send commands to turn it off and change the blink patterns. The user will then be able to read status messages from the NodeMCU-Devkit. At first glance, an internet-connected LED may not seem like the most useful of projects, but it opens the door to the powerful world of low-power, wireless communications through MQTT. 

 

Hardware

  • Computer with internet access
    • This tutorial demonstrates everything on a laptop running Ubuntu 15.04 Vivid Vervet, but everything can be adapted fairly easily to work on Windows or Mac
  • ESP8266 Development Board
  • Common Cathode RGB LED
  • 1x 100 ohm resistor
  • 2x 470 ohm resistors

 

Software

MQTT Description and Setup

The MQTT protocol uses a publish / subscribe communications model which allows for data to be sent and received asynchronously. A web service called a broker manages where the data is coming from and going to. It's similar to REST for HTTP communications but with several very important distinctions. As an example for IoT applications, a wireless thermometer client can publish temperature data to a topic called /sensors/temp and a humidity sensor client could publish its data to /sensors/humid. Depending on the type of application, remote clients can subscribe to one or both of the topics and use that information to perform various functions like a thermometer or environment monitor. The image below shows an example network with a broker that manages topics from sensors all around the world and end users can apply this data to create applications like earthquake monitors or weather stations.

 

Our particular case is much simpler, however. First of all, the broker is only handling topics within the local network, so there is no external "cloud" component. There also aren't any sensors, per se, only an RGB LED. A system diagram for this tutorial would look something like this:

 

The MQTT broker service we're going to use in this application is an open-source project managed by the Eclipse Foundation known as Mosquitto. It will run in our Linux terminal and allow the ESP8266 development board publish and subscribe to topics on the local network. On the same computer, we will also use a Python implementation of the Paho MQTT client (also managed by Eclipse) to send and receive data on the MQTT network. The following section will show how to get everything set up on an Ubuntu system.

 

Mosquitto and the Paho Python Client

This program is available as a package in the standard Ubuntu repositories but according to the Downloads Page:

"As of version 11.10 Oneiric Ocelot, mosquitto will be in the Ubuntu repositories so you can install as with any other package. If you are on an earlier version of Ubuntu or want a more recent version of mosquitto, add themosquitto-dev PPA to your repositories list – see the link for details. mosquitto can then be installed from your package manager."

I'm running 15.04 Vivid Vervet, but I like to live dangerously so I added the PPA and installed the developer branch. Open a terminal and type:

sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa
sudo apt-get update
sudo apt-get install mosquitto-dev mosquitto-clients # The clients are optional but are useful for debugging

 

When you install Mosquitto, the service automatically starts running (just type mosquitto in a terminal if it doesn't) using the default configuration and opens ipv4 and ipv6 listen sockets on port 1883 with no authentication. We'll use this for our test setup, but it is highly recommended that you take a look at the man page for a description of what Mosquitto is capable of. Some notable configuration options are:

  • Authentication
    • Username and password
    • SSL / TLS certificate support
  • Connections to other brokers (bridge mode)
  • Restrict access to certain network interfaces
  • Change default topic templates
  • Modify port and binding address

 

The default MQTT host is just the local IP address of your computer. You can find it by typing ifconfig into the terminal.

 

To install the Python Paho MQTT client, in a terminal type:

pip install paho-mqtt # You may have to prepend sudo to this if it complains about permissions

 

Now that the broker and terminal client are installed, we want to test the functionality by forcing a client to send and receive messages to a topic at the same time. This called a loopback test. Run the following Python script after modifying MQTT_HOST and MQTT_PORT for your setup to test that the broker and Paho client are working together. A copy is also located in the zip file at the bottom of the page under (path-to-zip)/nodemcu_mqtt_rgbled/demo_code/python/mqtt_loopback.py


#!/usr/bin/env python3

'''
Simple MQTT loopback terminal client example
by Patrick Lloyd

This simple MQTT client requires no input from the user and is used to test
compatibility with the Mosquitto broker. It subscribes to the /loopback/hello
topic and once every two seconds publishes a message with a counter
'''

# Library to connect with the broker
# See http://www.eclipse.org/paho/ for more info
import paho.mqtt.client as mqtt
import time  # Can never get enough...

MQTT_HOST = ""    # Change for your setup
MQTT_PORT = 1883  # Change for your setup


# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, rc):
    print("Connected with result code "+str(rc))
    #Subscribing in on_connect() means that if we lose the connection and
    # reconnect then subscriptions will be renewed.
    client.subscribe("loopback/hello")


# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
    print(msg.topic+": "+str(msg.payload))


client = mqtt.Client(client_id="python-loopback")  # Create a client instance

# Callback declarations
client.on_connect = on_connect
client.on_message = on_message

client.connect(MQTT_HOST, MQTT_PORT, 60)

# Blocking call that processes network traffic, dispatches callbacks and
# handles reconnecting.
# Other loop*() functions are available that give a threaded interface and a
# manual interface.
client.loop_start()
index = 0
while True:
    index = index + 1
    time.sleep(2)
    client.publish("loopback/hello", payload="Hiss... buzz... #"+str(index),
                   qos=0, retain=False)


A demonstration of the loopback test can be seen in the video below:

A Note on NodeMCU Firmware

This tutorial assumes you already have NodeMCU on your ESP8266 breakout but if not, take a look at this post. To flash the NodeMCU binary to the device using Esptool, type

python esptool.py -p (port) -b (baud) write_flash 0x00000 (firmware_name).bin

Where the -p and -b options, if omitted, default to /dev/ttyUSB0 and 9600 respectively. There is also a copy of Esptool and firmware binaries in the zip file attached at the end of the page.

 

Remember to serial in to the NodeMCU (screen /dev/ttyUSB0 9600) after a fresh install and format the filesystem with the 

file.format()

command before trying to upload any Lua files.
 

Hardware Setup

The hardware setup is very simple since the software is relatively complicated. The five parts are laid out in Fritzing below:

 

Software Setup

This tutorial uses lua to program the MCU. up movieThe lua code for the NodeMCU client in this tutorial is broken up into four files, each with a specific purpose:

  • config.lua - This is the only file that you should have to modify. It contains global variables for WiFi, the MQTT broker, and the pins on the Devkit
  • init_man.lua - The manual initialization file. It connects to WiFi and sets up the GPIO. It's considered manual since it doesn't run at bootup like init.lua
  • main.lua - This is where the code is for connecting to the MQTT broker, interpreting messages, and changing the LED accordingly
  • init.lua - Once everything is tested and working, upload this file to automatically load init_man.lua at bootup

The command for getting these files onto your NodeMCU file system using Luatool is as follows:

python luatool.py -p (port) -b (baud) -f (source_file_name).lua -vr # -vr provides verbose output and restarts the NodeMCU after upload

Where the -p and -b options, if omitted, default to /dev/ttyUSB0 and 9600 respectively. There is also a copy of Luatool in the zip file attached at the end of the page. 


--[[
config.lua
by Patrick Lloyd

Global variable configuration file for better portability
Change for your particular setup. This assumes default Mosquitto config
--]]

-- Pin Declarations
PIN_RED = 1
PIN_GRN = 2
PIN_BLU = 3
PIN_BTN = 4

-- WiFi
WIFI_SSID = ""
WIFI_PASS = ""

-- MQTT
MQTT_CLIENTID = "esp-blinkenlite"
MQTT_HOST = ""
MQTT_PORT = 1883

-- Confirmation message
print("\nGlobal variables loaded...\n")


--[[
init_man.lua
by Patrick Lloyd

Actual init file, but named something other than init.lua in order to 
manually test and debug initialization code.
--]]

-- Load all the global user-defined variables
dofile("config.lua")


-- Initialization routine for RGB LED pins
function rgb_init(freq, duty)
	-- Configure PWM (freq, 50% duty cycle[512/1203])
	pwm.setup(PIN_RED, freq, duty) -- Red
	pwm.setup(PIN_GRN, freq, duty) -- Green
	pwm.setup(PIN_BLU, freq, duty) -- Blue

	-- Start the PWM on those pins (Defaults to white)
	pwm.start(PIN_RED)
	pwm.start(PIN_GRN)
	pwm.start(PIN_BLU)
end 


-- Basic color function used for boot animations.
-- Set LED red, green, blue or white
function rgb_solid(mode)
	if mode == 0 then
		-- RED MODE
		pwm.setduty(PIN_RED, 512)
		pwm.setduty(PIN_GRN, 0)
		pwm.setduty(PIN_BLU, 0)
	elseif mode == 1 then
		-- GREEN MODE
		pwm.setduty(PIN_RED, 0)
		pwm.setduty(PIN_GRN, 512)
		pwm.setduty(PIN_BLU, 0)
	elseif mode == 2 then
		-- BLUE MODE
		pwm.setduty(PIN_RED, 0)
		pwm.setduty(PIN_GRN, 0)
		pwm.setduty(PIN_BLU, 512)
	else
		-- WHITE MODE
		pwm.setduty(PIN_RED, 512)
		pwm.setduty(PIN_GRN, 512)
		pwm.setduty(PIN_BLU, 512)
	end
end


-- LED Initialization
rgb_init(500, 512)


-- Put radio into station mode to connect to network
wifi.setmode(wifi.STATION)


-- Debug info
print('\n\nSTATION Mode:',	'mode='..wifi.getmode())
print('MAC Address: ',		wifi.sta.getmac())
print('Chip ID: ',			node.chipid())
print('Heap Size: ',		node.heap(),'\n')


-- Start the connection attempt
wifi.sta.config(WIFI_SSID, WIFI_PASS)


-- Count how many times you tried to connect to the network
local wifi_counter = 0


-- Create an alarm to poll the wifi.sta.getip() function once a second
-- If the device hasn't connected yet, blink through the LED colors. If it 
-- has, turn the LED white
tmr.alarm(0, 1000, 1, function()
	if wifi.sta.getip() == nil then
		print("Connecting to AP...\n")
		
		-- Rotate through RGB colors while waiting
		wifi_counter = wifi_counter + 1;
		rgb_solid(wifi_counter % 3)
   	else
    	ip, nm, gw = wifi.sta.getip()
      	
    	-- Debug info
      	print("\n\nIP Info: \nIP Address: ",ip)
      	print("Netmask: ",nm)
      	print("Gateway Addr: ",gw,'\n')
      	
      	tmr.stop(0)		-- Stop the polling loop
      	rgb_solid(4)	-- Turn LED white to indicate success
      	
      	-- Continue to main function after network connection
      	dofile("main.lua")
   	end
end)


--[[
main.lua
by Patrick Lloyd

This contains all the bits and pieces to connect to the broker, subscribe and
and publish to topics, run various functions that interact with GPIO, and so
on.
--]]

-- Holds dispatching keys to different topics. Serves as a makeshift callback
-- function dispatcher based on topic and message content
m_dis = {}


-- Standard counter variable. Used for modulo arithmatic
local count = 0

function animate(m, pl)
	-- Confirm that an animation message was received on the /mcu/cmd topic
	m:publish("/mcu/rgbled_status/", "--> ANIMATE COMMAND", 0, 0,
			function(m) print("ANIMATE COMMAND") end)
	
	-- Main option control structure. Pretty gross-looking but it works
	-- Option 0 turns everything off
	if pl == "0" then
		-- Confirm LED being turned off to serial terminal and MQTT broker
		m:publish("/mcu/rgbled_status/", "--> LED OFF", 0, 0,
			function(m) print("LED OFF") end)
		
		-- Reset the counter and stop the timer from another function
		count = 0
		tmr.stop(1)
		
		-- PWM process is still running but duty cycle is just set to zero
		pwm.setduty(PIN_RED, 0)
		pwm.setduty(PIN_GRN, 0)
		pwm.setduty(PIN_BLU, 0)
	
	-- RBG Mode
	elseif pl == "1" then
		-- Confirm LED in RGB mode to serial terminal and MQTT broker
		m:publish("/mcu/rgbled_status/", "--> R-G-B Mode", 0, 0,
			function(m) print("RGB Mode") end)
		
		-- Just stop the timer from another loop
		tmr.stop(1)
		
		-- Use function declared in init_man.lua to blink through red, green,
		-- and blue
		tmr.alarm(1,500,1,function()
			count = count + 1
			rgb_solid(count % 3)
		end)
	
	-- Pick a "random" color and make it breathe mode
	elseif pl == "2" then
		-- Confirm LED in random breathe mode to serial terminal and MQTT broker
		m:publish("/mcu/rgbled_status/", "--> Random-Breathe Mode", 0, 0,
			function(m) print("Random-Breathe Mode") end)
		
		-- Reset the counter and stop the timer from another function
		tmr.stop(1)
		count = 0
		
		-- Create variables run the breather alarm. Start with random color at
		-- full brightness (percent = 100)
		local percent = 100
		local count_up  = false
		local red = (tmr.now()*tmr.now()) % 512
		tmr.delay(red)
		local grn = (tmr.now()*tmr.now()) % 512
		tmr.delay(grn)
		local blu = (tmr.now()*tmr.now()) % 512

		-- Breather alarm function run every 20 ms
		tmr.alarm(1,20,1,function()
			-- Set the LED brightness
			pwm.setduty(PIN_RED, red * percent / 100)
			pwm.setduty(PIN_BLU, blu * percent / 100)
			pwm.setduty(PIN_GRN, grn * percent / 100)
			
			-- Logic to either dim or brighten
			if count_up == false then
				percent = percent - 1
				if percent < 0 then
					percent = 0
					count_up = true
				end
			else
				percent = percent + 1
				if percent > 100 then
					percent = 100
					count_up = false
				end
			end
		end)
	
	-- Lots of random blinking craziness
	elseif pl == "3" then
		-- Confirm LED in disco mode to serial terminal and MQTT broker
		m:publish("/mcu/rgbled_status/", "--> Disco Mode", 0, 0,
			function(m) print("Disco Mode") end)
		
		-- Reset the counter and stop the timer from another function
		tmr.stop(1)
		count = 0
		
		-- Crazy disco alarm every 20 ms
		tmr.alarm(1,20,1,function()
			pwm.setduty(PIN_GRN, (tmr.now()*tmr.now())%512)
			pwm.setduty(PIN_RED, (tmr.now()*tmr.now())%512)
			pwm.setduty(PIN_BLU, (tmr.now()*tmr.now())%512)
		end)
	
	-- Something went wrong somehow
	else
		-- Print and publish to serial terminal and MQTT broker respectively
		-- that something went wrong
		m:publish("/mcu/rgbled_status/", "--> Error: Unknown Command", 0, 0,
			function(m) print("ERROR: UNKNOWN COMMAND") end)
	end
end


-- As part of the dispatcher algorithm, this assigns a topic name as a key or
-- index to a particular function name
m_dis["/mcu/cmd/animate"] = animate

-- initialize mqtt client with keepalive timer of 60sec
m = mqtt.Client(MQTT_CLIENTID, 60, "", "") -- Living dangerously. No password!


-- Set up Last Will and Testament (optional)
-- Broker will publish a message with qos = 0, retain = 0, data = "offline"
-- to topic "/lwt" if client don't send keepalive packet
m:lwt("/lwt", "Oh noes! Plz! I don't wanna die!", 0, 0)


-- When client connects, print status message and subscribe to cmd topic
m:on("connect", function(m) 
	-- Serial status message
	print ("\n\n", MQTT_CLIENTID, " connected to MQTT host ", MQTT_HOST,
		" on port ", MQTT_PORT, "\n\n")

	-- Subscribe to the topic where the ESP8266 will get commands from
	m:subscribe("/mcu/cmd/#", 0,
		function(m) print("Subscribed to CMD Topic") end)
end)


-- When client disconnects, print a message and list space left on stack
m:on("offline", function(m)
	print ("\n\nDisconnected from broker")
	print("Heap: ", node.heap())
end)


-- On a publish message receive event, run the message dispatcher and
-- interpret the command
m:on("message", function(m,t,pl)
	print("PAYLOAD: ", pl)
	print("TOPIC: ", t)
	
	-- This is like client.message_callback_add() in the Paho python client.
	-- It allows different functions to be run based on the message topic
	if pl~=nil and m_dis[t] then
		m_dis[t](m,pl)
	end
end)


-- Connect to the broker
m:connect(MQTT_HOST, MQTT_PORT, 0, 1)


--[[
init.lua
by Patrick Lloyd

NodeMCU automatically runs init.lua on bootup so this acts as a launcher file
meant to load the actual init file only when everything has been tested and
debugged. 
--]]
dofile("init_man.lua")

In order to get NodeMCU to do anything, we have to transmit commands on the /mcu/cmd topic. This is done with the mqtt_commander.py script below or in (path-to-zip)/nodemcu_mqtt_rgbled/demo_code/python/mqtt_commander.py. It's a fleshed out version of the loopback test client above but adds the ability to have the user enter special commands to be published so that they can interact with the NodeMCU. It also listens to the  /mcu/rgbled_status/ topic for information about the success or failure of the commands sent to NodeMCU. You can run it by typing the following into the terminal:

<strong>python3 </strong>mqtt_commander.py # Different python version 

#!/usr/bin/env python3

'''
Simple MQTT PUB/SUB terminal client example
by Patrick Lloyd

This simple MQTT client allows the user to send messages to an ESP8266
running NodeMCU in order to interact with an RGB LED. The client ubscribes
to the "/mcu/#" topic and prints all messages from the "/mcu/rgbled_status/"
topic.

Valid inputs are only the numbers 0-3 but this can be expanded to accomodate
more complex projects.
'''

# Library to connect with the broker
# See http://www.eclipse.org/paho/ for more info
import paho.mqtt.client as mqtt

# ----- CHANGE THESE FOR YOUR SETUP -----
MQTT_HOST = ""
MQTT_PORT = 1883
# ---------------------------------------


# The callback function for when the client connects to broker
def on_connect(client, userdata, rc):
    print("\nConnected with result code " + str(rc) + "\n")

    #Subscribing in on_connect() means that if we lose the connection and
    # reconnect then subscriptions will be renewed.
    client.subscribe("/mcu/#")  # Connect to everything in /mcu topic
    print("Subscibed to /mcu/#")


# The callback function for when a message on /mcu/rgbled_status/ is published
def on_message_rgbled(client, userdata, msg):
    print("\n\t* LED UPDATED ("+msg.topic+"): " + str(msg.payload))


# Call this if input is invalid
def command_error():
    print("Error: Unknown command")


# Create an MQTT client instance
client = mqtt.Client(client_id="python-commander")

# Callback declarations (functions run based on certain messages)
client.on_connect = on_connect
client.message_callback_add("/mcu/rgbled_status/", on_message_rgbled)

# This is where the MQTT service connects and starts listening for messages
client.connect(MQTT_HOST, MQTT_PORT, 60)
client.loop_start()  # Background thread to call loop() automatically

# Main program loop
while True:

    # Get basic user input and process it
    animate_msg = input(
        "\n(0 = OFF, 1 = R-G-B, 2 = Random Breathe, 3 = Disco): ")

    # Check the input and sent it to the broker if it's valid
    if animate_msg == "0":
        client.publish("/mcu/cmd/animate", payload="0", qos=0, retain=False)
    elif animate_msg == "1":
        client.publish("/mcu/cmd/animate", payload="1", qos=0, retain=False)
    elif animate_msg == "2":
        client.publish("/mcu/cmd/animate", payload="2", qos=0, retain=False)
    elif animate_msg == "3":
        client.publish("/mcu/cmd/animate", payload="3", qos=0, retain=False)
    else:
        command_error()


The following videos demonstrate the functionality of the system:

The first one shows how the Python client (top left terminal) and the NodeMCU (right terminal) work through the Mosquitto MQTT broker (bottom left terminal). The NodeMCU is displaying debug messages through its serial console as it receives commands from the mqtt_commander Python client. You can see how the broker identifies the clients by their ID and how the NodeMCU posts a disconnect message at the end when Mosquitto is closed by pressing Ctrl+C.

The second video shows off the hardware commands. When nothing is running, the LED is blue (I'm not sure why... that is unintentional) but as soon as the init.lua file is run, the LED turns white to indicate that the pins were set up correctly. It then alternates red-green-blue while waiting to connect to the local WiFi network. Once connected it turns white again. The program written into main.lua supports four modes for the LED, each selectable by sending 0-3 through from the mqtt_commander client. They are as follows:

  • 0: Turn LED off
  • 1: Alternate red-green-blue every 500 ms
  • 2: Pick a random combination of red, green, and blue and "breathe" the LED by changing the duty cycles of the PWM controller
  • 3: Every 20 ms, pick a random color and flash it at full brightness, like you're at the discotec

Closing Remarks

Like I mentioned earlier on, an internet-connected LED that emulates a tiny disco doesn't have a lot of immediate practical value. However, the flexibility and more importantly the scalability of MQTT allows for a HUGE number of IoT applications. It can be used for weather monitors, room occupancy detectors, motion trackers, and so many more applications. In the next installment, I'm going to be using these concepts to create a color-based thermometer from an RGB LED strip and the SPI thermocouple amplifier that was designed back in this article. See you next time... same bat time, same bat channel. All the code and tools for the project can be accessed below:

nodemcu_mqtt_rgbled.zip

Give this project a try for yourself! Get the BOM.

 

Related Tutorials:

3 Comments
  • Marcel Stör September 17, 2015

    Very nice, well done! I think though you should handle the publish-if-offline case properly. If the ESP publishes to the MQTT broker after it disconnected (e.g. because 60s keep-alive was up) it’ll fail.

    Like. Reply
  • F
    fvgm January 24, 2016

    Wow. Excellent article! Simple and useful. Thanks.

    Like. Reply
  • kishanbhamre July 05, 2020

    how can we connect 2 or more nodemcu devices to single MQTT server (same username and password)?

    Like. Reply