ESP8266 Calling Home

Firmware over-the-air (OTA) is great. It makes you shiver whenever you throw an update to one of your devices. The ArduinoOTA library for ESP8266 is so easy to use it's almost magic. But once you have several devices deployed you start to think one step further.

Here I'm going to talk about two different options: writing an automated deployment script that performs OTA updates or giving your device the ability to call home querying for new updates, downloading them and flash itself into the latest version available.

A deployment script

First option is to have an automated deployment script. Something that grabs the latest stable version of your code and throws one update over the air for each device, probably providing specific options for each device, like “add DHT22 support”.

Since I use platformIO I can create custom environments for each device, like this:

[env:washer-device]
platform = espressif
framework = arduino
board = esp01_1m
lib_install = 89,64,19
topic = /home/cellar/washer/ip
build_flags = -Wl,-Tesp8266.flash.1m256.ld
build_flags = -D SONOFF -D DEBUG -D ENABLE_POWER -D ENABLE_DHT
upload_speed = 115200
upload_port = "192.168.1.114"
upload_flags = --auth=fibonacci --port 8266

It defines the platform, board, build flags or OTA parameters. But as you can see you have to declare the IP of the device. That is troublesome. Normally your device will keep its IP for a long time as long as it keeps connected or if your router leasing time is long enough for the device to claim its previous IP if it doesn't. But what if it changes? And it will. Eventually.

You might have notice a non standard parameter in the previous environment definition: topic. A simple bash script can use this information to query your local MQTT broker for the latest (the retained) value of this topic and use this value as the IP for over-the air flashing. This is the script:

#!/bin/bash

MQTT_HOST=192.168.1.10

function help() {
  echo "Syntax: $0 "
  devices
}

function devices() {
  echo "Defined devices:"
  cat platformio.ini | grep 'device]' | sed 's/\[env:/ - /g' | sed 's/\-device]//g'
}

function valid_ip() {
  local stat=0
  rx='([1-9]?[0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])'
  if [[ $ip =~ ^$rx\.$rx\.$rx\.$rx$ ]]; then
    stat=1
  fi
  return $stat
}

# Check arguments
if [ "$#" -ne 1 ]; then
  help
  exit 1
fi
device=$1-device

# Get IP
topic=`cat platformio.ini | grep $device -A 10 | grep "topic" | cut -d' ' -f3`
if [ "$topic" == "" ]; then
  echo "Unknown device $device or topic not defined"
  devices
  exit 2
fi

ip=`mosquitto_sub -t $topic -h $MQTT_HOST -N -C 1`
if valid_ip $ip; then
  echo "Could not get a valid IP from MQTT broker"
  exit 3
fi

platformio run -vv -e $device --target upload --upload-port $ip

Now you just have to declare the environment for each device in the platformio.ini file and call the script once for each device.

This option is great. If you write your code right you can create lightweight binaries targeted for each device from the same code base. The deploy script must check if the update has been successful by monitoring the update output. But I'd also provide a way to listen to client's “hello” messages after reboot. These messages should contain the current versions for the firmware and the file system.

This works fine for 10, maybe 15 devices. That is for a home or small office. Another requirement is that you must be in the same network the device is. What if you have to manage some tens or hundreds of devices? What if they are in different networks, buildings, towns, countries?! You will need some kind if unattended pull update process, something Windows has been criticised for a long time but, hey, we are talking about small, simple, one-task-oriented, unattended devices.

Automatic Over-The-Air Pull Updates

I've been testing this automatic over the air update process based on the official ESP8266httpUpdate library. The library handles the download and flashes the binary and it supports both firmware and file system binaries. There is even an example that can be used as a starting point.

I've added a way to discover available updates and wrapped it up in the same fashion the ArduinoOTA library does, providing a callback method to display debug messages or perform in-the-middle tasks.

The library is open source and available at the NoFUSS repository at bitbucket.

The Protocol

This first revision of the protocol is very simple. The client device does a GET request to a custom URL specifying its DEVICE and firmware VERSION this way:

GET http://myNofussServerURL/DEVICE/VERSION

For instance:

GET http://192.168.1.10/nofuss/SONOFF/0.1.0

The response is a JSON object. If there are no updates available it will be empty (that is: ‘{}'). Otherwise it will contain info about where to find the new firmware binaries:

{
'version': '0.1.1',
'firmware': '/firmware/sonoff-0.1.1.bin',
'spiffs': '/firmware/sonoff-0.1.1-spiffs.bin'
}

Binaries URLs (for the firmware and the SPIFFS file system) are relative to the server URL, so following with the example, the device will first download the SPIFFS binary from:

http://192.168.1.10/nofuss/firmware/sonoff-0.1.1-spiffs.bin

flash it and if everything went fine it will download the firmware from:

http://192.168.1.10/nofuss/firmware/sonoff-0.1.1.bin

flash it too and then restart the board.

Cool! So this is the output in the serial console:

Device : TEST
Version: 0.1.0
[NoFUSS] Start
[NoFUSS] Updating
         New version: 0.1.1
         Firmware: /firmware/test-0.1.1.bin
         File System: 
[NoFUSS] Firmware Updated
[NoFUSS] Resetting board

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x4010f000, len 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
v3ffe8408
@cp:0
ld
�

Device : TEST
Version: 0.1.1
[NoFUSS] Start
[NoFUSS] Already in the last version
[NoFUSS] End

Installing the server

The PHP server implementation depends on Slim Framework, Monolog and Akrabat IP Address Middleware. They are all set as dependencies in the composer.json file, so you just have to type

php composer.phar install

from the server folder.

Next you will have to configure your webserver to configure the URLs. If you are using Apache then all you have to do is create a new service pointing to the server/public folder. The .htaccess file there will take care of the rest. If you are using Nginx the create a new site file like this one:

server {
    listen 80 default_server;
    server_name nofuss.local;
    root /<path_to_project>/server/public/;
    try_files $uri $uri/ /index.php?$query_string;
    index index.php;
    include global/php5-fpm.conf;
}

Make sure the server has permissions to write on the logs folder, the server will store there a log with information on the devices that queried for updates, the version they are reporting and the answer they received.

Versions

The versions info is stored in the data/versions.json file. This file contains an array of objects with info about version matching and firmware files. Version matching is always “more or equal” for minimum version number and “less or equal” for maximum version number. An asterisk (*) means “any”. Device matching is “equals”.

The target key contains info about version number for the new firmware and paths to the firmware files relative to the public folder. If there is no binary for “firmware” or “spiffs” keys, just leave it empty.

[
    {
        "origin": {
            "device": "TEST",
            "min": "*",
            "max": "0.1.0"
        },
        "target": {
            "version": "0.1.1",
            "firmware": "/firmware/test-0.1.1.bin",
            "spiffs": ""
        }
    }
]

Using the client

The client library depends on Benoit Blanchon's ArduinoJson library. In the example it is set as a dependency in the platformio.ini file, so if you use PlatformIO it will automatically download.

To use it you only have to configure the global NoFUSSClient object with proper server URL, device name and version in your setup:

NoFUSSClient.setServer(NOFUSS_SERVER);
NoFUSSClient.setDevice(DEVICE);
NoFUSSClient.setVersion(VERSION);

And then call every so often NoFUSSClient.handle() to check for updates. You can also monitor the update flow providing a callback function to the onMessage method. Check the basic.cpp example for a real usage.

Wrap up

I wanted this library it to be easy to use and lightweight so I can still do OTA on 1Mb devices. The wrapper itself adds very little to the binary size but the JSON support certainty plays against this. Will give a try to plain text output.

As I said the code is free source and available at the NoFUSS repository at bitbucket. The project is currently in beta status, it works but I'm not 100% confident and I have doubts about the API of the service so any suggestions will be welcome.

"ESP8266 calling home" was first posted on 10 August 2016 by Xose Pérez on tinkerman.cat under Code, Projects and tagged bash, composer, deployment, esp8266, firmware, fota, monolog, mqtt, nginx, nofuss, ota, php, platformio, server, slim, update.

ESP8266 calling home