anything4me

ESP8266 calling home

Firmware over-the-air (OTA) is great. It makes you shiver whenever you throw an update to one of your devices. The ArduinoOTA library for ESP8266 is so easy to use it’s almost magic. But once you have several devices deployed you start to think one step further.

Here I’m going to talk about two different options: writing an automated deployment script that performs OTA updates or giving your device the ability to call home querying for new updates, downloading them and flash itself into the latest version available.

A deployment script

First option is to have an automated deployment script. Something that grabs the latest stable version of your code and throws one update over the air for each device, probably providing specific options for each device, like “add DHT22 support”.

Since I use platformIO I can create custom environments for each device, like this:

[env:washer-device]
platform = espressif
framework = arduino
board = esp01_1m
lib_install = 89,64,19
topic = /home/cellar/washer/ip
build_flags = -Wl,-Tesp8266.flash.1m256.ld
build_flags = -D SONOFF -D DEBUG -D ENABLE_POWER -D ENABLE_DHT
upload_speed = 115200
upload_port = "192.168.1.114"
upload_flags = --auth=fibonacci --port 8266

It defines the platform, board, build flags or OTA parameters. But as you can see you have to declare the IP of the device. That is troublesome. Normally your device will keep its IP for a long time as long as it keeps connected or if your router leasing time is long enough for the device to claim its previous IP if it doesn’t. But what if it changes? And it will. Eventually.

You might have notice a non standard parameter in the previous environment definition: topic. A simple bash script can use this information to query your local MQTT broker for the latest (the retained) value of this topic and use this value as the IP for over-the air flashing. This is the script:

#!/bin/bash

MQTT_HOST=192.168.1.10

function help() {
  echo "Syntax: $0 "
  devices
}

function devices() {
  echo "Defined devices:"
  cat platformio.ini | grep 'device]' | sed 's/\[env:/ - /g' | sed 's/\-device]//g'
}

function valid_ip() {
  local stat=0
  rx='([1-9]?[0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])'
  if [[ $ip =~ ^$rx\.$rx\.$rx\.$rx$ ]]; then
    stat=1
  fi
  return $stat
}

# Check arguments
if [ "$#" -ne 1 ]; then
  help
  exit 1
fi
device=$1-device

# Get IP
topic=`cat platformio.ini | grep $device -A 10 | grep "topic" | cut -d' ' -f3`
if [ "$topic" == "" ]; then
  echo "Unknown device $device or topic not defined"
  devices
  exit 2
fi

ip=`mosquitto_sub -t $topic -h $MQTT_HOST -N -C 1`
if valid_ip $ip; then
  echo "Could not get a valid IP from MQTT broker"
  exit 3
fi

platformio run -vv -e $device --target upload --upload-port $ip

Now you just have to declare the environment for each device in the platformio.ini file and call the script once for each device.

This option is great. If you write your code right you can create lightweight binaries targeted for each device from the same code base. The deploy script must check if the update has been successful by monitoring the update output. But I’d also provide a way to listen to client’s “hello” messages after reboot. These messages should contain the current versions for the firmware and the file system.

This works fine for 10, maybe 15 devices. That is for a home or small office. Another requirement is that you must be in the same network the device is. What if you have to manage some tens or hundreds of devices? What if they are in different networks, buildings, towns, countries?! You will need some kind if unattended pull update process, something Windows has been criticised for a long time but, hey, we are talking about small, simple, one-task-oriented, unattended devices.

Automatic Over-The-Air Pull Updates

I’ve been testing this automatic over the air update process based on the official ESP8266httpUpdate library. The library handles the download and flashes the binary and it supports both firmware and file system binaries. There is even an example that can be used as a starting point.

I’ve added a way to discover available updates and wrapped it up in the same fashion the ArduinoOTA library does, providing a callback method to display debug messages or perform in-the-middle tasks.

The library is open source and available at the NoFUSS repository at bitbucket.

The Protocol

This first revision of the protocol is very simple. The client device does a GET request to a custom URL specifying its DEVICE and firmware VERSION this way:

GET http://myNofussServerURL/DEVICE/VERSION

For instance:

GET http://192.168.1.10/nofuss/SONOFF/0.1.0

The response is a JSON object. If there are no updates available it will be empty (that is: ‘{}’). Otherwise it will contain info about where to find the new firmware binaries:

{
'version': '0.1.1',
'firmware': '/firmware/sonoff-0.1.1.bin',
'spiffs': '/firmware/sonoff-0.1.1-spiffs.bin'
}

Binaries URLs (for the firmware and the SPIFFS file system) are relative to the server URL, so following with the example, the device will first download the SPIFFS binary from:

http://192.168.1.10/nofuss/firmware/sonoff-0.1.1-spiffs.bin

flash it and if everything went fine it will download the firmware from:

http://192.168.1.10/nofuss/firmware/sonoff-0.1.1.bin

flash it too and then restart the board.

Cool! So this is the output in the serial console:

Device : TEST
Version: 0.1.0
[NoFUSS] Start
[NoFUSS] Updating
         New version: 0.1.1
         Firmware: /firmware/test-0.1.1.bin
         File System: 
[NoFUSS] Firmware Updated
[NoFUSS] Resetting board

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x4010f000, len 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
v3ffe8408
@cp:0
ld
�

Device : TEST
Version: 0.1.1
[NoFUSS] Start
[NoFUSS] Already in the last version
[NoFUSS] End

Installing the server

The PHP server implementation depends on Slim Framework, Monolog and Akrabat IP Address Middleware. They are all set as dependencies in the composer.json file, so you just have to type

php composer.phar install

from the server folder.

Next you will have to configure your webserver to configure the URLs. If you are using Apache then all you have to do is create a new service pointing to the server/public folder. The .htaccess file there will take care of the rest. If you are using Nginx the create a new site file like this one:

server {
    listen 80 default_server;
    server_name nofuss.local;
    root /<path_to_project>/server/public/;
    try_files $uri $uri/ /index.php?$query_string;
    index index.php;
    include global/php5-fpm.conf;
}

Make sure the server has permissions to write on the logs folder, the server will store there a log with information on the devices that queried for updates, the version they are reporting and the answer they received.

Versions

The versions info is stored in the data/versions.json file. This file contains an array of objects with info about version matching and firmware files. Version matching is always “more or equal” for minimum version number and “less or equal” for maximum version number. An asterisk (*) means “any”. Device matching is “equals”.

The target key contains info about version number for the new firmware and paths to the firmware files relative to the public folder. If there is no binary for “firmware” or “spiffs” keys, just leave it empty.

[
    {
        "origin": {
            "device": "TEST",
            "min": "*",
            "max": "0.1.0"
        },
        "target": {
            "version": "0.1.1",
            "firmware": "/firmware/test-0.1.1.bin",
            "spiffs": ""
        }
    }
]

Using the client

The client library depends on Benoit Blanchon’s ArduinoJson library. In the example it is set as a dependency in the platformio.ini file, so if you use PlatformIO it will automatically download.

To use it you only have to configure the global NoFUSSClient object with proper server URL, device name and version in your setup:

NoFUSSClient.setServer(NOFUSS_SERVER);
NoFUSSClient.setDevice(DEVICE);
NoFUSSClient.setVersion(VERSION);

And then call every so often NoFUSSClient.handle() to check for updates. You can also monitor the update flow providing a callback function to the onMessage method. Check the basic.cpp example for a real usage.

Wrap up

I wanted this library it to be easy to use and lightweight so I can still do OTA on 1Mb devices. The wrapper itself adds very little to the binary size but the JSON support certainty plays against this. Will give a try to plain text output.

As I said the code is free source and available at the NoFUSS repository at bitbucket. The project is currently in beta status, it works but I’m not 100% confident and I have doubts about the API of the service so any suggestions will be welcome.

CC BY-SA 4.0 ESP8266 calling home by Tinkerman is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

6 thoughts on “ESP8266 calling home

  1. jd

    hey dude i want to ask you a technical question. email me [removed]

    i just need a little bit of guidance on how to connect either a bus pirate or logic analyzer to an interface on programmable scanner to make a TTL cable thats no longer manufactured.

    I dont know how to send you pix or get a reply so im asking you here.. thanks.. love your articles

    Reply
  2. Vamshi Chinthakindi

    Hi Xose,

    Thanks for the nice post on Automatic over the air update.

    At this moment I am engaged in developing the ESP8266 module which measures temperature and humidity.

    Concerning Over The Air (OTA) update , I need to do it using HTTP Server such that ESP8266 automatically checks for updates on the server and automatically updates itself.

    But I did not find much resources in detailed on the internet. I found the code at the NoFuss repository bitbucket which you have been recommended in the post.

    But I cannot be able to understand the control flow of the program code and theoretical aspects of the code.

    Could you help me regarding this?

    Regards,
    Vamshi

    Reply
    1. Xose Pérez Post author

      The solutions has two parts: a webserver and a client library for the ESP8266.

      You can find an implementation of the webserver in the server folder of the repo. It is a PHP app. Instructions on how to install it are in the README file in the root of the repo. This webservice listens to requests and checks against a JSON file if there is a firmware update available. At the moment you will have to modify that JSON file manually (there is no backoffice). An example of that file is server/data/version.json. for each entry of that file the “origin” defines the device and version that the requester provides and the “target” defines the available update files (both firmware and file system image).

      From the client point of view you just have to include the library definitions and in you setup initialize the values, then call the handle method as often as you want to check if there are available updates.


      #include "NoFUSSClient.h"
      ...
      void setup() {
      ...
      NoFUSSClient.setServer("http://nofussserver/");
      NoFUSSClient.setDevice("MYSENSOR");
      NoFUSSClient.setVersion("1.0.0");
      ...
      }

      void loop() {
      ...
      (every X minutes) NoFUSSClient.handle();
      ...
      }

      The “handle” method performs an GET request against the server URL provided by the “setServer” method, passing by the device and version provided in the “setDevice” and “setVersion” methods in the setup(). Then it parses the response and downloads and flashes the filesystem image (if any) and the firmware image (if any) before rebooting the device.

      To troubleshoot the installation you should test the server first to see if it’s working and the responses are valid. For any given requests the server should response with the “target” content of the first match in the versions.json file. Also test that the paths to the images are valid (they are relative to the URL of the service).

      If you need more info or detail feel free to contact me by email.

      Reply

Leave a Reply (all comments are moderated, be patient)