Embed Your Website in Your ESP8266 Firmware Image

A few months ago I wrote about the process I was using to optimize my website files for SPIFFS prior to upload them to the ESP8266. The goal was to reduce the number and size of the files to help the microcontroller to cope with them in an easier way. Smaller size mean faster downloads and less files mean less concurrency.

The process is done using Gulp, a tool to automate processes, and defined in a javascript file. It certainly requieres some initial setup (install node.js and the file dependencies) but it's pretty straight forward.

I first used this process in my ESPurna firmware but since then I have used it in every ESP8266 project with web interface. And of course I have sometimes hit problems and found new solutions. This post is an update on the original one, dealing with different problems I have faced, like having multiple entry points (more than one HTML) or inlining images.

From the begining

Merging HTML and Javascript or CSS is conceptually easy to understand. After all, inline javascript or CSS is something we have all done at the begining, before someone told us that was wrong, you should clearly separate your view from your logic.

But a small microcontroller like the ESP8266 might not like having different files to serve. Modern browsers tend to improve speed by opening several requests at the same time and the ESP8266 might struggle with 2 o 3 concurrent requests.

But this is my html folder in ESPurna:

[16:18:04] xose@canopus:~/workspace/espurna-firmware/current/code/html
$ tree
.
├── checkboxes.css
├── checkboxes.js
├── custom.css
├── custom.js
├── favicon.ico
├── grids-responsive-min.css
├── images
│   ├── border-off.png
│   ├── border-on.png
│   ├── handle-center.png
│   ├── handle-left.png
│   ├── handle-right.png
│   ├── label-off.png
│   └── label-on.png
├── index.html
├── jquery-1.12.3.min.js
├── jquery.wheelcolorpicker-3.0.2.min.js
├── pure-min.css
├── side-menu.css
└── wheelcolorpicker.css

1 directory, 19 files

[16:19:53] xose@canopus:~/workspace/espurna-firmware/current/code/html
$ du -chs
296K	.
296K	total

One HTML file, 4 javascript files, 6 CSS files and 8 images, including an icon. Wow, that's 19 files and 296Kb… How to convert this into a single file? In my previous post about the subject I explained how I was merging all the “text” files (HTML, JS and CSS) into a single file, but back then I had no option for the images, aside from joining them into a sprite. Now I have an easy way (sprites are hard to manage) and also I've been testing a way to dramatically improve speed and reduce overall size at the same time.

Embed images

Sprites are hard to manage. Of course there are way to automate the creation of the sprite file and replace the references in CSS to use the unified file. But still you end up with one extra file.

A more convenient way to work with images is to embed them in the HTML, the same way we do with CSS and JS files. Basically you have to get your image content and convert it to a char string in base64, then you can just copy it to your HTML or CSS file this way:

div.image {
  width: 60px;
  height: 4px;
  background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADIA...);
}

Embedding the image into the HTML or the CSS efectively reduces the number of files in your project. You might complain it also increases the size of it but mind that we will be gzipping everything at the end so there will be no big difference in size. You can also state that the browser requieres more processing power to render an embedded image. Don't know if that is true, but even so I certainly prefer to move the work load to the more capable device, and your computer is certainly way more capable than the ESP8266.

Multiple entry points

At some point in the project I had two different static entry points to my website: index.html and password.html. The later is the page the controller sends when the password is still the default one for force the user to change it. My first approach was to create two different compressed files:

index.html.gz
password.html.gz

But since the password.html file shared most of the styles and some functionality with the index.html there were lots of dupped code between the two files. I then tried to dedup all that code into different files (still minified, compressed and with embedded images):

index.html.gz
password.html.gz
style.css.gz
script.js.gz

But I was loosing part of the original motivation of the project. Now the microcontroller was receiving 3 single requests for every page. So finally I decided to merge the functionality of the password file into the index.html and manage the visibility of the different parts in code.

Create a byte array in PROGMEM

So there I was. A single file with all HTML, JS, CSS and images embedded into it, cleaned, minified and gzipped. A total of less than 60Kb. But there is one thing better than just one file: no files at all. The idea came from reading the documentation of the ESPAsyncWebServer library by me-no-dev I use in ESPurna, specifically the section that shows how to send a binary content from PROGMEM. What if…?

[07:47:11] xose@canopus:~/workspace/espurna-firmware/current/code
$ cat espurna/static/index.html.gz.h 
#define index_html_gz_len 59956
const uint8_t index_html_gz[] PROGMEM = {
0x1f,0x8b,0x08,0x00,0x00,0x00,0x00,0x00,0x00,0x03,0xec,0xbd,0xe9,0x7a,
0xea,0x48,0x12,0x20,0xfa,0x2a,0x98,0xea,0x76,0xa3,0x46,0x60,0x16,0x63,
...
};

So I added to my gulp build script the required code to automatically generate a header file with the contents of the index.html.gz as a binary array in program memory. This has a few very interesting consequences:

Unless your program has other requirements, you can forget about the SPIFFS image. No need to flash your board twice with the firmware and the SPIFFS image. I can tell you that the #1 problem with ESPurna was that the web returned a 404. Reason: you forgot to flash the SPIFFS image. Well, no more.
Since you don't need SPIFFS anymore, that code is not added to your firmware image. As a result the overall image size is smaller. For the current ESPurna build the firmware without the webpage but with SPIFFS support uses 390400 bytes and the gzipped web content 59956 bytes, that is 450356 bytes. The firmware with the webpage embedded but without SPIFFS support uses only 423632 bytes.
Even thou your webcontent is 60Kb the uploader generates and uploads an image the size of your SPIFFS partition (partition is the word I use to refer to each section in the memory layout_)_. So if that _partition_ is 3Mb it will upload 3Mb of data to the device. And that takes quite some time to flash. So you actually **deploy a lot faster**.
Even thou you can create custom memory layouts, the predefined ones are 64Kb, 128Kb, 256Kb, etc. Due to the way the mkspiffs tool works (the one that creates to SPIFFS firmware image) a 60Kb file does not fit into a 64Kb partition. You actually need a 128Kb partition. But that leaves the 512b boards out since the max predefined SPIFFS size for them is 64Kb. Embedding the web content in the firmware allows you to remove the SPIFFS section and use the full 512Kb for code and EEPROM. That means you can have the full web site on a 512Kb flash size and you still have 74Kb free for more code!
Same applies for OTA on bigger flash sizes. For the common 1Mb devices (Sonoffs) if you define you layout to not use SPIFFS at all (not standard) you can have up to around 500Kb of room for an OTA-enabled firmware. So you have more room for OTA updates. Your code size can be larger and you will still be able to do OTA.
And since you don't need to fetch the file using SPIFFS but just dump the raw contents from a memory location the embedded web server serves the requests faster.

Doing it

Doing it is much simpler that it may look like. Gulp has a ton of different modules meant to preprocess HTML files.

Gulp script

A Gulp script is a series of task with dependencies. When you call “gulp” without options it will run de “default” task defined in the “gulpfile.js” file. You can also specify the script file or directly call any of the tasks. Each task can have a list of dependencies (task to call before itself) and a definition (a function). Dependencies are “prepended” so in the code bellow when you call the default task it actually runs (in order): clean, build_inline and build_embedded. “Clean” is pretty obvious: remove contents from the “espurna/static” folder.

The build_inline task is responsible for creating a unified and compressed file. Reading through the code you will see that it:

Embeds the favicon in the HTML
Clean CSS files
“Uglifies” (compresses) JS files
Inlines images, CSS and JS files in the HTML
Cleans and minifies the HTML (including the now inlined styles and scripts)
Compresses the file into a single index.html.gz file

This is more or less what I was already doing, except for the images. But now the output of the build_inline task is processed by the build_embedded task. And you can see it's plain javascript. It creates a file which defines a index_html_gz_len with the size of the contents and then creates a byte array in progmem with them. The final file (index.html.gz.h) is pretty big (5 bytes to represent every byte in the array) but it will be compressed when building the firmware image.

First you will need the required components. The file below defines the dependencies. Simply save it as package.json and run “npm -i” and it will downloadlocally or the required components.

{
  "name": "esp8266-filesystem-builder",
  "version": "0.2.0",
  "description": "Gulp based build system for ESP8266 file system files",
  "main": "gulpfile.js",
  "author": "Xose Pérez <xose.perez@gmail.com>",
  "license": "GPL-3.0",
  "devDependencies": {
    "del": "^2.2.1",
    "gulp": "^3.9.1",
    "gulp-base64-favicon": "^1.0.2",
    "gulp-clean-css": "^3.4.2",
    "gulp-css-base64": "^1.3.4",
    "gulp-gzip": "^1.4.0",
    "gulp-htmlmin": "^2.0.0",
    "gulp-inline": "^0.1.1",
    "gulp-uglify": "^1.5.3"
  },
  "dependencies": {}
}

Let's now move to the gulp script.

/*

ESP8266 file system builder

Copyright (C) 2016-2017 by Xose Pérez <xose dot perez at gmail dot com>;

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

*/

// -----------------------------------------------------------------------------
// File system builder
// -----------------------------------------------------------------------------

const fs = require('fs');
const gulp = require('gulp');
const htmlmin = require('gulp-htmlmin');
const cleancss = require('gulp-clean-css');
const uglify = require('gulp-uglify');
const gzip = require('gulp-gzip');
const del = require('del');
const inline = require('gulp-inline');
const inlineImages = require('gulp-css-base64');
const favicon = require('gulp-base64-favicon');

const dataFolder = 'espurna/static/';

gulp.task('clean', function() {
    del([ dataFolder + '*']);
    return true;
});

gulp.task('buildfs_embeded', ['buildfs_inline'], function() {

    var source = dataFolder + 'index.html.gz';
    var destination = dataFolder + 'index.html.gz.h';

    var wstream = fs.createWriteStream(destination);
    wstream.on('error', function (err) {
        console.log(err);
    });

    var data = fs.readFileSync(source);

    wstream.write('#define index_html_gz_len ' + data.length + '\n');
    wstream.write('const uint8_t index_html_gz[] PROGMEM = {')

    for (i=0; i<data.length; i++) {
        if (i % 1000 == 0) wstream.write("\n");
        wstream.write('0x' + ('00' + data[i].toString(16)).slice(-2));
        if (i<data.length-1) wstream.write(',');
    }

    wstream.write('\n};')
    wstream.end();

    del([source]);

});

gulp.task('buildfs_inline', ['clean'], function() {
    return gulp.src('html/*.html')
        .pipe(favicon())
        .pipe(inline({
            base: 'html/',
            js: uglify,
            css: [cleancss, inlineImages],
            disabledTypes: ['svg', 'img']
        }))
        .pipe(htmlmin({
            collapseWhitespace: true,
            removecomments: true
aside: true,
            minifyCSS: true,
            minifyJS: true
        }))
        .pipe(gzip())
        .pipe(gulp.dest(dataFolder));
})

gulp.task('default', ['buildfs_embeded']);

ESPAsyncWebserver code

The final step is to add code to respond to HTTP requests with the contents in PROGMEM. The ESPAsyncWebServer library comes with built-in support to do that. I'm not going to post a full working example. Instead I will copy here the basic bits to get it working with the output of the gulp script.


// Include the libraries
#include <ESP8266WiFi.h>
#include <ESPAsyncWebServer.h>

// Instantiate the webserver object
AsyncWebServer server = AsyncWebServer(80);

// Variable to hold the last modification datetime
char last_modified[50];

// Include the header file we create with gulp
#include "static/index.html.gz.h"

// Callback for the index page
void onHome(AsyncWebServerRequest *request) {

    // Check if the client already has the same version and respond with a 304 (Not modified)
    if (request->header("If-Modified-Since").equals(last_modified)) {
        request->send(304);

    } else {

        // Dump the byte array in PROGMEM with a 200 HTTP code (OK)
        AsyncWebServerResponse *response = request->beginResponse_P(200, "text/html", index_html_gz, index_html_gz_len);

        // Tell the browswer the contemnt is Gzipped
        response->addHeader("Content-Encoding", "gzip");

        // And set the last-modified datetime so we can check if we need to send it again next time or not
        response->addHeader("Last-Modified", last_modified);

        request->send(response);

    }
}

// Nothing in the loop (it's async!!)
void loop() {}

// We configure everything in the setup
void setup() {

  // Populate the last modification date based on build datetime
  sprintf(last_modified, "%s %s GMT", __DATE__, __TIME__);

  // Configure you wifi access
  WiFi.mode(WIFI_STA);
  WiFi.begin("...", "...");
  while (WiFi.status() != WL_CONNECTED) delay(1);

  // Configure the webserver
  server.rewrite("/", "/index.html");
  server.on("/index.html", HTTP_GET, onHome);
  server.onNotFound([](AsyncWebServerRequest *request){ request->send(404); });
  server.begin();

}

As you can see the server works by calling back functions based on the request URL and type. This is a common paradigm in async languages. The code also checks the Last-Modified header and returns a 304 (Not Modified) if it's the same datetime the client already has, this of course reduces the payload from 60Kb to a few bytes for most request (all but the first one).

Conclusion

I'm using this technique in all my ESP8266 based projects that require a web interface. The overall image size is smaller and it's faster both to flash and to execute. The only drawback I faced is that you will want to have just on “static” entry point to your web contents.

"Embed your website in your ESP8266 firmware image" was first posted on 21 July 2017 by Xose Pérez on tinkerman.cat under Code, Learning, Tutorial and tagged css, esp8266, espurna, firmware, gulp, html, javascript, optimize, progmem, spiffs.

Embed your website in your ESP8266 firmware image