The ESP8266 flash layout defines a series of blocks of memory for each “partition”. There is a block for the user code (the “sketch”), there is a block for the OTA update file, another one for the emulated EEPROM, another for the WIFI configuration and one for the File System.

This last one uses Peter Andersson's SPIFFS (SPI Flash File System) code to store files in a similar fashion our computers do, but taking into account the special requirements of an embedded system and a flash memory chip.

This is great because we can store a whole static website there (html, css, js, images,…) and use the official WebServer library that comes with the Arduino Core for ESP8266 project to serve files and execute server side code that updates our static site via AJAX or WebSockets, for instance.

But the ESP8266 is nothing more than a (powerful) microcontroller and the WebServer library has its limitations and if you start to work on a complex website, with multiple files (stylesheets, scripts,…) it will soon fail…

image
Size is not that important, but the number of files is. Too many files lead to failed downloads and long rendering times...

The solution is merging. All javascript files together, all stylesheet files together. But what if you are using third party files, some of them minified? What if you want to keep them separated so you can easily swap from one component to another by just removing a line and adding a new one to your HTML file? You will need some way to process your HTML to get the scripts and stylesheets you are using, merge them and replace the references in the HTML with the unified file.

Well, that's what most web developers do on a daily basis. They work on a development version and they “process it” to get working code (clean, minimized, renderable). They even have “watchers” that work in the background translating every modification they do on the development version in real time. To do that they use compilers (for SASS, LESS, CoffeeScript,…) and build managers. One of such build managers is Gulp.

So why not use Gulp to process our website before uploading it? Why? So let's do it.

Installing Gulp

Gulp is actually a Node.js module which is great since Node.js is a really powerful frontend and scripting language and the building file that we will code can benefit from all that power. Installing Node.js and npm (Node.js packet manager) is out of scope here so visit Node.js download page and help yourself.

Next we will need a file to define all the module dependencies we are going to use. These modules are gulp itself and some gulp plugins that will take care of the different tasks (minimizing, injecting, cleaning,…). Lets take a look at it:

{
  "name": "esp8266-filesystem-builder",
  "version": "0.1.0",
  "description": "Gulp based build script for ESP8266 file system files",
  "main": "gulpfile.js",
  "author": "Xose Pérez <xose.perez@gmail.com>",
  "license": "MIT",
  "devDependencies": {
    "del": "^2.2.1",
    "gulp": "^3.9.1",
    "gulp-clean-css": "^2.0.10",
    "gulp-gzip": "^1.4.0",
    "gulp-htmlmin": "^2.0.0",
    "gulp-if": "^2.0.1",
    "gulp-inline": "^0.1.1",
    "gulp-plumber": "^1.1.0",
    "gulp-uglify": "^1.5.3",
    "gulp-useref": "^3.1.2",
    "yargs": "^5.0.0"
  },
  "dependencies": {}
}

As you can see it is a JSON document with some header fields and a list of dependencies. Save it in your code folder as “package.json” and run:

npm install

It will fetch and install all those modules in a node_modules subfolder (I suggest you to add this folder to your .gitignore file). Good, let's move ahead.

The builder script

I am going to start by showing the script:

/*

ESP8266 file system builder with PlatformIO support

Copyright (C) 2016 by Xose Pérez <xose dot perez at gmail dot com>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

*/

// -----------------------------------------------------------------------------
// File system builder
// -----------------------------------------------------------------------------

const gulp = require('gulp');
const plumber = require('gulp-plumber');
const htmlmin = require('gulp-htmlmin');
const cleancss = require('gulp-clean-css');
const uglify = require('gulp-uglify');
const gzip = require('gulp-gzip');
const del = require('del');
const useref = require('gulp-useref');
const gulpif = require('gulp-if');
const inline = require('gulp-inline');

/* Clean destination folder */
gulp.task('clean', function() {
    return del(['data/*']);
});

/* Copy static files */
gulp.task('files', function() {
    return gulp.src([
            'html/**/*.{jpg,jpeg,png,ico,gif}',
            'html/fsversion'
        ])
        .pipe(gulp.dest('data/'));
});

/* Process HTML, CSS, JS  --- INLINE --- */
gulp.task('inline', function() {
    return gulp.src('html/*.html')
        .pipe(inline({
            base: 'html/',
            js: uglify,
            css: cleancss,
            disabledTypes: ['svg', 'img']
        }))
        .pipe(htmlmin({
            collapseWhitespace: true,
            removecomments: true
aside: true,
            minifyCSS: true,
            minifyJS: true
        }))
        .pipe(gzip())
        .pipe(gulp.dest('data'));
})

/* Process HTML, CSS, JS */
gulp.task('html', function() {
    return gulp.src('html/*.html')
        .pipe(useref())
        .pipe(plumber())
        .pipe(gulpif('*.css', cleancss()))
        .pipe(gulpif('*.js', uglify()))
        .pipe(gulpif('*.html', htmlmin({
            collapseWhitespace: true,
            removecomments: true
aside: true,
            minifyCSS: true,
            minifyJS: true
        })))
        .pipe(gzip())
        .pipe(gulp.dest('data'));
});

/* Build file system */
gulp.task('buildfs', ['clean', 'files', 'html']);
gulp.task('buildfs2', ['clean', 'files', 'inline']);
gulp.task('default', ['buildfs']);

// -----------------------------------------------------------------------------
// PlatformIO support
// -----------------------------------------------------------------------------

const spawn = require('child_process').spawn;
const argv = require('yargs').argv;

var platformio = function(target) {
    var args = ['run'];
    if ("e" in argv) { args.push('-e'); args.push(argv.e); }
    if ("p" in argv) { args.push('--upload-port'); args.push(argv.p); }
    if (target) { args.push('-t'); args.push(target); }
    const cmd = spawn('platformio', args);
    cmd.stdout.on('data', function(data) { console.log(data.toString().trim()); });
    cmd.stderr.on('data', function(data) { console.log(data.toString().trim()); });
}

gulp.task('uploadfs', ['buildfs'], function() { platformio('uploadfs'); });
gulp.task('upload', function() { platformio('upload'); });
gulp.task('run', function() { platformio(false); });

Let's focus on the “File system builder”. There are 5 tasks:

  • clean will delete all the contents of the data folder
  • files will copy all images to the data folder (right now we are not processing them)
  • html processes the HTML files (more on this later)
  • buildfs calls the prior 3 to build the project
  • default is just a convenient task to call buildfs if no task provided.

The “html” task looks for all the *.html files in the development folder and reads them to extract references to *.js and *.css files (that's what the “useref” plugin does). Then it generates 3 streams. The first one gets all the *.js files, merges them and minifies them (that's what the “uglify” plugin does). Another one merges all the *.css files and minifies them (that's what the “cleancss” plugin does). An the final one replaces all the references to the *.js and *.css files with the newly generated unified files in the HTML, and minifies it (that's what the “htmlmin” plugin does). Finally all the resulting files get gzipped and stored in the data folder, along with all the images.

The good thing about this builder script is that you don't have to worry about the files you add or delete from your HTML file. It will just read them and merge them in the same order they are in your code. That's very important since often there are dependencies between files (css precedence, scripts that require jQuery to be defined,…).

Injecting code

The “useref” plugin requires some metadata in the HTML file to know where it should read the files from and inject the new code. A typical file would look like this:

<!DOCTYPE html>
<html>
   <head>

       <title>WEB CONFIG</title>
       <meta charset="utf-8" />
       <link rel="shortcut icon" type="image/x-icon" href="favicon.ico" />
       <meta name="viewport" content="width=device-width, initial-scale=1.0">

       <!-- build:css style.css -->
       <link rel="stylesheet" href="pure-min.css" />
       <link rel="stylesheet" href="side-menu.css" />
       <link rel="stylesheet" href="grids-responsive-min.css" />
       <link rel="stylesheet" href="custom.css" />
       <!-- endbuild -->

   </head>

   <body>

...

    </body>

    <!-- build:js script.js -->
    <script src="jquery-1.12.3.min.js"></script>
    <script src="custom.js"></script>
    <!-- endbuild -->

</html>

As you can see the places where the plugin should read and inject new code are defined by “build” comments like this one: “". You have to define the type of files it contains (css) and the name of the output merged file (style.css). All the code from this opening comment to the "” closing comment will be replaced by:

<link href="style.css" rel="stylesheet"></link>

Same for the javascript block.  Now, instead of naming your file system folder “data” like the ESP8266FS documentation suggests, name it “html” (or whatever other name, but the code above requires it to be “html”). To run the builder just type from the command line in your code folder:

gulp

If everything goes OK your data folder will have 3 files at least (not counting images): index.html.gz, style.css.gz and script.js.gz.

Inline JS and CSS?

[UPDATE 2016-09-05]

I've added the option to get the CSS and JS files and inject them inline in the HTML. This way you end up having just one HTML file, albeit a big one. My tests show that the overall performance is around a 15% faster, both downloading the code (the compressed index.html.gz file is 50 bytes lighter in my test project and the browser only downloads one file instead of 3) and rendering it.

If you had already copied the previous code chunks you will have to update them, both the package.json and the gulpfile.js have modifications, then remember to “npm install” to add new dependencies. Then you can use this “inline” option by calling:

gulp buildfs2

Anyway I still don't know which is better, since I think the split version can benefit from caching, which is the next step in the investigation…

GZIPping?

Compressing the files makes them lighter and that means you can fit more code in the same room and it will download faster too. But how does the ESP8266 handle it? Well, this code is not mine and it's pretty standard when using the WebServer library from the Arduino Core for ESP8266, let's see what it does:

bool handleFileRead(String path) {

    if (path.endsWith("/")) path += "index.html";
    String contentType = getContentType(path);
    String pathWithGz = path + ".gz";
    if (SPIFFS.exists(pathWithGz)) path = pathWithGz;

    if (SPIFFS.exists(path)) {
        File file = SPIFFS.open(path, "r");
        size_t sent = server.streamFile(file, contentType);
        size_t contentLength = file.size();
        file.close();
        return true;
    }

    return false;

}

As you can see the code gets the requested path, translates “/” to “index.html” for the home and grabs the content type from the file extension (not in the code, somewhere else, but that's what it does). Then it checks if there is a file with the same name plus “.gz”, and if it exists that's the file it will read. The code is completely agnostic to the contents of the file, it just streams it. But then, deep in the guts of the WebServer library we find this:

template<typename T> size_t streamFile(T &amp;file, const String&amp; contentType){
  setContentLength(file.size());
  if (String(file.name()).endsWith(".gz") &amp;&amp;
      contentType != "application/x-gzip" &amp;&amp;
      contentType != "application/octet-stream"){
    sendHeader("Content-Encoding", "gzip");
  }
  send(200, contentType, "");
  return _currentClient.write(file);
}

So, if the file we are reading ends with “.gz” and we are not serving a gzip file or a binary file, then we are sending something else compressed and hence we should set the Content-Encoding header to “gzip”. Voilà. Our browser will receive the file and it will know it has to decompress it prior to render it.

See the difference?

image
No erros, less files, less download size, waaaaay faster

What about the images?

The builder script just copies the images at the moment. There is room for improvement here: compressing them, creating sprites,…

PlatformIO

The builder script above has 3 more tasks to use them with PlatformIO. The main reason for this is that it's convenient to run “gulp buildfs” (or “gulp” since the default tasks is “buildfs”) before uploading the file system. I'm not good at remembering such things and end up wasting time wondering why I can't see my changes. So instead of running “gulp” and then do and “uploadfs” from platformio I just do:

gulp uploadfs -e ota -p 192.168.1.113

And my changes are built and flashed via OTA to the device at 192.168.1.113. Neat!

[UPDATE 2016-09-05]

As Ivan Kravets, CTO at PlatformIO, commented below, I could use the “extra_script” feature to call a python script where I can define callbacks or hooks for certain actions, for instance to call my “gulp buildfs” before uploading the file system to the board. The code he added in his comment is a good example of the feature, but I didn't work for me. Apparently the pre-hook for “uploadfs” is actually called after calling “mkspiffs”. A simple test with a unexistant “data” folder throws an error, but if it had called the gulp task there would be a data folder.

Nevertheless I found a way around the problem when I read in the documentation that they are using filenames as targets for the AddPreAction method. I tested with the spiffs.bin file and it worked! Here's the python script:

#!/bin/python

from SCons.Script import DefaultEnvironment
env = DefaultEnvironment()

def before_build_spiffs(source, target, env):
    env.Execute("gulp buildfs")

env.AddPreAction(".pioenvs/%s/spiffs.bin" % env['PIOENV'], before_build_spiffs)

Now you just have to add “extra_script = pio_hooks.py” (or the name of your script) to all the environments you want in your platformio.ini file and you are good to go, no need to call gulp, just do as usual:

pio run -e node -t uploadfs

Even neater!

"Optimizing files for SPIFFS with Gulp" was first posted on 02 September 2016 by Xose Pérez on tinkerman.cat under Code, Tutorial and tagged build, css, esp8266, gulp, gzip, html, javascript, node.js, npm, platformio, spiffs, stylesheet.