spiffs-ko-header

Optimizing files for SPIFFS with Gulp

The ESP8266 flash layout defines a series of blocks of memory for each “partition”. There is a block for the user code (the “sketch”), there is a block for the OTA update file, another one for the emulated EEPROM, another for the WIFI configuration and one for the File System.

This last one uses Peter Andersson’s SPIFFS (SPI Flash File System) code to store files in a similar fashion our computers do, but taking into account the special requirements of an embedded system and a flash memory chip.

This is great because we can store a whole static website there (html, css, js, images,…) and use the official WebServer library that comes with the Arduino Core for ESP8266 project to serve files and execute server side code that updates our static site via AJAX or WebSockets, for instance.

But the ESP8266 is nothing more than a (powerful) microcontroller and the WebServer library has its limitations and if you start to work on a complex website, with multiple files (stylesheets, scripts,…) it will soon fail…

Too many files

Size is not that important, but the number of files is. Too many files lead to failed downloads and long rendering times…

The solution is merging. All javascript files together, all stylesheet files together. But what if you are using third party files, some of them minified? What if you want to keep them separated so you can easily swap from one component to another by just removing a line and adding a new one to your HTML file? You will need some way to process your HTML to get the scripts and stylesheets you are using, merge them and replace the references in the HTML with the unified file.

Well, that’s what most web developers do on a daily basis. They work on a development version and they “process it” to get working code (clean, minimized, renderable). They even have “watchers” that work in the background translating every modification they do on the development version in real time. To do that they use compilers (for SASS, LESS, CoffeeScript,…) and build managers. One of such build managers is Gulp.

So why not use Gulp to process our website before uploading it? Why? So let’s do it.

Installing Gulp

Gulp is actually a Node.js module which is great since Node.js is a really powerful frontend and scripting language and the building file that we will code can benefit from all that power. Installing Node.js and npm (Node.js packet manager) is out of scope here so visit Node.js download page and help yourself.

Next we will need a file to define all the module dependencies we are going to use. These modules are gulp itself and some gulp plugins that will take care of the different tasks (minimizing, injecting, cleaning,…). Lets take a look at it:

{
  "name": "esp8266-filesystem-builder",
  "version": "0.1.0",
  "description": "Gulp based build script for ESP8266 file system files",
  "main": "gulpfile.js",
  "author": "Xose Pérez <[email protected]>",
  "license": "MIT",
  "devDependencies": {
    "del": "^2.2.1",
    "gulp": "^3.9.1",
    "gulp-clean-css": "^2.0.10",
    "gulp-gzip": "^1.4.0",
    "gulp-htmlmin": "^2.0.0",
    "gulp-if": "^2.0.1",
    "gulp-inline": "^0.1.1",
    "gulp-plumber": "^1.1.0",
    "gulp-uglify": "^1.5.3",
    "gulp-useref": "^3.1.2",
    "yargs": "^5.0.0"
  },
  "dependencies": {}
}

As you can see it is a JSON document with some header fields and a list of dependencies. Save it in your code folder as “package.json” and run:

npm install

It will fetch and install all those modules in a node_modules subfolder (I suggest you to add this folder to your .gitignore file). Good, let’s move ahead.

The builder script

I am going to start by showing the script:

/*

ESP8266 file system builder with PlatformIO support

Copyright (C) 2016 by Xose Pérez <xose dot perez at gmail dot com>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

*/

// -----------------------------------------------------------------------------
// File system builder
// -----------------------------------------------------------------------------

const gulp = require('gulp');
const plumber = require('gulp-plumber');
const htmlmin = require('gulp-htmlmin');
const cleancss = require('gulp-clean-css');
const uglify = require('gulp-uglify');
const gzip = require('gulp-gzip');
const del = require('del');
const useref = require('gulp-useref');
const gulpif = require('gulp-if');
const inline = require('gulp-inline');

/* Clean destination folder */
gulp.task('clean', function() {
    return del(['data/*']);
});

/* Copy static files */
gulp.task('files', function() {
    return gulp.src([
            'html/**/*.{jpg,jpeg,png,ico,gif}',
            'html/fsversion'
        ])
        .pipe(gulp.dest('data/'));
});

/* Process HTML, CSS, JS  --- INLINE --- */
gulp.task('inline', function() {
    return gulp.src('html/*.html')
        .pipe(inline({
            base: 'html/',
            js: uglify,
            css: cleancss,
            disabledTypes: ['svg', 'img']
        }))
        .pipe(htmlmin({
            collapseWhitespace: true,
            removeComments: true,
            minifyCSS: true,
            minifyJS: true
        }))
        .pipe(gzip())
        .pipe(gulp.dest('data'));
})

/* Process HTML, CSS, JS */
gulp.task('html', function() {
    return gulp.src('html/*.html')
        .pipe(useref())
        .pipe(plumber())
        .pipe(gulpif('*.css', cleancss()))
        .pipe(gulpif('*.js', uglify()))
        .pipe(gulpif('*.html', htmlmin({
            collapseWhitespace: true,
            removeComments: true,
            minifyCSS: true,
            minifyJS: true
        })))
        .pipe(gzip())
        .pipe(gulp.dest('data'));
});

/* Build file system */
gulp.task('buildfs', ['clean', 'files', 'html']);
gulp.task('buildfs2', ['clean', 'files', 'inline']);
gulp.task('default', ['buildfs']);

// -----------------------------------------------------------------------------
// PlatformIO support
// -----------------------------------------------------------------------------

const spawn = require('child_process').spawn;
const argv = require('yargs').argv;

var platformio = function(target) {
    var args = ['run'];
    if ("e" in argv) { args.push('-e'); args.push(argv.e); }
    if ("p" in argv) { args.push('--upload-port'); args.push(argv.p); }
    if (target) { args.push('-t'); args.push(target); }
    const cmd = spawn('platformio', args);
    cmd.stdout.on('data', function(data) { console.log(data.toString().trim()); });
    cmd.stderr.on('data', function(data) { console.log(data.toString().trim()); });
}

gulp.task('uploadfs', ['buildfs'], function() { platformio('uploadfs'); });
gulp.task('upload', function() { platformio('upload'); });
gulp.task('run', function() { platformio(false); });

Let’s focus on the “File system builder”. There are 5 tasks:

  • clean will delete all the contents of the data folder
  • files will copy all images to the data folder (right now we are not processing them)
  • html processes the HTML files (more on this later)
  • buildfs calls the prior 3 to build the project
  • default is just a convenient task to call buildfs if no task provided.

The “html” task looks for all the *.html files in the development folder and reads them to extract references to *.js and *.css files (that’s what the “useref” plugin does). Then it generates 3 streams. The first one gets all the *.js files, merges them and minifies them (that’s what the “uglify” plugin does). Another one merges all the *.css files and minifies them (that’s what the “cleancss” plugin does). An the final one replaces all the references to the *.js and *.css files with the newly generated unified files in the HTML, and minifies it (that’s what the “htmlmin” plugin does). Finally all the resulting files get gzipped and stored in the data folder, along with all the images.

The good thing about this builder script is that you don’t have to worry about the files you add or delete from your HTML file. It will just read them and merge them in the same order they are in your code. That’s very important since often there are dependencies between files (css precedence, scripts that require jQuery to be defined,…).

Injecting code

The “useref” plugin requires some metadata in the HTML file to know where it should read the files from and inject the new code. A typical file would look like this:

<!DOCTYPE html>
<html>
   <head>

       <title>WEB CONFIG</title>
       <meta charset="utf-8" />
       <link rel="shortcut icon" type="image/x-icon" href="favicon.ico" />
       <meta name="viewport" content="width=device-width, initial-scale=1.0">

       <!-- build:css style.css -->
       <link rel="stylesheet" href="pure-min.css" />
       <link rel="stylesheet" href="side-menu.css" />
       <link rel="stylesheet" href="grids-responsive-min.css" />
       <link rel="stylesheet" href="custom.css" />
       <!-- endbuild -->

   </head>

   <body>

...

    </body>

    <!-- build:js script.js -->
    <script src="jquery-1.12.3.min.js"></script>
    <script src="custom.js"></script>
    <!-- endbuild -->

</html>

As you can see the places where the plugin should read and inject new code are defined by “build” comments like this one: “ “. You have to define the type of files it contains (css) and the name of the output merged file (style.css). All the code from this opening comment to the “ ” closing comment will be replaced by:


Same for the javascript block.  Now, instead of naming your file system folder “data” like the ESP8266FS documentation suggests, name it “html” (or whatever other name, but the code above requires it to be “html”). To run the builder just type from the command line in your code folder:

gulp

If everything goes OK your data folder will have 3 files at least (not counting images): index.html.gz, style.css.gz and script.js.gz.

Inline JS and CSS?

[UPDATE 2016-09-05]

I’ve added the option to get the CSS and JS files and inject them inline in the HTML. This way you end up having just one HTML file, albeit a big one. My tests show that the overall performance is around a 15% faster, both downloading the code (the compressed index.html.gz file is 50 bytes lighter in my test project and the browser only downloads one file instead of 3) and rendering it.

If you had already copied the previous code chunks you will have to update them, both the package.json and the gulpfile.js have modifications, then remember to “npm install” to add new dependencies. Then you can use this “inline” option by calling:

gulp buildfs2

Anyway I still don’t know which is better, since I think the split version can benefit from caching, which is the next step in the investigation…

GZIPping?

Compressing the files makes them lighter and that means you can fit more code in the same room and it will download faster too. But how does the ESP8266 handle it? Well, this code is not mine and it’s pretty standard when using the WebServer library from the Arduino Core for ESP8266, let’s see what it does:

bool handleFileRead(String path) {

    if (path.endsWith("/")) path += "index.html";
    String contentType = getContentType(path);
    String pathWithGz = path + ".gz";
    if (SPIFFS.exists(pathWithGz)) path = pathWithGz;

    if (SPIFFS.exists(path)) {
        File file = SPIFFS.open(path, "r");
        size_t sent = server.streamFile(file, contentType);
        size_t contentLength = file.size();
        file.close();
        return true;
    }

    return false;

}

As you can see the code gets the requested path, translates “/” to “index.html” for the home and grabs the content type from the file extension (not in the code, somewhere else, but that’s what it does). Then it checks if there is a file with the same name plus “.gz”, and if it exists that’s the file it will read. The code is completely agnostic to the contents of the file, it just streams it. But then, deep in the guts of the WebServer library we find this:

template<typename T> size_t streamFile(T &amp;file, const String&amp; contentType){
  setContentLength(file.size());
  if (String(file.name()).endsWith(".gz") &amp;&amp;
      contentType != "application/x-gzip" &amp;&amp;
      contentType != "application/octet-stream"){
    sendHeader("Content-Encoding", "gzip");
  }
  send(200, contentType, "");
  return _currentClient.write(file);
}

So, if the file we are reading ends with “.gz” and we are not serving a gzip file or a binary file, then we are sending something else compressed and hence we should set the Content-Encoding header to “gzip”. Voilà. Our browser will receive the file and it will know it has to decompress it prior to render it.

See the difference?

Optimized SPIFFS contents

No erros, less files, less download size, waaaaay faster

What about the images?

The builder script just copies the images at the moment. There is room for improvement here: compressing them, creating sprites,…

PlatformIO

The builder script above has 3 more tasks to use them with PlatformIO. The main reason for this is that it’s convenient to run “gulp buildfs” (or “gulp” since the default tasks is “buildfs”) before uploading the file system. I’m not good at remembering such things and end up wasting time wondering why I can’t see my changes. So instead of running “gulp” and then do and “uploadfs” from platformio I just do:

gulp uploadfs -e ota -p 192.168.1.113

And my changes are built and flashed via OTA to the device at 192.168.1.113. Neat!

[UPDATE 2016-09-05]

As Ivan Kravets, CTO at PlatformIO, commented below, I could use the “extra_script” feature to call a python script where I can define callbacks or hooks for certain actions, for instance to call my “gulp buildfs” before uploading the file system to the board. The code he added in his comment is a good example of the feature, but I didn’t work for me. Apparently the pre-hook for “uploadfs” is actually called after calling “mkspiffs”. A simple test with a unexistant “data” folder throws an error, but if it had called the gulp task there would be a data folder.

Nevertheless I found a way around the problem when I read in the documentation that they are using filenames as targets for the AddPreAction method. I tested with the spiffs.bin file and it worked! Here’s the python script:

#!/bin/python

from SCons.Script import DefaultEnvironment
env = DefaultEnvironment()

def before_build_spiffs(source, target, env):
    env.Execute("gulp buildfs")

env.AddPreAction(".pioenvs/%s/spiffs.bin" % env['PIOENV'], before_build_spiffs)

Now you just have to add “extra_script = pio_hooks.py” (or the name of your script) to all the environments you want in your platformio.ini file and you are good to go, no need to call gulp, just do as usual:

pio run -e node -t uploadfs

Even neater!

CC BY-SA 4.0 Optimizing files for SPIFFS with Gulp by Tinkerman is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

18 thoughts on “Optimizing files for SPIFFS with Gulp

  1. ikravets

    Hi,

    Thanks a lot for great article and hint with optimization. I added it to our list http://docs.platformio.org/en/latest/articles.html

    I would like to share a hint how to keep PlatformIO workflow without changes and re-use your `gulp` idea. PlatformIO allows to define custom `extra_script` ( http://docs.platformio.org/en/latest/projectconf.html#extra-script ) where you will be able to control current build targets or add new. For example, in your case we need to execute specific command before `uploadfs` target. Let’s do it:

    “`python
    Import(“env”)

    def before_uploadfs(source, target, env):
    env.Execute(“gulp buildfs”)

    env.AddPreAction(“uploadfs”, before_uploadfs)
    “`

    This approach allows to use PlatformIO IDE for SPIFFS uploading without modifications and Terminal.

    Regards, Ivan
    The PlatformIO Team

    Reply
    1. Xose Post author

      Thanks a lot for your comment, Ivan. I also want to thank you guys for that awesome tool, “platformio init” is my first step in every project I work.
      But it looks like I have not dived deep enough in it. The custom script feature is indeed a much better solution and will give it a try soon. Thanks again!

      Reply
    2. Xose Post author

      I updated the post with info about using the extra_script feature to process the files before uploading them. Had some issues on the road but I made it! Thanks so much.

      Reply
  2. ikravets

    > “platformio init” is my first step in every project I work.

    Do you do it with new project or existing?

    Reply
    1. Xose Post author

      Both. In the past I’ve used makefiles and ino (inotool.org). Whenever I have to update any of those projects I first migrate them to use platformio.

      Reply
    1. Xose Post author

      In theory yes, but I guess they will require some changes in the flash layout to use those 16Mbytes memories. I will know soon, I have ordered a pair.

      Reply
      1. Joe Lippa

        Thanks, the ‘gulp-inline’ package seems to be working well here for inlining both css and js resources:

        1) I npm installed the package
        2) I included the new ‘require’ in gulpfile.js: const inline = require(‘gulp-inline’);
        3) I injected the new task into your existing html pipeline e.g.: .pipe(inline({
        base: ‘data.unminified/’,
        js: uglify,
        css: cleancss
        }))

        The end result is and tags are replaced with inlined minified css and js source within the minified gzipped html files.

        This is actually quite marvelous for me because this is a process I was hand cranking until you showed me the light 🙂

        Thanks again
        Joe

        Reply
        1. Xose Post author

          Great, just updated the post with my own findings on the subject. It’s actually faster than having 3 files (html, js and css) but I think that the 3 files pattern could benefit from caching files… more on that in the future. Thank you very much for your idea!

          Reply
          1. Joe Lippa

            Cool good job. I agree that reducing the number of requests / responses by inlining all resources is good a thing from an ESP8266 performance perspective. That said, the reason I started down this path a few months ago was for stability reasons rather than performance reasons as I found the device to be more stable under load with a single html document request pattern as opposed to multiple requests for html, css, js.

            I’ve implemented 304 ‘not modified’ server/client response caching in my own firmware and that can help hugely too and I’d say it’s definitely something worth doing. But most recently I’ve been serving device content through an nginx reverse proxy caching server for TLS/SSL purposes which in a way mostly makes the implementation of onboard device caching redundant (https://jjssoftware.github.io/secure-your-esp8266/)

          2. Xose Post author

            Absolutely must read article, thank you!
            I’ve worked with nginx for years and it’s a great choice, but it ass a level of complexity to the overall solution, only for the tech guy out there. Most people with that requirement (accessing their ESP8266 devices from the internet) will just forward a port in their router, that if they know how to, so basically non-tech people out there will not have this requirement in the first place.
            About the caching, I was thinking on migrating to ESPAsyncWebServer by no-me-dev. It has a bunch of goodies, including caching, authentification and filtering requests.

  3. Joe Lippa

    I’m glad you found that useful, share and share alike and all that 🙂

    As I mentioned in that article, my primary goal there was to implement TLS/SSL protection for my ESP devices and as I ended up with an nginx reverse proxy solution, it came with nice caching benefits too. I agree not a solution for non-techie types although I expect most guys playing with ESP8266 / networked mcu devices are going to be a bit nerdy anyway.

    I expect I’m out of touch with routers and what’s possible with a semi decent one nowadays. My own router is a bit rubbish and port forwarding just forwards unprotected traffic onto some other host. So I ended up port forwarding traffic through my router to my nginx proxy which handles the job of TLS/SSL offload and proxying traffic onto my backend ESPs.

    My firmware stuff is based on the fabulous ESP8266 Arduino Core too and I’m in two minds whether to implement an ESPAsyncWebServer version of my web server class. I’m undecided because the ESP8266WebServer implementation I’m using right now is working well enough and if I had to make the switch to an async server implementation, I guess it would have to be for performance reasons.

    Anyway sorry to drift off topic from your article, I know this isn’t the place for general discussion. Feel free to delete this post if you don’t want to publish it no worries.

    Reply
    1. Xose Post author

      This conversation is very interesting. I’m glad you found my post and decided to write a comment on it. I’ll take a close look at your work in the future.

      Reply
  4. sticilface

    I’m not very familiar with gulp… I can get the default to work and produce 3 files.. but what i’m really interested in is the single file with everything inlined…

    I get this error
    http://pastebin.com/htWNJWWk

    any ideas why this fails?

    Cheers

    Reply
  5. sticilface

    for whatever reason it is the inclusion of

    that is causing it to bork! this is the js file for jQuery mobile…

    any clues?

    Reply
    1. Xose Post author

      Hi
      Don’t know why it fails, apparently it’s malformed but I guess it works when not inline, right? I will try it myself today to see if I can reproduce the error.

      Reply

Leave a Reply