Automated unit testing in the metal
Unit testing your code is peace of mind. It has two main direct benefits that impact on your confidence about the code you are writing:
- Testing expected behaviors
- Avoiding regressions (i.e. breaking something that was working fine before)
Unit testing embedded systems is a bit more involved since there is the additional constraint of the hardware itself, sometimes more than one device or even different platforms. Quitting (or not even thinking about it) is the easy answer to the problem. All of us have debugged expected behaviors with inline prints. But automating some of these tasks have a huge benefit on the code quality and development speed.
Automating builds
I have been doing automated builds for some time now. This is the very first (basic) way to test your code: does it build? For some projects, like ESPurna, it really makes the difference since it has so many different targets and setting combinations it would be a nightmare to test them all manually. Instead, we are using Travis to build several fake images with different combinations to actually test that most of the code builds, i.e. it doesn't have typos, unmet dependencies,…
Travis also provides a way to create deployment images for the different supported boards in ESPurna. When you download a binary image from the releases page in the ESPurna repository, that file has been automatically created by Travis from a tagged release. That is so cool! You can see how this is done in the .travis.xml file in the root of the repository.
But this is not what I wanted to talk about here.
Existing options
The fact that the project builds, does not mean that it works. The only way to really know that it does what it is supposed to do is to test it on the hardware. This is where we must start using special tools to evaluate conditions (actual versus expected results) and provide an output. This output will probably be via the serial port of the device, although we could think about other fashionable ways to show the result (LEDs, buzzers,…).
Here we have specific tools to do the job. These tools are very much like their “native” counterparts, used for desktop or web languages like Java, PHP, Python… They are usually referred to as testing frameworks. If you are using the Arduino framework you should know about some of these solutions:
- ArduinoUnit. It has no recent activity but it's still the preferred choice by many people. There are two relevant contributors: Warren MacEvoy and Matthew Murdoch.
- AUnit. It is actively developed by Bryan Parks and it has no other relevant contributor.
- GoogleTest. It is a generic C++ test suite but they have recently started developing support for Arduino framework. It is very active and has a big community but it is still a WIP.
- ArduinoCI. It started in 2018 just like the AUnit test suite but has had no activity since September and remains as “beta”. Anyway, it claims to have a really interesting set of features. It is based around mocked-up hardware. It has a single main developer named Ian.
- PlatformIO Unit Testing. This is the only non-free and closed solution. And that's a pity since it has really impressive options.
There are other available options like Arduino-TestSuite or ArduTest, but they are abandoned.
Visually testing it
All the tools above allow you to “visually” test the code. I mean: you run the tests and they will output a result on the serial monitor. “PASSED” or “OK” mean everything is good. The tools in the previous section allow you (or will allow you) to do that, either on the hardware itself or in a mocked-up version of the hardware.
I will focus here on two of the tools above: AUnit and PlatformIO Unit Test. Both are free to use in this stage and provide a very similar feature set. The project I'll be using to test them is something I've been working recently: an RPN calculator for ESP8266 and ESP32 platforms.
The RPNlib library is released under the Lesser GPL v3 license as **free open software **and can be checked out at my RPNlib repository on GitHub.
The library is an RPN calculator that can process c-strings of commands and output a stack of results. Testing this is quite simple: you have an input and an output you can compare to the expected output. Let's see how this can be tested with both solutions.
Testing it with AUnit
AUnit is a testing library by Brian Park. It's inspired and almost 100% compatible with ArduinoUnit but it uses way less memory than the later and supports platforms as ESP8266 or ESP32. It features a full set of test methods and allows you to use wrapper classes with setup and teardown methods to isolate your tests. That's pretty cool.
Here you have an example of usage with one of those classes and the output:
#include <Arduino.h>
#include <rpnlib.h>
#include <AUnit.h>
using namespace aunit;
// -----------------------------------------------------------------------------
// Test class
// -----------------------------------------------------------------------------
class CustomTest: public TestOnce {
protected:
virtual void setup() override {
assertTrue(rpn_init(ctxt));
}
virtual void teardown() override {
assertTrue(rpn_clear(ctxt));
}
virtual void run_and_compare(const char * command, unsigned char depth, float * expected) {
assertTrue(rpn_process(ctxt, command));
assertEqual(RPN_ERROR_OK, rpn_error);
assertEqual(depth, rpn_stack_size(ctxt));
float value;
for (unsigned char i=0; i<depth; i++) {
assertTrue(rpn_stack_get(ctxt, i, value));
assertNear(expected[i], value, 0.000001);
}
}
rpn_context ctxt;
};
// -----------------------------------------------------------------------------
// Tests
// -----------------------------------------------------------------------------
testF(CustomTest, test_math) {
float expected[] = {3};
run_and_compare("5 2 * 3 + 5 mod", sizeof(expected)/sizeof(float), expected);
}
testF(CustomTest, test_math_advanced) {
float expected[] = {1};
run_and_compare("10 2 pow sqrt log10", sizeof(expected)/sizeof(float), expected);
}
testF(CustomTest, test_trig) {
float expected[] = {1};
run_and_compare("pi 4 / cos 2 sqrt *", sizeof(expected)/sizeof(float), expected);
}
testF(CustomTest, test_cast) {
float expected[] = {2, 1, 3.1416, 3.14};
run_and_compare("pi 2 round pi 4 round 1.1 floor 1.1 ceil", sizeof(expected)/sizeof(float), expected);
}
// -----------------------------------------------------------------------------
// Main
// -----------------------------------------------------------------------------
void setup() {
Serial.begin(115200);
delay(2000);
Printer::setPrinter(&Serial);
//TestRunner::setVerbosity(Verbosity::kAll);
}
void loop() {
TestRunner::run();
delay(1);
}
As you can see, you can define any specific testing methods in the library and create and use them directly from the testF methods. This way you can create new tests very fast. Now I just have to build and upload the test to the target hardware, in this case, an ESP32 board:
$ pio run -s -e esp32 -t upload ; monitor
--- Miniterm on /dev/ttyUSB0 115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
ets Jun 8 2016 00:22:57
rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:952
load:0x40078000,len:6084
load:0x40080000,len:7944
entry 0x40080310
TestRunner started on 4 test(s).
Test CustomTest_test_cast passed.
Test CustomTest_test_math passed.
Test CustomTest_test_math_advanced passed.
Test CustomTest_test_trig passed.
Test test_memory passed.
TestRunner duration: 0.059 seconds.
TestRunner summary: 4 passed, 0 failed, 0 skipped, 0 timed out, out of 4 test(s).
You can check the full AUnit test suite for the RPNlib in the repo.
Testing it with PlatformIO
Let's now see how you can do the very same using the PlatformIO Unit Test feature. As you can see it's very much the same, albeit you don't have the wrapping class feature by default, but you can still use helper methods. Of course, this means you have to take care of the code isolation yourself.
#include <Arduino.h>
#include "rpnlib.h"
#include <unity.h>
// -----------------------------------------------------------------------------
// Helper methods
// -----------------------------------------------------------------------------
void run_and_compare(const char * command, unsigned char depth, float * expected) {
float value;
rpn_context ctxt;
TEST_ASSERT_TRUE(rpn_init(ctxt));
TEST_ASSERT_TRUE(rpn_process(ctxt, command));
TEST_ASSERT_EQUAL_INT8(RPN_ERROR_OK, rpn_error);
TEST_ASSERT_EQUAL_INT8(depth, rpn_stack_size(ctxt));
for (unsigned char i=0; i<depth; i++) {
TEST_ASSERT_TRUE(rpn_stack_get(ctxt, i, value));
TEST_ASSERT_EQUAL_FLOAT(expected[i], value);
}
}
// -----------------------------------------------------------------------------
// Tests
// -----------------------------------------------------------------------------
void test_math(void) {
float expected[] = {3};
run_and_compare("5 2 * 3 + 5 mod", sizeof(expected)/sizeof(float), expected);
}
void test_math_advanced(void) {
float expected[] = {1};
run_and_compare("10 2 pow sqrt log10", sizeof(expected)/sizeof(float), expected);
}
void test_trig(void) {
float expected[] = {1};
run_and_compare("pi 4 / cos 2 sqrt *", sizeof(expected)/sizeof(float), expected);
}
void test_cast(void) {
float expected[] = {2, 1, 3.1416, 3.14};
run_and_compare("pi 2 round pi 4 round 1.1 floor 1.1 ceil", sizeof(expected)/sizeof(float), expected);
}
// -----------------------------------------------------------------------------
// Main
// -----------------------------------------------------------------------------
void setup() {
delay(2000);
UNITY_BEGIN();
RUN_TEST(test_math);
RUN_TEST(test_math_advanced);
RUN_TEST(test_trig);
RUN_TEST(test_cast);
UNITY_END();
}
void loop() {
delay(1);
}
To test it you can use the built-in test command in PlatformIO Core.
$ pio test -e esp32
PIO Plus (https://pioplus.com) v1.5.3
Verbose mode can be enabled via `-v, --verbose` option
Collected 2 items
=== [test/piotest] Building... (1/3) ===
Please wait...
=== [test/piotest] Uploading... (2/3) ===
Please wait...
=== [test/piotest] Testing... (3/3) ===
If you don't see any output for the first 10 secs, please reset board (press reset button)
ets Jun 8 2016 00:22:57
rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:952
load:0x40078000,len:6084
load:0x40080000,len:7944
entry 0x40080310
test/piotest/main.cpp:175:test_math [PASSED]
test/piotest/main.cpp:176:test_math_advanced [PASSED]
test/piotest/main.cpp:177:test_trig [PASSED]
test/piotest/main.cpp:178:test_cast [PASSED]
-----------------------
4 Tests 0 Failures 0 Ignored
=== [TEST SUMMARY] ===
test/aunit/env:esp8266 [IGNORED]
test/aunit/env:esp32 [IGNORED]
test/piotest/env:esp8266 [IGNORED]
test/piotest/env:esp32 [PASSED]
=== [PASSED] Took 9.84 seconds ===
Automating your tests
Next step would be to run these tests unassisted. That's it: every time you commit a change to the repo, you want to run the tests on the metal to ensure the results are the expected ones and nothing is broken. Now, this is more involved and both options above (AUnit and PlatformIO) have solutions for that.
The AUnit solution is based on the AUniter script, also maintained by Brian, and Jenkins, an open source continuous integration tool you can install locally or in a server of your own. The AUniter script is actually a wrapper around the Arduino binary in headless mode. This implies two strong conditions for me: a specific folder structure and pre-installed libraries. PlatformIO is more flexible here. Of course, if you are already using the Arduino IDE these conditions might not be hard to meet. Still, you are pretty much limited by the possibilities of the IDE. Maybe when the ArduinoCLI project would leave the alpha stage this will change.
The PlatformIO solution supports a number of CI tools, including Jenkins and Travis. Travis is a very good option since it integrates very well with GitHub or GitLab, so you can have a cloud solution for free. But you might say: “How am I suppose to plug the hardware to the GitHub servers?". Well, the very cool think about PlatformIO is that it supports remote flashing, deploying and testing. The bad news is that these features are not for free and you will have to have a Professional PIO Plus account which is USD36/year for non-commercial products.
Remote testing with PlatformIO
Let me go briefly through the steps to set a testing server locally so you can use it from Travis with PlatformIO. Basically, you will need to have PlatformIO Core installed and a PlatformIO Agent running connected to your PIO Plus account. Let's assume you start with a new Raspbian installation on a Raspberry PI (with internet access already configured).
Let's first install PlatformIO Core (from the Installation page in the documentation of PlatformIO):
$ sudo python -c "$(curl -fsSL https://raw.githubusercontent.com/platformio/platformio/develop/scripts/get-platformio.py)"
And now register to our PIO Plus account (the first time it will install some dependencies):
$ pio account login
PIO Plus (https://pioplus.com) v1.5.3
E-Mail: ************
Password:
Successfully authorized!
And request a token, you will be using this token to start the agent on boot and also to run the tests from Travis:
$ pio account token
PIO Plus (https://pioplus.com) v1.5.3
Password:
Personal Authentication Token: 0123456789abcdef0123456789abcdef01234567
Now, try to manually start the agent. You can see it's named after the Raspberry Pi hostname, acrux in this case:
$ pio remote agent start
2018-12-26 22:57:48 [info] Name: acrux
2018-12-26 22:57:48 [info] Connecting to PIO Remote Cloud
2018-12-26 22:57:49 [info] Successfully connected
2018-12-26 22:57:49 [info] Authenticating
2018-12-26 22:57:49 [info] Successfully authorized
We are almost ready to run code remotely, just some final touch. Add your user to the dialout group so it has access to the serial ports:
$ sudo adduser $USER dialout
And make your life a little easier by using udev rules to create symlinks to the devices you have attached to the Raspberry Pi, this way you will be able to refer to their ports “by name”. You can first list all the connected devices to find the ones you want. In this example below I had just one Nano32 board which uses a FTDI chip:
$ lsusb
Bus 001 Device 005: ID 0403:6015 Future Technology Devices International, Ltd Bridge(I2C/SPI/UART/FIFO)
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. SMC9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Now create the rules and apply them (the Nano32 above and a D1 Mini board):
$ sudo cat /etc/udev/rules.d/99-usb-serial.rules
SUBSYSTEM=="tty", ATTRS{idVendor}=="1a86", ATTRS{idProduct}=="7523", SYMLINK+="d1mini"
SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6015", SYMLINK+="nano32"
$ sudo udevadm control --reload-rules
$ sudo udevadm trigger
OK, let's try to run the code remotely. Go back to your PC and log into your PIO account as before:
$ pio account login
PIO Plus (https://pioplus.com) v1.5.3
E-Mail: ************
Password:
Successfully authorized!
Check if you see the agent on the Raspberry Pi:
$ pio remote agent list
PIO Plus (https://pioplus.com) v1.5.3
acrux
-----
ID: e49b5710a4c7cbf60cb456a3b227682d7bbc1add
Started: 2018-12-26 22:57:49
What devices does it have attached? Here you see the Nano32 in /dev/ttyUSB0 using the FTDI231X USB2UART chip (unfortunately you don't see the aliases, but you can still use them from the platformio.ini file):
$ pio remote device list
PIO Plus (https://pioplus.com) v1.5.3
Agent acrux
===========
/dev/ttyUSB0
------------
Hardware ID: USB VID:PID=0403:6015 SER=DO003GKK LOCATION=1-1.2
Description: FT231X USB UART
/dev/ttyAMA0
------------
Hardware ID: 3f201000.serial
Description: ttyAMA0
And finally, run the tests. This won't be fast, communication is slow and the first time it will install all the dependencies remotely too, so give it some time:
$ pio remote -a acrux test -e esp32
PIO Plus (https://pioplus.com) v1.5.3
Building project locally
Verbose mode can be enabled via `-v, --verbose` option
Collected 2 items
=== [test/piotest] Building... (1/3) ===
Please wait...
Testing project remotely
Verbose mode can be enabled via `-v, --verbose` option
Collected 2 items
=== [test/piotest] Uploading... (2/3) ===
Please wait...
=== [test/piotest] Testing... (3/3) ===
If you don't see any output for the first 10 secs, please reset board (press reset button)
ets Jun 8 2016 00:22:57
rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
ets Jun 8 2016 00:22:57
rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:952
load:0x40078000,len:6084
load:0x40080000,len:7944
entry 0x40080310
test/piotest/main.cpp:175:test_math [PASSED]
test/piotest/main.cpp:176:test_math_advanced [PASSED]
test/piotest/main.cpp:177:test_trig [PASSED]
test/piotest/main.cpp:178:test_cast [PASSED]
-----------------------
4 Tests 0 Failures 0 Ignored
=== [TEST SUMMARY] ===
test/aunit/env:esp8266 [IGNORED]
test/aunit/env:esp32 [IGNORED]
test/piotest/env:esp8266 [IGNORED]
test/piotest/env:esp32 [PASSED]
=== [PASSED] Took 13.10 seconds ===
Amazing! You have run the tests on a physical device attached to a different machine. Let's automate this further.
Running tests from Travis
First, let's run the agent when the Raspberry Pi boots. To do it add the following line to the /etc/rc.local file before the final exit 0. The PLATFORMIO_AUTH_TOKEN environment variable should be set to the token we retrieved before, so it will register to the same account.
PLATFORMIO_AUTH_TOKEN=0123456789abcdef0123456789abcdef01234567 pio remote agent start
We now need to set up the PlatformIO project in the root of the library defining the environments to test:
$ cat platformio.ini
[platformio]
src_dir = .
lib_extra_dirs = .
[env:esp8266]
platform = espressif8266
board = esp12e
framework = arduino
upload_port = /dev/d1mini
test_port = /dev/d1mini
upload_speed = 921600
test_ignore = aunit
[env:esp32]
platform = espressif32
board = nano32
framework = arduino
upload_port = /dev/nano32
test_port = /dev/nano32
test_ignore = aunit
You might have noticed we are using the named ports and also ignoring AUnit tests in the same repository. That's fine. This is what we have been running already in our previous examples. Now let's check the Travis configuration file:
$ cat .travis.yml
language: python
python:
- '2.7'
sudo: false
cache:
directories:
- "~/.platformio"
install:
- pip install -U platformio
script:
- pio remote -a acrux test
So simple: just run all the tests using the acrux agent (our Raspberry Pi). Now the final setting, you have to link you PIO account from Travis. Of course, you will not set the token in the wild or configure you credentials visible in the Travis configuration file. You have two options here: either encrypt the credentials in the file or add it to your project environment variables (in the Settings page of your project page in Travis):
Now we are ready. Do any commit and the code will be tested from Travis in you local tester machine. Enjoy!
"Automated unit testing in the metal" was first posted on 26 December 2018 by Xose Pérez on tinkerman.cat under Tutorial and tagged arduino, arduinoci, arduinounit, aunit, deployment, embedded, esp32, esp8266, espurna, github, googletest, jenkins, platformio, raspberry pi, regression, rpn, rpnlib, travis, unit test.