RFM69 WIFI Gateway

Some 3 years ago I started building my own wireless sensor network at home. The technology I used at the moment has proven to be the right choice, mostly because it is flexible and modular.

MQTT is the keystone of the network. The publisher-subscriber pattern gives the flexibility to work on small, replaceable, simple components that can be attached or detached from the network at any moment. Over this time is has gone through some changes, like switching from a series of python daemons to Node-RED to manage persistence, notifications and reporting to several “cloud” services.

But MQTT talks TCP, which means you need some kind of translators for other “languages”. The picture below is from one of my firsts posts about my Home Monitoring System, and it shows some components I had working at the time.

All those gears in the image are those translators, sometimes called drivers, sometimes bridges, sometimes gateways. Most of them have been replaced by Node-RED nodes. But not all of them. This is the story of one of those gateways.

Moving from XBee to Moteino

I want you to focus on the “Xbee MQTT Client” in the previous image. XBees are fairly expensive, hard to configure but also pretty powerful and full featured. I could deploy battery powered end devices that run for months from a coin cell like my door sensor or backed by a solar panel like in my weatherstation.

A year ago I started playing with LowPowerLab's Moteinos as a replacement for the XBee network. A Moteino is an ATMega328 with the Arduino bootloader and HopeRF RFM69 radio on board, everything running at 16MHz and 3.3V. It is a truly low power device and the guy behind LowPowerLab, Felix Rusu, along with the Moteino community have put together an awesome set of libraries to use them. RFM69 use the ISM 868MHz band (here in Europe) with a longer range than the 2.4GHz XBees (although the later have mesh capabilities).

But Moteinos, just like XBees, do not speak TCP. They speak their own language over a FSK modulated radio signal at 868MHz. So you need a gateway to translate messages back and forth the two networks. That's it, an RFM69 to MQTT bridge.

My first approach a year ago was to copy the XBee gateway idea I was using at the time, basically an XBee in coordinator mode listening to packets from the nodes and forwarding them over serial port to a computer running a python daemon with the python-xbee library to decode API frames and map them to MQTT topics using the Mosquitto python library (before it was donated to the Eclipse Paho project). So I wrote a simple program adapted from Felix Rusu's gateway.ino example running in a Moteino with an FTDI adapter (yes, there is a MoteinoUSB but I don't have it) that passes messages over serial port to the computer that translates them to MQTT messages.

Standalone gateway

A Moteino with an FTDI adapter, passing data over serial in a custom protocol to a python script to translate them to MQTT… and back. I was not satisfied with the solution and I never finished the migration from XBee to Moteino.

But now I've done another step in what I feel like is a good solution. A standalone ESP8266-based gateway with an RFM69CW radio module on board. I'm not going to say this is the final solution, still have some doubts about it but I like it because it's a microcontroller based solution, that does just that, bridging two different networks, without help from any other component.

For the first version I had to take some decisions:

• Through hole components except for the radio modules, the AMS1117 and the buttons. I wanted it to be easy to solder.
• The board form factor was that of the Sick of Beige DP6037 case by SeeedStudio.
• Also I used RFM69CW footprint because it's pin-compatible with the RFM12b, so in theory (I have not test it), you could make it work with old Moteinos or JeeNodes.
• Non FTDI-like programming header. This was a hard decision but I had not very much free room and decided to bring out a GPIO instead, in case I wanted to attach some digital one-wire sensor.

Version 0.1 of the board has some errors, but still it's usable. There was a fail in the ESP12 footprint I used, GPIO4 and GPIO5 were swapped. As a consequence DIO0 in the RFM69 module is tied to GPIO5 instead, nothing that could not be fixed changing a value in code. Also somehow I drew M2 holes in the board instead of M3 and the programming header is too close to the bottom-right hole so I have problems with common standoffs. And finally, there are problems with some of the silkscreen labels being too small (the button labels for instance) or missing (like the component values). This error has no impact on the functionality but I made it also on two other designs I sent to fab at the same time (ouch!).

Features

The project firmware packs some interesting features, IMHO. Let me summarize them first:

• Up to 3 configurable WIFI networks
• Over-The-Air firmware update (OTA)
• MQTT support, off course
• Dynamically map nodes/keys and MQTT topics
• WebServer for configuration using the great PureCSS framework
• Persistent configuration across reboots using Embedis
• Optional default topic for non mapped nodes/keys.
• Configurable IP and Heartbeat topics
• Visual status of the received messages via the ESP12 LED

Topic mapping

As I explained in another post about MQTT topic naming I think the gateway should be the only responsible for translating messages from one network to another, and this means it has to have the logic to publish messages to the right MQTT topic, for instance. So I wanted to be able to dynamically define a map between nodes, variables and topics.

The gateway expects to receive messages with the key, the value and optionally the packetID (from the sender point of view) separated by colons (i.e. “BAT:2890:34”). This format is not the best in terms of size, but it's compatible with Rusu's MotionMote, for instance, and I had already used it in my XBee network. You can then map the combination of nodeID (available in the header of the message) and key to an MQTT topic.

You can also define a default topic that will be used when no match has been found in the map. The default topic can contain {nodeid} and {key} as placeholders to create custom on-the-fly MQTT topics.

Persistence

The other great feature is the configuration module. Just one word: Embedis. You have to use this library by PatternAgents. It's basically a key-value database for microcontrollers that supports different platforms (including Arduino for AVRs and ESP8266) and different storage like SPI or I2C memories, EEPROM or emulated EEPROM like in the ESP8266. It's easy to use, robust and powerful, and comes with console management with customizable commands as bonus feature.

Web configuration portal

Another aspect of previous projects I wanted to improve is the web configuration portal. I wanted to give PureCSS by Yahoo a try and it worked great, with jQuery as client language and ArduinoJson and Embedis in the backend. The layout is simple but looks great both on my laptop and on my phone and it's much more usable than my previous efforts.

Wifi & Radio management

Also, I have worked in encapsulating wifi and rmf69 functionality in two classes. The first one, JustWifi, is inspired by WifiManager but it just handles wifi connection (hence the name) ripping off all the webserver and DNS stuff. The second one, RFM69Manager, encapsulates the setup and receiving code and outputs a custom packet to the provided callback with all the useful information for a message (nodeID, key, value, packetID and rssi). It also wraps the send method to format a compatible message (“key:value:packetID”).

RFM69 and ESP8266

The moment I read this post in the LowPowerLabs forum I decided to do this project. Felix's library for the RFM69 was compatible with the ESP8266 almost without modification! But the comments there were a bit confusing. So this is how I made it work:

• Modify the SPIFlash.cpp file to check SPCR & SPSR before using. My solution is to wrap them between #if clauses to avoid compile errors if not defined, which happens in ESP8266. I've forked Felix's library. Here you have the diff for the changes I made. UPDATE [2016-08-26]: I am no longer using this library for my ESP8266 firmware so this is not needed any more.

• UPDATE [2016-08-26]: Modify RFM69.cpp file to change SPI clock divider to 2. In my tests this results in a great improvement when sending ACKs (or any other message). This is the patch you should apply on the file:

diff --git a/RFM69.cpp b/RFM69.cpp
index a1e1eeb..ad2e30b 100644
--- a/RFM69.cpp
+++ b/RFM69.cpp
@@ -450,7 +450,7 @@ void RFM69::select() {
    // set RFM69 SPI settings
    SPI.setDataMode(SPI_MODE0);
    SPI.setBitOrder(MSBFIRST);
-  SPI.setClockDivider(SPI_CLOCK_DIV4); // decided to slow down from DIV2 after SPI stalling in some instances, especially visible on mega1284p when RFM69 and FLASH chip both present
+  SPI.setClockDivider(SPI_CLOCK_DIV2); // speeding it up for the ESP8266
    digitalWrite(_slaveSelectPin, LOW);
  }

• Initialize the RFM69_ATC object (or the RFM69, whichever you use) passing the SS and interrupt pins.

RFM69_ATC radio = RFM69_ATC(SPI_CS, IRQ_PIN, false, IRQ_NUM);

This is what the console output looks like. Messages are from a node that's continuously sending  the same payload. The node was powered after the gateway came to life, that's why the first packetID is 1. You can also see how the RSSI value goes down. This is due to the Auto Transmission Control in the RFM69_ATC library that's set to a minimum of -70. The transmitter will keep on lowering the output power to match this RSSI value, saving power and reducing radio pollution. Key “BAT” for node ID 10 is mapped to “/home/frontdoor/battery” topic (see screen capture above).

[RADIO] Listening at 868 Mhz...
[RADIO] RFM69_ATC Enabled (Auto Transmission Control)
[WIFI] Connecting to last successful network: daoiz
[WIFI] Connected to daoiz with IP 192.168.1.111
[MQTT] Connecting to broker at 192.168.1.100 anonymously: connected!
[MQTT] Sending /raw/rfm69gw/ip 192.168.1.111
[MESSAGE] nodeID:10 packetID:1 name:BAT value: 2310 rssi:-31
[MQTT] Sending /home/frontdoor/battery 2310
[MESSAGE] nodeID:10 packetID:2 name:BAT value: 2310 rssi:-31
[MQTT] Sending /home/frontdoor/battery 2310
[MESSAGE] nodeID:10 packetID:3 name:BAT value: 2310 rssi:-32
[MQTT] Sending /home/frontdoor/battery 2310
[MESSAGE] nodeID:10 packetID:4 name:BAT value: 2310 rssi:-35
[MQTT] Sending /home/frontdoor/battery 2310
[MESSAGE] nodeID:10 packetID:5 name:BAT value: 2310 rssi:-37
[MQTT] Sending /home/frontdoor/battery 2310

Similar projects

Funny enough, while waiting for the boards to arrive from a chinese factory, a couple of very similar projects have come to public attention, the Ebulobo by James Coxon and the Espism by Johan Kanflo.

Both boards are smaller than mine, basically because they place each module on one side of the PCB. They have both chosen to use an USB-A socket to power the board resembling an USB stick, while I use a 2.1mm barrel jack. The Ebulobo board has an SMA connector to plug an antenna, but the Espism uses a pigtail wire as antenna. My board has both options.

James’ Ebulobo uses a header to program it, very much like I do, but without power line (the board has to be plugged to a powered USB port) and instead of having a FLASH button on-board to pull GPIO0 down on boot, it brings out the GPIO0 in the header and you have to use a jumper to tie it to ground. Johan uses a custom programming connector (the Esprog) to flash the firmware on the Espism. This way he reduces board size but the payload is… that you need a custom programming connector.

Both projects have open sourced their schematics, board and code, so they are great projects to learn from. I like their form factor, not sure about the USB connector, not even if there was an FTDI chip and circuitry to program it like you would do to a NodeMCU. OTA is a better option in any case. If the reason is the ubiquity of a power supply connector a microUSB socket would be a better option.

Improvements

I'm just starting to deploy devices using this gateway. I guess I will find things to improve on the way. Right now, the main improvement I have in mind is to support sending messages from MQTT to a remote actuator with an RFM69 radio. Right now the gateway is only one-direction.

The problems I have identified in the board layout are fixed with version 0.3 in the repository, but since the board is fully usable I don't plan to send it to fab. Yet.

I'm happy with the firmware, it's a huge improvement from what I had been doing lately and I plan to migrate other projects like my Espurna Smart Socket firmware to use PureCSS and Embedis.

Again, any comment will be welcome. Tinker and have fun!

This project code (firmware and web interface), PCB schematics and designs are available as free open source software & hardware on the RFM69 Gateway Bitbucket repository.

"RFM69 WIFI Gateway" was first posted on 19 August 2016 by Xose Pérez on tinkerman.cat under Code, Projects and tagged ams1117, bitbucket, edulobo, eeprom, embedis, esp8266, espism, espurna, gateway, github, hoperf, jeenode, lowpowerlab, mosquitto, moteino, mqtt, node-red, nodemcu, paho, purecss, rfm69, rfm69cw, wifimanager, wsn, xbee.