20160819_002338x

RFM69 WIFI Gateway

Some 3 years ago I started building my own wireless sensor network at home. The technology I used at the moment has proven to be the right choice, mostly because it is flexible and modular.

MQTT is the keystone of the network. The publisher-subscriber pattern gives the flexibility to work on small, replaceable, simple components that can be attached or detached from the network at any moment. Over this time is has gone through some changes, like switching from a series of python daemons to Node-RED to manage persistence, notifications and reporting to several “cloud” services.

But MQTT talks TCP, which means you need some kind of translators for other “languages”. The picture below is from one of my firsts posts about my Home Monitoring System, and it shows some components I had working at the time.

Home WSN version 2

All those gears in the image are those translators, sometimes called drivers, sometimes bridges, sometimes gateways. Most of them have been replaced by Node-RED nodes. But not all of them. This is the story of one of those gateways.

Moving from XBee to Moteino

I want you to focus on the “Xbee MQTT Client” in the previous image. XBees are fairly expensive, hard to configure but also pretty powerful and full featured. I could deploy battery powered end devices that run for months from a coin cell like my door sensor or backed by a solar panel like in my weatherstation.

A year ago I started playing with LowPowerLab’s Moteinos as a replacement for the XBee network. A Moteino is an ATMega328 with the Arduino bootloader and HopeRF RFM69 radio on board, everything running at 16MHz and 3.3V. It is a truly low power device and the guy behind LowPowerLab, Felix Rusu, along with the Moteino community have put together an awesome set of libraries to use them. RFM69 use the ISM 868MHz band (here in Europe) with a longer range than the 2.4GHz XBees (although the later have mesh capabilities).

But Moteinos, just like XBees, do not speak TCP. They speak their own language over a FSK modulated radio signal at 868MHz. So you need a gateway to translate messages back and forth the two networks. That’s it, an RFM69 to MQTT bridge.

My first approach a year ago was to copy the XBee gateway idea I was using at the time, basically an XBee in coordinator mode listening to packets from the nodes and forwarding them over serial port to a computer running a python daemon with the python-xbee library to decode API frames and map them to MQTT topics using the Mosquitto python library (before it was donated to the Eclipse Paho project). So I wrote a simple program adapted from Felix Rusu’s gateway.ino example running in a Moteino with an FTDI adapter (yes, there is a MoteinoUSB but I don’t have it) that passes messages over serial port to the computer that translates them to MQTT messages.

Standalone gateway

A Moteino with an FTDI adapter, passing data over serial in a custom protocol to a python script to translate them to MQTT… and back. I was not satisfied with the solution and I never finished the migration from XBee to Moteino.

But now I’ve done another step in what I feel like is a good solution. A standalone ESP8266-based gateway with an RFM69CW radio module on board. I’m not going to say this is the final solution, still have some doubts about it but I like it because it’s a microcontroller based solution, that does just that, bridging two different networks, without help from any other component.

20160819_002338x

RFM69GW board v0.1

rfm69gw-0.3.20160731-topx

RFM69GW v0.3 – top view: fixed ESP12 footprint error, improved silkscreen labels and bigger holes

rfm69gw-0.3.20160731-bottomx

RFM69GW v0.3 – bottom view

For the first version I had to take some decisions:

  • Through hole components except for the radio modules, the AMS1117 and the buttons. I wanted it to be easy to solder.
  • The board form factor was that of the Sick of Beige DP6037 case by SeeedStudio.
  • Also I used RFM69CW footprint because it’s pin-compatible with the RFM12b, so in theory (I have not test it), you could make it work with old Moteinos or JeeNodes.
  • Non FTDI-like programming header. This was a hard decision but I had not very much free room and decided to bring out a GPIO instead, in case I wanted to attach some digital one-wire sensor.

Version 0.1 of the board has some errors, but still it’s usable. There was a fail in the ESP12 footprint I used, GPIO4 and GPIO5 were swapped. As a consequence DIO0 in the RFM69 module is tied to GPIO5 instead, nothing that could not be fixed changing a value in code. Also somehow I drew M2 holes in the board instead of M3 and the programming header is too close to the bottom-right hole so I have problems with common standoffs. And finally, there are problems with some of the silkscreen labels being too small (the button labels for instance) or missing (like the component values). This error has no impact on the functionality but I made it also on two other designs I sent to fab at the same time (ouch!).

Features

20160819_002242x

Testing the gateway. The moteino in the picture has Rusu’s example gateway firmware, so I could check that nodes were sending well formed messages.

The project firmware packs some interesting features, IMHO. Let me summarize them first:

  • Up to 3 configurable WIFI networks
  • Over-The-Air firmware update (OTA)
  • MQTT support, off course
  • Dynamically map nodes/keys and MQTT topics
  • WebServer for configuration using the great PureCSS framework
  • Persistent configuration across reboots using Embedis
  • Optional default topic for non mapped nodes/keys.
  • Configurable IP and Heartbeat topics
  • Visual status of the received messages via the ESP12 LED

Topic mapping

As I explained in another post about MQTT topic naming I think the gateway should be the only responsible for translating messages from one network to another, and this means it has to have the logic to publish messages to the right MQTT topic, for instance. So I wanted to be able to dynamically define a map between nodes, variables and topics.

screenshot-mapping

Web configuration page for the node-to-mqtt map

The gateway expects to receive messages with the key, the value and optionally the packetID (from the sender point of view) separated by colons (i.e. “BAT:2890:34”). This format is not the best in terms of size, but it’s compatible with Rusu’s MotionMote, for instance, and I had already used it in my XBee network. You can then map the combination of nodeID (available in the header of the message) and key to an MQTT topic.

You can also define a default topic that will be used when no match has been found in the map. The default topic can contain {nodeid} and {key} as placeholders to create custom on-the-fly MQTT topics.

Persistence

The other great feature is the configuration module. Just one word: Embedis. You have to use this library by PatternAgents. It’s basically a key-value database for microcontrollers that supports different platforms (including Arduino for AVRs and ESP8266) and different storage like SPI or I2C memories, EEPROM or emulated EEPROM like in the ESP8266. It’s easy to use, robust and powerful, and comes with console management with customizable commands as bonus feature.

Web configuration portal

Another aspect of previous projects I wanted to improve is the web configuration portal. I wanted to give PureCSS by Yahoo a try and it worked great, with jQuery as client language and ArduinoJson and Embedis in the backend. The layout is simple but looks great both on my laptop and on my phone and it’s much more usable than my previous efforts.

screenshot-wifi

The JustWifi library supports up to 3 different networks. If it fails to connect to one of them it will try the next one.

Wifi & Radio management

Also, I have worked in encapsulating wifi and rmf69 functionality in two classes. The first one, JustWifi, is inspired by WifiManager but it just handles wifi connection (hence the name) ripping off all the webserver and DNS stuff. The second one, RFM69Manager, encapsulates the setup and receiving code and outputs a custom packet to the provided callback with all the useful information for a message (nodeID, key, value, packetID and rssi). It also wraps the send method to format a compatible message (“key:value:packetID”).

RFM69 and ESP8266

The moment I read this post in the LowPowerLabs forum I decided to do this project. Felix’s library for the RFM69 was compatible with the ESP8266 almost without modification! But the comments there were a bit confusing. So this is how I made it work:

    • Modify the SPIFlash.cpp file to check SPCR & SPSR before using. My solution is to wrap them between #if clauses to avoid compile errors if not defined, which happens in ESP8266. I’ve forked Felix’s library. Here you have the diff for the changes I made. UPDATE [2016-08-26]: I am no longer using this library for my ESP8266 firmware so this is not needed any more.
    • UPDATE [2016-08-26]: Modify RFM69.cpp file to change SPI clock divider to 2. In my tests this results in a great improvement when sending ACKs (or any other message). This is the patch you should apply on the file:
diff --git a/RFM69.cpp b/RFM69.cpp
index a1e1eeb..ad2e30b 100644
--- a/RFM69.cpp
+++ b/RFM69.cpp
@@ -450,7 +450,7 @@ void RFM69::select() {
   // set RFM69 SPI settings
   SPI.setDataMode(SPI_MODE0);
   SPI.setBitOrder(MSBFIRST);
-  SPI.setClockDivider(SPI_CLOCK_DIV4); // decided to slow down from DIV2 after SPI stalling in some instances, especially visible on mega1284p when RFM69 and FLASH chip both present
+  SPI.setClockDivider(SPI_CLOCK_DIV2); // speeding it up for the ESP8266
   digitalWrite(_slaveSelectPin, LOW);
 }
  • Initialize the RFM69_ATC object (or the RFM69, whichever you use) passing the SS and interrupt pins.
RFM69_ATC radio = RFM69_ATC(SPI_CS, IRQ_PIN, false, IRQ_NUM);

This is what the console output looks like. Messages are from a node that’s continuously sending  the same payload. The node was powered after the gateway came to life, that’s why the first packetID is 1. You can also see how the RSSI value goes down. This is due to the Auto Transmission Control in the RFM69_ATC library that’s set to a minimum of -70. The transmitter will keep on lowering the output power to match this RSSI value, saving power and reducing radio pollution. Key “BAT” for node ID 10 is mapped to “/home/frontdoor/battery” topic (see screen capture above).

[RADIO] Listening at 868 Mhz...
[RADIO] RFM69_ATC Enabled (Auto Transmission Control)
[WIFI] Connecting to last successful network: daoiz
[WIFI] Connected to daoiz with IP 192.168.1.111
[MQTT] Connecting to broker at 192.168.1.100 anonymously: connected!
[MQTT] Sending /raw/rfm69gw/ip 192.168.1.111
[MESSAGE] nodeID:10 packetID:1 name:BAT value: 2310 rssi:-31
[MQTT] Sending /home/frontdoor/battery 2310
[MESSAGE] nodeID:10 packetID:2 name:BAT value: 2310 rssi:-31
[MQTT] Sending /home/frontdoor/battery 2310
[MESSAGE] nodeID:10 packetID:3 name:BAT value: 2310 rssi:-32
[MQTT] Sending /home/frontdoor/battery 2310
[MESSAGE] nodeID:10 packetID:4 name:BAT value: 2310 rssi:-35
[MQTT] Sending /home/frontdoor/battery 2310
[MESSAGE] nodeID:10 packetID:5 name:BAT value: 2310 rssi:-37
[MQTT] Sending /home/frontdoor/battery 2310

Similar projects

Funny enough, while waiting for the boards to arrive from a chinese factory, a couple of very similar projects have come to public attention, the Ebulobo by James Coxon and the Espism by Johan Kanflo.

Both boards are smaller than mine, basically because they place each module on one side of the PCB. They have both chosen to use an USB-A socket to power the board resembling an USB stick, while I use a 2.1mm barrel jack. The Ebulobo board has an SMA connector to plug an antenna, but the Espism uses a pigtail wire as antenna. My board has both options.

James’ Ebulobo uses a header to program it, very much like I do, but without power line (the board has to be plugged to a powered USB port) and instead of having a FLASH button on-board to pull GPIO0 down on boot, it brings out the GPIO0 in the header and you have to use a jumper to tie it to ground. Johan uses a custom programming connector (the Esprog) to flash the firmware on the Espism. This way he reduces board size but the payload is… that you need a custom programming connector.

Both projects have open sourced their schematics, board and code, so they are great projects to learn from. I like their form factor, not sure about the USB connector, not even if there was an FTDI chip and circuitry to program it like you would do to a NodeMCU. OTA is a better option in any case. If the reason is the ubiquity of a power supply connector a microUSB socket would be a better option.

Improvements

I’m just starting to deploy devices using this gateway. I guess I will find things to improve on the way. Right now, the main improvement I have in mind is to support sending messages from MQTT to a remote actuator with an RFM69 radio. Right now the gateway is only one-direction.

The problems I have identified in the board layout are fixed with version 0.3 in the repository, but since the board is fully usable I don’t plan to send it to fab. Yet.

I’m happy with the firmware, it’s a huge improvement from what I had been doing lately and I plan to migrate other projects like my Espurna Smart Socket firmware to use PureCSS and Embedis.

Again, any comment will be welcome.
Tinker and have fun!

This project code (firmware and web interface), PCB schematics and designs are available as free open source software & hardware on the RFM69 Gateway Bitbucket repository.

CC BY-SA 4.0 RFM69 WIFI Gateway by Tinkerman is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

17 thoughts on “RFM69 WIFI Gateway

  1. Steve

    What values did you use to initialise the RFM69 class? I am trying this with mine and I can’t get it to work.

    Reply
    1. Xose Post author

      You should first create an instance of the RFM69 (or RFM69_ATC) class with:

      RFM69 radio = RFM69(SPI_CS, IRQ_PIN, false, IRQ_NUM);
      

      Where SPI_CS is the GPIO linked to the NSS pin in the radio (SS if you use the standard SPI pins in your ESP8266) and IRQ_PIN and IRQ_NUM both the same GPIO that’s attached to the DIO0 in the radio module. So in my case it’s:

      #define SPI_CS              SS
      #define IRQ_PIN             5
      #define IRQ_NUM             5
      

      Then I initialize the module the usual way (networkID, nodeID,…)

      Reply
  2. Pingback: RFM69 WIFI Gateway - Electronics-Lab

  3. Pingback: Monteino Door Monitor | Tinkerman

  4. Pingback: Moteino Door Monitor | Tinkerman

    1. Xose Post author

      It’s a nice board, like a Moteino with a lipo charger highly efficient step up power supply that can be used standalone or a a battery backup. But a bit pricey I wwas looking at the 3 pack price, one Talk2 Wishper is USD23.66. A Moteino with a PowerShield is USD28.

      Reply
      1. Talk²

        Hi Xose, actually the board has a step-up and a dual LDO regulator. So you can power it from as low as 0.8V, like a single AA battery… or just use the battery as backup. IMO Lipo is a great way to power a project but only non-rechargeable batteries will hold power for years. Ideal for use-cases like a sensor node in the roof or inside a water tank, or a simple RF switch in the wall where you don’t plan to replace batteries very often.

        The board is shipped with RFM69 module, SPI 4Mbit Flash and external SMD antenna by default, so the under USD25 price is basically for the components themselves.

        Cheers

        Reply
        1. Xose Post author

          Yes, I noticed my mistake on the price, it was the price for a 3 pack. Actually I just bought one Talk2 Whisper to test it.
          My use case is a solar panel powered device with a lipo as a backup battery so for me a low power device with an RFM69 and a the lipo charger is a good idea 🙂
          Thanks for your comment!

          Reply
  5. Pingback: .NET i jiné ... : Odkazy z prohlížeče – 8.9.2016

  6. mersing

    Great job there !
    I’ve run the Gateway code.
    From a Radio stand point everything is OK.
    To only focus & test the Radio part, I did comment into gateway.ino : // mqtt.publish( topic, message );
    e.g at that stage I didn’t want to send message to MQTT broker.

    I’m receiving the pakets sent by my node : 6 messages sent every 60s.
    Function processMessage from gateway.ino is always displaying 6 messages, every 60s.
    Example below of 1 of the 6 message :
    [MESSAGE] messageID:271 nodeID:4 packetID:0 name:D7644424636D932F;D7644424636D932F;300;UPTIME value: 14184 rssi:-38

    Now, as radio is OK, I uncomment mqtt.publish( topic, message );
    Message are sent to my broker, but I’m losing some of them :
    Function processMessage from gateway.ino is not anymore displaying 6 messages, but some of them.

    I think mqtt.publish( topic, message ) is not treating the incoming messages as fast as they arrive, therefore some of them are dropping.

    I’m facing the same issue when using other code, such as https://github.com/bbx10/nanohab

    I’ll have a look on http://johan.kanflo.com/bridging-ism-radio-and-wifi-for-lunch-money/

    Have you faced such as issue ? Any clue to resolve it ?

    Cheers

    Reply
    1. Xose Pérez Post author

      Six messages every minute is not a heavy load. But maybe they are all sent at the same time? It might be due to the MQTT client struggling to send the message to the broker, and thus delaying the rest…
      Is the broker heavy loaded? Are the client and the broker on the same network?

      Reply
  7. mersing

    Yes, the 6 messages are sent in the same time.
    You are right, I think the MQTT client is struggling to send the message to the broker. The broker is not heavy loaded, but located in the internet. This is why I guess RFM messages are lost.
    I need to find a way of buffering the incoming RFM messages on the Gateway, and having mqtt.publish not blockling. I tried with ArduinoProcessScheduler / ArduinoRingBuffer librairies, but as mqtt.publish is blocking, messages are lost anyway.
    I’m currently looking at using RTOS if I could get better result.

    Reply
    1. Xose Pérez Post author

      I would investigate a little further on non-blocking techniques. But another option you have is to set up a local Mosquitto MQTT broker and configure a bridge to your external one. This way you will free the ESP8266 from dealing with the internet connection lag and move that to a more powerful solution.

      Reply
          1. Xose Pérez Post author

            Good.
            I’ll take a look at it. I still think an asynchronous MQTT library (and WIFI stack) will also do the trick.

Leave a Reply