Category Archives: Learning

Decoding 433MHz RF data from wireless switches

[Update 2013-03-01] I have added more documentation on the codes these remotes use in a different post.

I’m starting to move towards not only gathering information but also acting. My first project in this subject will be controlling some lights and the house heaters. So last week I visited the urban market of “Els Encants” in Barcelona and bought some very cheap wireless outlets.

Two different remotes

I bought two sets of three wall plugs, each set with it’s own remote. They all transmit in the 433MHz frequency and I already had a set of transmitter and receiver for that frequency so as soon as I had some time I started trying to intercept and reproduce the codes the remotes were sending.

Sample outlets from each set plus remotes

Sample outlets from each set plus remotes

In the image above you can see an outlet and the remote for each of the sets. The left one is branded “Noru” and each outlet is rated 3kW (good for the heaters) and it features auto switch off time (1, 2, 4 or 8 hours). The remote can control a maximum of 3 outlets and apparently it is programmable, since you first have to sync the outlets with the remote.

The right one is branded “Avidsen”, also 433Mhz but rated 1000W, just below the consumption of my house electrical heaters, but good to control lights and other appliances. It’s got the very common dip switch to sync the remote and up to five outlets. There are 32 different channels available. So if your lights switch on and off randomly maybe you neighbour is using the same channel you are, then you better change the channel.

Available libraries for Arduino

I started reading documentation about the protocol these devices use and found out there is some very useful information out there. In fact there are even a couple of libraries for Arduino. The first one is called RemoteSwitch and it is a little old, it has not been ported to Arduino 1.0 but if you are like me you will keep a copy of Arduino 0023 just for this kind of situations.

The second library is called RCSwitch and I have to say it is a neat piece of code. It has been ported to the Raspberry Pi, although the port is not as updated as the original Arduino library.

My first tests with the RemoteSwitch library were encouraging. The Show_received_code sketch dumped the codes for the Avidsen remote one by one. I though: if it can decode it, it should be able to recreate it. And it worked from scratch. Good!

But by then I knew I wanted to use the newer library. There were several reason for this: it is being actively developed, it supports more protocols, the code is much more elegant and object oriented and it has a port to RPi, which I plan to use as a central gateway soon. So I checked which one of the RCSwitch protocols matched the one I had successfully used with RemoteSwitch and started doing some more tests…

Slightly different protocol

Here was when things started to get complicated. The thing did not work. So I spent a couple of hours studying the code for both libraries, decoding the codes the RemoteSwitch library had dumped before and trying to find the difference. Until I found it: RCSwitch.cpp, line 239, that ‘0’ should be a ‘1’… and everything started working again. Very good! I started a thread in the library forum to find out whether this is a bug or a slightly different protocol.

Index: RCSwitch.cpp
===================================================================
--- RCSwitch.cpp	(revision 219)
+++ RCSwitch.cpp	(working copy)
@@ -284,7 +284,7 @@
         if (sGroup[i] == '0') {
             sDipSwitches[j++] = 'F';
         } else {
-            sDipSwitches[j++] = '0';
+            sDipSwitches[j++] = '1';
         }
     }

By the way, the protocol of these things is pretty interesting. It’s worth a look at Sui’s post to get an overview of the implementation. The tri-bit concept is really awkward.

Using the Bus Pirate and OLS Logic Analyser

Then I moved to the other set of outlets. These are rated 3000W so I plan to use them to control my house heaters, which is the main reason for all this work. I followed the same steps, starting with getting the codes with the Show_received_code sketch. But weird enough the library was only able to decode some of the button presses… Only the SET button for outlet #1, the ON and OFF buttons for outlet #2, the ALL OFF button or the 2, 4 and 8H timeout buttons seemed to work.

This time it was going to be harder, since I didn’t even have all the codes. Well, a good opportunity to use my Bus Pirate!

Bus Pirate to the rescue!

Bus Pirate to the rescue!

So I plugged the RF receiver to the Bus Pirate and launched the OLS Logic Analyser to capture the signal from the remote.

You don’t have to configure anything to use the Bus Pirate as a (low speed) logic analyser. But since I wanted to power the radio receiver with the BP I had to enable the Power Supply mode. To do so you have to open a terminal session, type ‘?’ to get a prompt, select one of the modes that allow enabling the power supply typing ‘m’ and selecting the mode (like UART, for instance) and then type ‘W’ (uppercase to enable, lowercase to disable). Then you can close the session and it will keep the power on the 5V and 3V3 lines as long as it is plugged to the computer. Mind you have to free the port so the logic analyser software can use it. I had problems doing it with screen or minicom, but it worked great with picocom.

After some tests with the Avidsen remote (I knew what the codes were so I could compare the signal output with the actual code) I started getting the signals for each and every button in the Noru remote.

The image below shows the signal for the ON button for the outlet #1.

Signal for the #1 ON button of the Noru remote

Signal for the #1 ON button of the Noru remote

Now, since the RemoteSwitch library was able to decode some of the signals, the protocol could not be that different. So I started to decode manually all the signals applying the same protocol. The signal is a series of 12 tri-bits plus a sync-bit. For the Avidsen-like remotes there are 3 different tri-bit values (logically), they are called 0, 1 and F, for “floating”. Each tri-bit has a pulses shape. The following tables describes the pulses:

Tri-bit Pulses
0 short high + long low + short high + long low
1 long high + short low + long high + short low
F short high + long low + long high + short low

The long pulses are about 3 times the length of the sort ones. The overall period is a characteristic of each remote. There is also a trailing high pulse followed by a long low which is called “sync bit”.

A fourth tri-bit?

Decoding the signals from the Noru remote I found out that there was a fourth tri-bit value (well maybe I should call them tetra-bits now). In fact it is obvious since there is a forth option for an alternate sequence of 4 highs and lows. I’ve named the new tetra-bit X (for unknown, but also after my name :P). The full table for the Noru remotes is:

Tretra-bit Pulses
0 short high + long low + short high + long low
1 long high + short low + long high + short low
F short high + long low + long high + short low
X long high + short low + short high + long low

Now the previous image for the ON#1 button can be decoded as 1F000001FFX0S. With a small patch I could make this work with the RCSwitch library. The library cannot create the code but you can feed it to the sendTriState method to generate the signal.

Index: RCSwitch.h
===================================================================
--- RCSwitch.h	(revision 219)
+++ RCSwitch.h	(working copy)
@@ -106,6 +106,7 @@
     void sendT0();
     void sendT1();
     void sendTF();
+    void sendTX();
     void send0();
     void send1();
     void sendSync();
Index: RCSwitch.cpp
===================================================================
--- RCSwitch.cpp	(revision 219)
+++ RCSwitch.cpp	(working copy)
@@ -441,6 +441,9 @@
         case '1':
           this->sendT1();
         break;
+        case 'X':
+          this->sendTX();
+        break;
       }
       i++;
     }
@@ -561,6 +564,16 @@
 void RCSwitch::sendTF() {
   this->transmit(1,3);
   this->transmit(3,1);
+}
+
+/**
+ * Sends a Tri-State "X" Bit
+ *            ___   _
+ * Waveform: |   |_| |___
+ */
+void RCSwitch::sendTX() {
+  this->transmit(3,1);
+  this->transmit(1,3);
 }

 /**

And this is a sample code for Arduino that switches on and off outlet #1 every 2 seconds.

#include <RCSwitch.h>

RCSwitch mySwitch = RCSwitch();

void setup() {

  Serial.begin(9600);

  // Transmitter is connected to Arduino Pin #11
  mySwitch.enableTransmit(11);

  // Optional set pulse length.
  mySwitch.setPulseLength(302);

  // Optional set protocol (default is 1, will work for most outlets)
  mySwitch.setProtocol(1);

  // Optional set number of transmission repetitions.
  mySwitch.setRepeatTransmit(6);

}

void loop() {
    mySwitch.sendTriState("1F000001FFX0");
    delay(2000);
    mySwitch.sendTriState("1F000001FFFX");
    delay(2000);
}

Again, comments are more than welcome!

Storing and publishing sensor data

Now that I have started to monitor some magnitudes at home, like power consumption or the front door opening, I have to do something with this information. The sensor information can be useful for two purposes, mainly:

  • analysing it to know more about your environment or your life-style patterns (like when and how you spend the money you spend on energy)
  • taking real-time actions based on events from your sensors (open a light when you pass by a dark corridor at night, receive a notification when someone enters your house,…)

To analyse the information you will first have to store it in some way you could retrieve it later, graph it, summarize it, perform different time range roll ups or moving averages or detect time patterns,… whatever. Even if the goal of the sensor information is to trigger events (watch out: the door sensor is running out of batteries!) you will probably want to have it stored somewhere and save a log of actions as well.

So, where to store this information? You have different options available:

  • Relational databases (RDBMS) like MySQL or PostgreSQL. If you have some programming background this is pretty easy, we are all used to store data on a MySQL or similar. But mind the tables will GROW very quickly. A sensor reporting every minute will create more than half a million entries a year.
  • NO-SQL databases like MongoDB or Cassandra. They are popular and everyone wants to use them. Cassandra is often used to store time-series data like server load or log files and the Internet is full of information about how to store it, retrieve it, partition it,… MongoDB has also some nice features like capped collections.
  • Round-robin databases. They have been around for a while. RRDtool version 1.0 was born in 1999 and MRTG is even older. Data is stored in circular buffers, you can define the size of the buffer and the step or how often a new value is inserted into the buffer. When the buffer is full old data gets overwritten and this happens over and over. So you have another buffer for aggregated data (like daily data) and another one for even larger scale data (like weekly data) and so on.
  • Time series databases. And finally there are a number of databases that have been specifically designed to store time-series data: fast inserts, fast range lookups, roll ups,…

Now, some companies are providing time series storage services online. They provide a public API, graphing capabilities and some of them even roll ups or data aggregation. They have been born around the Internet of Things hype. Maybe the most popular is Cosm (formerly Pachube) but it’s not the only one: TempoDB, Nimbits or ThingSpeak are very good options.

Cosm has been around for a while and it is still the most used by the IoT community. AFAIK, there is no restriction on the number of data points you can store for free if you keep your data “public”. Data is structured in “feeds” and “datastreams”. They have an API you can use to push data individually or in batches. You can also use it to backup the data you have in Cosm. Their charts are nice looking but they have predefined time ranges that automatically aggregate your data so you have little control on what and how you want to graph. They have a number of goodies like triggers, graph builder (to insert your Cosm charts in your site), tags, apps, localization based search,…

TempoDB is a relatively new player. It’s free up to 5 million data points per database, that’s roughly 9.5 years for a 1 samples/minute data. You can have several “series” of data per database. Their API is really good and they have a number of clients including a python one :^). Their charts might not be as “pretty” as those from Cosm but you have full control on the data you are visualizing: date range, interval and aggregation functions, and they are FAST (Cosm can be really sluggish sometimes). Your data is private, so there is no sense in having a search tool. They have tags and I’ve been told they are about to release a notification service.

Nimbits has a desktop-like interface base on ExtJS. It is basically a paid service (their free quota is 1000 API requests per day, which is OK for 1 sensor reporting at a rate of 1 samples every 2 minutes) and it costs $20/2 million requests (roughly $20 per year if you have 4 sensors reporting at a 1 sample/minute rate).

ThingSpeak is very similar to Cosm, data is presented in a “channel” where you can see multiple charts for different magnitudes (they call them “fields”), a map and even a video from Youtube. It’s open source, so you can clone the code from Github and install it in you server. Their web service is limited to 1 API request every 15 seconds per channel, which is OK as long as you group all the data from one channel in a single request (that’s it if you have more than one field per channel). For each field you have a lot of options to define your chart: time range, time scale, different aggregation functions, colors, scale,…

The information in this post is not very detailed but I hope it can be an introduction to anyone who is looking for ways to store or publish sensor data. Right now I’m using Cosm and TempoDB, as well as storing all my data in a local MySQL database (not the best option but it works for now). There are plenty of options I still have to explore. In my next post I will talk about the daemon I’m using to push data to Cosm and TempoDB, in the meantime you can check the code on Github.

Smartmeter pulse counter (3)

The smartmeter pulse counter will be the first standalone sensor I will deploy so power economy is a requirement.

I will have to power an Arduino Pro Mini and an Xbee radio. I plan to power the Xbee from the 3V3 regulated output. The Arduino VCC output can provide as much as 200mA which is far enough to power the Xbee. The built-in regulator requires at least 3.35V and up to 12V so 3 AA cells will provide enough potential to drive the whole setup. But for how long?

The Arduino will wake briefly to count every pulse and the radio has been configured to wake once every minute for 200ms and ask the uC to report the count number for since the last report. The figures for the number of pulses are based on an average consumption of 300W, that is one pulse every 3 seconds or 1200 per hour.

Once I had breadboarded the sensor I started to take some measurements in different operational statuses. You can check my results in the following table:

Arduino XBee mA ms/event events/h ms/h mA (avg)
normal transmiting ~54 200 60 12000 0.180
normal sleep 7.940 1 1200 1200 0.003
sleep sleep 0.290 - - 3586800 0.289
0.472

So 0.472 mA is the average power consumption of the sensor. The AA cells I will be using are rated 2000mAh so the sensor should be able to run for 4237 hours or about 177 days. Not bad. Off course there is room for improvement, like 290uA when doing nothing is really too much but I prefer to give it a try now before over optimizing. After all these are theoretical values and I want to know if the will match the reality.

I plan to test different approaches to power the next sensors: coin cells, LiPo cells, solar panels, rechargable NiMH batteries,…

Smartmeter pulse counter (1)

Endesa has recently updated my home energy meter to meet new UE and Spanish regulations. The new meter it’s a smartmeter by Meters and More and it basically provides the means to perform remote meter readings and supply changes. The goal of the new regulation is to provide better and more accurate information to final customers regarding their energy consumption habits to promote energy efficiency and saving.

Endesa smartmeterThe reason I’m writing about this is that the meters has two communication ports on the front side, an I/O optical port with unknown protocol and a simple LED labeled “4000 Imp/KWh”. This means that every 4 pulses you have used 1Wh of energy or the equivalent: 1 pulse/sec its 900W of instantaneous power consumption (2 pulses/sec 1800W, 3 pulses/sec 2700W or 1 pulse every 2 secs 450W and so on).

So I’m planning to set up a simple pulse counter and send the information to my IoT gateway and store it somewhere (cosm.com or my own webapp). The idea is to use a voltage divider with a resistor and a photocell with the output attached to one of the hardware interrupt pins of an Arduino Mini Pro. The resistor should be at least four times bigger than the resistance of the photocell when illuminated by the pulse light so the Arduino would interpret it as a HIGH. Then count the pulses in the interrupt method and either send a message to the server every say 40 pulses (10 Wh) or every minute using a RTC with an alarm triggering the other interrupt pin in the Arduino.

The first critical issue was to be able to accurately count all the pulses. I was sceptical about using software debouncing since the processor will be busy with turning on the radio and sending the data for maybe one second every minute and it won’t be able to debounce incoming pulses in the meantime so I needed clean signals in the interrupt pin.

I did some quick tests with a CdS photocell in a voltage divider configuration with a 47kOhm resistor and the response of the sensor seemed to be good enough to capture the pulse. The photocell resistance easily dropped below 5kOhms and the pulse looked like some 50ms width. But I needed more precise readings.

And here it came the DSO Nano v2 to the rescue. I attached the probes to the voltage divider output and to ground and after some testing trying to find the right parameters in the scope I managed to get a pretty decent image of the output.

Smart meter pulse reading by the photocell

The image shows a pulse with 2.96 Vpp (I was using a 3V3 source) and some 50ms width. The pulse has soft rising and falling edges that resemble those from a RC circuit (albeit one very curious RC circuit with a time constant of around 10ms when rising and 20ms when falling).

Good news is that the output seems ripple-free so there is no need for further softening and I can just plug it to a Schmitt trigger to get sharp edges. I used a 74HC14 I had from another (unfinished) project around although the packaging is a little overkill (I’m only using 1 of the 6 gates it has).

Pulse after Schmitt trigger

The final pulse has sharp edges and is 80ms width. This gives me time to capture up to 12 pulses per second. That would be a instantaneous power consumption of 10.8kW which is almost double I have at home.

So far so good.