Monitor your TTN gateways with Node-RED

This time I'd like to write a quick post about a small project we've been working on at the The Things Network** community in Barcelona** (@ttncat). We were worried about the monitoring of the gateways we have deployed, both as a community but also as individuals. Some of our partners have also deployed their own gateways and they are part of the community.

So how do we get (almost) real-time notifications of incidences in the local TTN network? We want to be able to track them as soon as possible and open the proper ticket in our management system to solve them if the solution is in our hands or notify the entity responsible for the gateway.

An undocumented API

There are several ways to get info from the TTN network. To be honest it is somewhat untidy at the moment. Of course, you can use the console but that's not an option if you want to automate processes. MQTT entry points are fine for data retrieval but there are no (known) MQTT entry points for gateway metadata. Instead, we have to fall back to gRPC or an HTTP API.

There is a simple (documentation missing) API entry point you can use to query all gateways in a given zone, a country, or even the whole world!

# query all gateways 50km around Barcelona
https://www.thethingsnetwork.org/gateway-data/location?latitude=41.3870734&longitude=2.1700660&distance=50000

# query all gateways in Spain
https://www.thethingsnetwork.org/gateway-data/country/es

# query all gateways in the world!
https://www.thethingsnetwork.org/gateway-data/

The output from these queries is a JSON object with all the gateways for that specific scope. Like this:

{
  eui-b827ebfffea93d27: {
    id: "eui-b827ebfffea93d27",
    description: "xp-gw01-rpi3-ic880a",
    owner: "xoseperez",
    owners: [
      "xoseperez"
    ],
    location: {
      latitude: 41.60128807,
      longitude: 2.62262466,
      altitude: 10
    },
    country_code: "es",
    attributes: {
      brand: "Raspberry Pi DIY",
      frequency_plan: "EU_863_870",
      model: "IMST",
      placement: "indoor"
    },
    last_seen: "2018-10-29T14:29:33Z"
  },
  eui-240ac4fffe13da78: {
    id: "eui-240ac4fffe13da78",
    description: "xp-gw04-esp32-rak833",
    owner: "xoseperez",
    owners: [
      "xoseperez"
    ],
    location: {
      latitude: 41.6013278,
      longitude: 2.62261325,
      altitude: 0
    },
    country_code: "es",
    attributes: {
      frequency_plan: "EU_863_870",
      placement: "indoor"
    },
    last_seen: "2018-10-23T11:13:57Z"
  },
  ...
}

You could easily parse the contents looking for the gateways you are interested in and getting the “last_seen” attribute to check if they are online.

Or you can use yet another undocumented API:

$ wget -q -O - http://noc.thethingsnetwork.org:8085/api/v2/gateways/eui-b827ebfffea93d27 | jq --unbuffered 
{
  "timestamp": "2018-10-29T14:47:33.052648468Z",
  "uplink": "10270",
  "downlink": "3329",
  "location": {
    "latitude": 41.60129,
    "longitude": 2.6226184,
    "altitude": 10
  },
  "frequency_plan": "EU_863_870",
  "platform": "IMST + Rpi",
  "gps": {
    "latitude": 41.60129,
    "longitude": 2.6226184,
    "altitude": 10
  },
  "time": "1540824453052648468",
  "rx_ok": 10270,
  "tx_in": 3329
}

As you can see this API entry point lets you target a specific gateway (by its EUI) and provides interesting information about uplink and downlink packets. On the other side, you do not get the info about the owner or the description. This is what I meant about “untidy”, there is no single place where you could get all the info.

Node-RED implementation

As a first approach let's monitor a series of known gateways (the ones in our community) using the second API above. Doing it from Node-RED is very simple as we only have to trigger (inject) a message every X minutes to perform an HTTP request and parse the output. Then we can store it in an InfluxDB database or use an RBE node (report-by-exception) to check status changes.

This is an example of the flow to check several gateways (both private, community managed and third-party owned) around the Barcelona area:

The reporting is done using Telegram with a simple “The eui-0000… gateway status has changed to offline". You can check and download a template flow based on the above one from the Node-RED flow library here:  TheThingsNetwork gateway monitor with Telegram alarms.

Community monitoring

It's not that we want to be the big brother of the community, but sometimes it can be useful to know what's going on around you. At The Things Network Catalunya, we are trying to be visible and accessible to anyone willing to know about TTN or how to start using the network. We want to join forces with anyone interested in a bottom-up IoT network. It is important to know who's rowing in the same direction.

The flow below checks new gateway activations at 4 levels: 50km around Barcelona, the approximate territory of Catalunya, Spain and the whole world. The data for total and online gateways for each scope is stored into an InfluxDB database.

And also, the list of gateways for a given region (Catalunya in our case) is compared after every request to the previous list (stored in disk), trying to find new activations. Each new activation is then reported to a private group channel in Telegram along with the location, EUI, and description of the gateway. Optionally it uses the  Open Caged Data geocoding API based on OpenStreetMaps to reverse geocode the location. It's our very own ttn_gateways service.

You can check and download a template flow based on the above one from the Node-RED flow library here:  The Things Network - New community gateways.

And the final result in Telegram looks like this:

It's funny to notice that Telegram uses different map providers depending on the platform. The image above is the app for Android and the image belowe is the web interface using Yandex.

Using Grafana to visualize gateway status

A final step is to use Grafana to visualize the status of the gateways along time, and also since we are already grabbing information of total and online gateways per zone, showing a few graphics of that too.

"Monitor your TTN gateways with Node-RED" was first posted on 29 October 2018 by Xose Pérez on tinkerman.cat under Projects and tagged bot, geocoding, grafana, grpc, http api, lora, lorawan, node-red, node-red-contrib-power-monitor, open caged data, open street maps, telegram, the things network, ttncat.