Smart meter part 1: Getting the meter data

How I got access to my realtime electricity usage connecting to my smart meter's HAN port.

The new smart meters all come with a so-called HAN port. Using that port it's possible to get full access to your own electricity usage realtime. 📸: Paul Kleiven / NTB Scanpix
The new smart meters all come with a so-called HAN port. Using that port it's possible to get full access to your own electricity usage realtime. 📸: Paul Kleiven / NTB ScanpixVis mer

By now all electricity consumers in Norway are supposed to have got their electricity meters replaced with new smart meters. (The smart meters are often referred to as AMS (Advanced Metering System)).

The new smart meters register electricity consumption at least every hour and automatically sends it to the power company. There's no more use for manually reporting the usage.

What's interesting is that the new smart meters all come with a so-called HAN port (short for Home Area Network). Using that port it's possible to get full access to your own electricity usage realtime. While I'm sure great services (and APIs) for using this data will be provided by both my energy company and third party vendors I didn't want to sit back and wait.

(You can today get access to a bit of delayed hourly usage if you log in to https://plugin.elhub.no/. They also have some nice Ajax calls which are easy to understand and tweak.)

"By default the physical HAN ports of the smart meters are closed off and not sending any data." 📸: Roy Solberg
"By default the physical HAN ports of the smart meters are closed off and not sending any data." 📸: Roy Solberg Vis mer

Step 1 - opening the HAN port

By default the physical HAN ports of the smart meters are closed off and not sending any data. All you need to do is to contact customer support at your power company and they'll quickly open it remotely.

My power company - Norgesnett - used almost a month to open it up as they said the newly installed meter had to first be registered in some computer system. Of course I also noticed a pretty bad security vulnerability while at it.

👉 Dinside: Sjekk om du kan aktivere HAN-porten

Step 2 - getting the hardware

The smart meters use the M-Bus standard for the physical data transfer. So to read the data stream you need to get some kind of M-Bus converter. The smart meters act as a so-called master and the receiver must be a slave. The master gives enough power to run a slave.

Hardware - first try
From what I read from the forums a lot of people are using - successfully - this (or similar) M-Bus to USB master/slave found on AliExpress, but for me and others it didn't work. I received shorter packages than expected and only the first part of it was readable for me.

Not being an electrical engineer and not knowing how to debug or resolve this I just threw money at the problem and bought another converter.

Hardware - second try
Another commonly used M-Bus to USB master/slave from eBay did the trick for me. I connected it to my good old Raspberry Pi (Model B Rev 2). The HAN port in the smart meters has a RJ-45 connector with the signal being transmitted on pin 1 + 2.

So I just used an old network cable to connect the smart meter and converter.

I just used an old network cable to connect the smart meter and converter. 📸: Roy Solberg
I just used an old network cable to connect the smart meter and converter. 📸: Roy Solberg Vis mer

Step 3 - reading raw data

Python is not my mother tongue, but it's a language I really like and enjoy writing. It's almost always available on whatever system you're on, and the standard library is pretty extensive.

Do a simple pip install pyserial and you're ready to read data from the USB port.

The serial port settings for the data stream for me was 2400 baud, parity bit none and byte size 8 bits. So doing something similar to this I got the raw data stream (code works in both Python version 2.7 and 3.4):

import serial
import codecs
import sys

ser = serial.Serial(
    port='/dev/ttyUSB0',
    baudrate=2400,
    parity=serial.PARITY_NONE,
    stopbits=serial.STOPBITS_ONE,
    bytesize=serial.EIGHTBITS,
    timeout=4)
print("Connected to: " + ser.portstr)

while True:
    bytes = ser.read(1024)
    if bytes:
        print('Got %d bytes:' % len(bytes))
        bytes = ('%02x' % int(codecs.encode(bytes, 'hex'), 16)).upper()
        bytes = ' '.join(bytes[i:i+2] for i in range(0, len(bytes), 2))
        print(bytes)
    else:
        print('Got nothing')

It would output something similar to this (I have anonymized the data a bit):

Connected to: /dev/ttyUSB0
Got 228 bytes:
7E A0 E2 2B 21 13 23 9A E6 E7 00 0F 00 00 00 00 0C 07 E3 06 12 02 14
2F 32 FF 80 00 80 02 19 0A 0E 4B 61 6D 73 74 72 75 70 5F 56 30 30 30
31 09 06 01 01 00 00 05 FF 0A 10 32 32 30 30 35 36 37 32 32 33 31 39
37 37 31 34 09 06 01 01 60 01 01 FF 0A 12 36 38 34 31 31 33 31 42 4E
32 34 33 31 30 31 30 34 30 09 06 01 01 01 07 00 FF 06 00 00 06 A7 09
06 01 01 02 07 00 FF 06 00 00 00 00 09 06 01 01 03 07 00 FF 06 00 00
00 00 09 06 01 01 04 07 00 FF 06 00 00 01 E0 09 06 01 01 1F 07 00 FF
06 00 00 00 88 09 06 01 01 33 07 00 FF 06 00 00 02 36 09 06 01 01 47
07 00 FF 06 00 00 00 6D 09 06 01 01 20 07 00 FF 12 00 EB 09 06 01 01
34 07 00 FF 12 00 EB 09 06 01 01 48 07 00 FF 12 00 EB 83 77 7E

Step 4 - understanding OBIS

So what are those bytes coming from the HAN port? They are following the DLMS (Device Language Message Specification) protocol and are sent inside HDLC frames and contains OBIS (Object Identification System) codes that describes the electricity usage. Everything is part of IEC 62056 which is a set of standards for electricity metering data exchange.

How often the messages arrives varies from one meter vendor to another. The same goes for the actual format of the messages. I don't know if there are any other vendors, but at least Aidon, Kaifa and Kamstrup have made smart meters for the Norwegian market, and they all provide documentation for their own OBIS messages.

On my Kamstrup meter I get the current power used every 10 second + the total kWh usage every hour.

To really understand the HDLC and OBIS codes you need dig into different sources around the Internet, but the Norwegian forums at hjemmeautomasjon.no is a great source of information. There are so many knowledgeable people sharing their work and helping each other out.

Realtime usage - every 10 second
I assume I haven't got everything figured out and there are likely some errors, but this is my interpretation of the message:

Header:

7E          <-- Frame start flag
A           <-- 4 bits, A = 0b1010 = frame format type 3
0E2         <-- 1 bit, segmentation bit + 11 bits, frame length sub-field, 0xE2 = 226 bytes (excluding opening and closing frame flags)
2B          <-- Destination address, 1 bit, 0b1 = unicast + 6 bit, node address, 0b010101 = 21 + 1 bit, address size, 0b1 = 1 byte
21          <-- Source address, 1 bit, 0b1 = unicast + 6bit, node address, 0b010000 = 16 + 1 bit, address size, 0b1 = 1 byte
13          <-- Control field
23 9A       <-- Header check sequence (HCS) field, CRC-16/X-25


Information:

E6                  <-- Destination LSAP
E7                  <-- Source LSAP, LSB = 0b1 = command
00                  <-- LLC Quality
0F                  <-- LLC Service Data Unit
00 00 00 00         <-- "Long-Invoke-Id-And-Priority"?
0C                  <-- string length?, 0C = 12
   07 E3            <-- Full year, 0x07E3 = 2019
   06               <-- Month, June
   12               <-- Day of month, 0x12 = 18
   02               <-- Day of week, Tuesday
   14               <-- Hour of day, 0x14 = 20
   2F               <-- Minute of hour, 0x2F = 47
   32               <-- Second of minute, 0x32 = 50
   FF               <-- Hundredths of second, 0xFF = not specified
   80 00            <-- Deviation (offset from UTC), 0x8000 = not specified
   80               <-- Clock status, 0x80 = 0b10000000, MSB 1 = summer time
   
02                                                        <-- struct
   19                                                     <-- 0x19 = 25 elements

0A                                                        <-- visible-string
   0E                                                     <-- string length 0x0E = 14 bytes
       4B 61 6D 73 74 72 75 70 5F 56 30 30 30 31          <-- OBIS List Version Identifier, Kamstrup_V0001

09                                                        <-- octet-string
   06                                                     <-- string length, 0x06 = 6 bytes
      01 01 00 00 05 FF                                   <-- OBIS for Meter ID, 1.1.0.0.5.255
0A                                                        <-- visible-string
   10                                                     <-- string length, 10 = 16 bytes  
       32 32 30 30 35 36 37 32 32 33 31 39 37 37 31 34    <-- Meter ID, altered

09                                                        <-- octet-string
   06                                                     <-- string length, 0x06 = 6 bytes
      01 01 60 01 01 FF                                   <-- OBIS for meter type, 1.1.96.1.1.255
0A                                                        <-- visible-string
   12                                                     <-- string lenth, 0x12 = 18 bytes
   36 38 34 31 31 33 31 42 4E 32 34 33 31 30 31 30 34 30  <-- Meter type, 6841131BN243101040)

09                          <-- octet-string
   06                       <-- string length, 0x06 = 6 bytes
      01 01 01 07 00 FF     <-- OBIS for Active Power +, 1.1.1.7.0.255
06                          <-- unsigned, 4 bytes
   00 00 06 A7              <-- 0x06A7 = 1703 Watt

09                          <-- octet-string
   06                       <-- string length, 0x06 = 6 bytes
      01 01 02 07 00 FF     <-- OBIS for Active Power -, 1.1.2.7.0.255
06                          <-- unsigned, 4 bytes
    00 00 00 00             <-- 0 Watt
    
09                          <-- octet-string
   06                       <-- string length, 0x06 = 6 bytes
      01 01 03 07 00 FF     <-- OBIS for Reactive Power +, 1.1.3.7.0.255
06                          <-- unsigned, 4 bytes
    00 00 00 00             <-- 0 Watt

09                          <-- octet-string
   06                       <-- string length, 0x06 = 6 bytes
      01 01 04 07 00 FF     <-- OBIS for Reactive Power -, 1.1.4.7.0.255
06                          <-- unsigned, 4 bytes
   00 00 01 E0              <-- 0x01E0 = 480 Watt

09                          <-- octet-string
   06                       <-- string length, 0x06 = 6 bytes
      01 01 1F 07 00 FF     <-- OBIS for L1 Current, 1.1.31.7.0.255
06                          <-- unsigned, 4 bytes
   00 00 00 88              <-- 1.36 Ampere

09                          <-- octet-string
   06                       <-- string length, 0x06 = 6 bytes
      01 01 33 07 00 FF     <-- OBIS for L2 Current, 1.1.51.7.0.255
06                          <-- unsigned, 4 bytes
   00 00 02 36              <-- 5.66 Ampere

09                          <-- octet-string
   06                       <-- string length, 0x06 = 6 bytes
      01 01 47 07 00 FF     <-- OBIS for L3 Current, 1.1.71.7.0.255
06                          <-- unsigned, 4 bytes
   00 00 00 6D              <-- 1.09 Ampere
   
09                          <-- octet-string
   06                       <-- string length, 0x06 = 6 bytes
      01 01 20 07 00 FF     <-- OBIS for L1 Voltage, 1.1.32.7.0.255
12                          <-- unsigned, 2 bytes
   00 EB                    <-- 235 Volt

09                          <-- octet-string
   06                       <-- string length, 0x06 = 6 bytes
      01 01 34 07 00 FF     <-- OBIS for L2 Voltage, 1.1.52.7.0.255
12                          <-- unsigned, 2 bytes
   00 EB                    <-- 235 Volt

09                          <-- octet-string
   06                       <-- string length, 0x06 = 6 bytes
      01 01 48 07 00 FF     <-- OBIS for L3 Voltage, 1.1.72.7.0.255
12                          <-- unsigned, 2 bytes
   00 EB                    <-- 235 Volt


End:

83 77                       <-- Frame check sequence (FCS) field, CRC-16/X-25, altered
7E                          <-- Frame end flag

For me the most interesting part of this message is the OBIS for Active Power + (1.1.1.7.0.255) which tells how much power - in Watt - that is currently being used. If you have a house that produces electricity and exports it to the grid (e.g. if you have solar cells) the exported power would appear as the OBIS for Active Power - (1.1.2.7.0.255).

Total usage - every hour
The message appearing hourly is similar to the one that comes every 10 second, but contains a bit more information:

Connected to: /dev/ttyUSB0
Got 302 bytes:
7E A1 2C 2B 21 13 FC 04 E6 E7 00 0F 00 00 00 00 0C 07 E3 07 09 02 14
00 05 FF 80 00 80 02 23 0A 0E 4B 61 6D 73 74 72 75 70 5F 56 30 30 30
31 09 06 01 01 00 00 05 FF 0A 10 32 32 30 30 35 36 37 32 32 33 31 39
37 37 31 34 09 06 01 01 60 01 01 FF 0A 12 36 38 34 31 31 33 31 42 4E
32 34 33 31 30 31 30 34 30 09 06 01 01 01 07 00 FF 06 00 00 01 6C 09
06 01 01 02 07 00 FF 06 00 00 00 00 09 06 01 01 03 07 00 FF 06 00 00
00 00 09 06 01 01 04 07 00 FF 06 00 00 01 42 09 06 01 01 1F 07 00 FF
06 00 00 00 85 09 06 01 01 33 07 00 FF 06 00 00 00 5C 09 06 01 01 47
07 00 FF 06 00 00 00 3F 09 06 01 01 20 07 00 FF 12 00 EB 09 06 01 01
34 07 00 FF 12 00 EB 09 06 01 01 48 07 00 FF 12 00 EB 09 06 00 01 01
00 00 FF 09 0C 07 E3 07 09 02 14 00 05 FF 80 00 80 09 06 01 01 01 08
00 FF 06 00 38 DE 2A 09 06 01 01 02 08 00 FF 06 00 00 00 00 09 06 01
01 03 08 00 FF 06 00 00 00 1F 09 06 01 01 04 08 00 FF 06 00 09 00 85
83 77 7E

The only part that I really care about is the Active energy A+ (OBIS code 1.1.1.8.0.255), which is total power usage - in kilowatt hour (kWh) - since the installation of the smart meter. Keeping track of this value one knows the hourly power consumption. This is the value you have to pay for. If you produce and exports power it would appear as Active energy A- (1.1.2.8.0.255).

Error detection
Error detection is supported through cyclic redundancy check (CRC) in both the header and footer of the frame. In the start there is a header check sequence (HCS), and in the end there is a frame check sequence (FCS). The checksum algorithm used is the CRC-16/X-25. There are libraries for all kinds of programming languages implementing all sorts of checksum calculations. I have used the Python library crccheck which provides the class CrcX25 which takes care of this.

Links to more information
Here are some of the sources of information I've used for deciphering the messages. I'll leave them here for anyone wanting to dive deeper:

Rounding it up

That's it. In this post I've shown how I connected to the HAN port of my smart meter, how I read the data and how to transform the byte arrays into meaningful information.

In the next post I'll discuss how I store the data and calculate the price of the electricity usage.