By now all electricity consumers in Norway are supposed to have got their electricity meters replaced with new smart meters. (The smart meters are often referred to as AMS (Advanced Metering System)).
The new smart meters register electricity consumption at least every hour and automatically sends it to the power company. There's no more use for manually reporting the usage.
What's interesting is that the new smart meters all come with a so-called HAN port (short for Home Area Network). Using that port it's possible to get full access to your own electricity usage realtime. While I'm sure great services (and APIs) for using this data will be provided by both my energy company and third party vendors I didn't want to sit back and wait.
(You can today get access to a bit of delayed hourly usage if you log in to https://plugin.elhub.no/. They also have some nice Ajax calls which are easy to understand and tweak.)
Step 1 - opening the HAN port
By default the physical HAN ports of the smart meters are closed off and not sending any data. All you need to do is to contact customer support at your power company and they'll quickly open it remotely.
My power company - Norgesnett - used almost a month to open it up as they said the newly installed meter had to first be registered in some computer system. Of course I also noticed a pretty bad security vulnerability while at it.
👉 Dinside: Sjekk om du kan aktivere HAN-porten
Hanne lager IoT-bonger for Oslo Pride
Meldte seg som frivillig, endte opp med å digitalisere bongene. - Tror ikke jeg har hatt det så gøy på mange år!
Step 2 - getting the hardware
The smart meters use the M-Bus standard for the physical data transfer. So to read the data stream you need to get some kind of M-Bus converter. The smart meters act as a so-called master and the receiver must be a slave. The master gives enough power to run a slave.
Hardware - first try
From what I read from the forums a lot of people are using - successfully - this (or similar) M-Bus to USB master/slave found on AliExpress, but for me and others it didn't work. I received shorter packages than expected and only the first part of it was readable for me.
Not being an electrical engineer and not knowing how to debug or resolve this I just threw money at the problem and bought another converter.
Hardware - second try
Another commonly used M-Bus to USB master/slave from eBay did the trick for me. I connected it to my good old Raspberry Pi (Model B Rev 2). The HAN port in the smart meters has a RJ-45 connector with the signal being transmitted on pin 1 + 2.
So I just used an old network cable to connect the smart meter and converter.
Step 3 - reading raw data
Python is not my mother tongue, but it's a language I really like and enjoy writing. It's almost always available on whatever system you're on, and the standard library is pretty extensive.
Do a simple pip install pyserial and you're ready to read data from the USB port.
The serial port settings for the data stream for me was 2400 baud, parity bit none and byte size 8 bits. So doing something similar to this I got the raw data stream (code works in both Python version 2.7 and 3.4):
import serial
import codecs
import sys
ser = serial.Serial(
port='/dev/ttyUSB0',
baudrate=2400,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS,
timeout=4)
print("Connected to: " + ser.portstr)
while True:
bytes = ser.read(1024)
if bytes:
print('Got %d bytes:' % len(bytes))
bytes = ('%02x' % int(codecs.encode(bytes, 'hex'), 16)).upper()
bytes = ' '.join(bytes[i:i+2] for i in range(0, len(bytes), 2))
print(bytes)
else:
print('Got nothing')It would output something similar to this (I have anonymized the data a bit):
Connected to: /dev/ttyUSB0
Got 228 bytes:
7E A0 E2 2B 21 13 23 9A E6 E7 00 0F 00 00 00 00 0C 07 E3 06 12 02 14
2F 32 FF 80 00 80 02 19 0A 0E 4B 61 6D 73 74 72 75 70 5F 56 30 30 30
31 09 06 01 01 00 00 05 FF 0A 10 32 32 30 30 35 36 37 32 32 33 31 39
37 37 31 34 09 06 01 01 60 01 01 FF 0A 12 36 38 34 31 31 33 31 42 4E
32 34 33 31 30 31 30 34 30 09 06 01 01 01 07 00 FF 06 00 00 06 A7 09
06 01 01 02 07 00 FF 06 00 00 00 00 09 06 01 01 03 07 00 FF 06 00 00
00 00 09 06 01 01 04 07 00 FF 06 00 00 01 E0 09 06 01 01 1F 07 00 FF
06 00 00 00 88 09 06 01 01 33 07 00 FF 06 00 00 02 36 09 06 01 01 47
07 00 FF 06 00 00 00 6D 09 06 01 01 20 07 00 FF 12 00 EB 09 06 01 01
34 07 00 FF 12 00 EB 09 06 01 01 48 07 00 FF 12 00 EB 83 77 7EStep 4 - understanding OBIS
So what are those bytes coming from the HAN port? They are following the DLMS (Device Language Message Specification) protocol and are sent inside HDLC frames and contains OBIS (Object Identification System) codes that describes the electricity usage. Everything is part of IEC 62056 which is a set of standards for electricity metering data exchange.
Slik lages hardware med algoritmer
Jonas forklarer deg "noe de færreste IT-folk vet": Hvordan man koder fram en mikrobrikke.
How often the messages arrives varies from one meter vendor to another. The same goes for the actual format of the messages. I don't know if there are any other vendors, but at least Aidon, Kaifa and Kamstrup have made smart meters for the Norwegian market, and they all provide documentation for their own OBIS messages.
On my Kamstrup meter I get the current power used every 10 second + the total kWh usage every hour.
To really understand the HDLC and OBIS codes you need dig into different sources around the Internet, but the Norwegian forums at hjemmeautomasjon.no is a great source of information. There are so many knowledgeable people sharing their work and helping each other out.
Realtime usage - every 10 second
I assume I haven't got everything figured out and there are likely some errors, but this is my interpretation of the message:
Header:
7E <-- Frame start flag
A <-- 4 bits, A = 0b1010 = frame format type 3
0E2 <-- 1 bit, segmentation bit + 11 bits, frame length sub-field, 0xE2 = 226 bytes (excluding opening and closing frame flags)
2B <-- Destination address, 1 bit, 0b1 = unicast + 6 bit, node address, 0b010101 = 21 + 1 bit, address size, 0b1 = 1 byte
21 <-- Source address, 1 bit, 0b1 = unicast + 6bit, node address, 0b010000 = 16 + 1 bit, address size, 0b1 = 1 byte
13 <-- Control field
23 9A <-- Header check sequence (HCS) field, CRC-16/X-25
Information:
E6 <-- Destination LSAP
E7 <-- Source LSAP, LSB = 0b1 = command
00 <-- LLC Quality
0F <-- LLC Service Data Unit
00 00 00 00 <-- "Long-Invoke-Id-And-Priority"?
0C <-- string length?, 0C = 12
07 E3 <-- Full year, 0x07E3 = 2019
06 <-- Month, June
12 <-- Day of month, 0x12 = 18
02 <-- Day of week, Tuesday
14 <-- Hour of day, 0x14 = 20
2F <-- Minute of hour, 0x2F = 47
32 <-- Second of minute, 0x32 = 50
FF <-- Hundredths of second, 0xFF = not specified
80 00 <-- Deviation (offset from UTC), 0x8000 = not specified
80 <-- Clock status, 0x80 = 0b10000000, MSB 1 = summer time
02 <-- struct
19 <-- 0x19 = 25 elements
0A <-- visible-string
0E <-- string length 0x0E = 14 bytes
4B 61 6D 73 74 72 75 70 5F 56 30 30 30 31 <-- OBIS List Version Identifier, Kamstrup_V0001
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 00 00 05 FF <-- OBIS for Meter ID, 1.1.0.0.5.255
0A <-- visible-string
10 <-- string length, 10 = 16 bytes
32 32 30 30 35 36 37 32 32 33 31 39 37 37 31 34 <-- Meter ID, altered
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 60 01 01 FF <-- OBIS for meter type, 1.1.96.1.1.255
0A <-- visible-string
12 <-- string lenth, 0x12 = 18 bytes
36 38 34 31 31 33 31 42 4E 32 34 33 31 30 31 30 34 30 <-- Meter type, 6841131BN243101040)
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 01 07 00 FF <-- OBIS for Active Power +, 1.1.1.7.0.255
06 <-- unsigned, 4 bytes
00 00 06 A7 <-- 0x06A7 = 1703 Watt
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 02 07 00 FF <-- OBIS for Active Power -, 1.1.2.7.0.255
06 <-- unsigned, 4 bytes
00 00 00 00 <-- 0 Watt
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 03 07 00 FF <-- OBIS for Reactive Power +, 1.1.3.7.0.255
06 <-- unsigned, 4 bytes
00 00 00 00 <-- 0 Watt
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 04 07 00 FF <-- OBIS for Reactive Power -, 1.1.4.7.0.255
06 <-- unsigned, 4 bytes
00 00 01 E0 <-- 0x01E0 = 480 Watt
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 1F 07 00 FF <-- OBIS for L1 Current, 1.1.31.7.0.255
06 <-- unsigned, 4 bytes
00 00 00 88 <-- 1.36 Ampere
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 33 07 00 FF <-- OBIS for L2 Current, 1.1.51.7.0.255
06 <-- unsigned, 4 bytes
00 00 02 36 <-- 5.66 Ampere
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 47 07 00 FF <-- OBIS for L3 Current, 1.1.71.7.0.255
06 <-- unsigned, 4 bytes
00 00 00 6D <-- 1.09 Ampere
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 20 07 00 FF <-- OBIS for L1 Voltage, 1.1.32.7.0.255
12 <-- unsigned, 2 bytes
00 EB <-- 235 Volt
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 34 07 00 FF <-- OBIS for L2 Voltage, 1.1.52.7.0.255
12 <-- unsigned, 2 bytes
00 EB <-- 235 Volt
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 48 07 00 FF <-- OBIS for L3 Voltage, 1.1.72.7.0.255
12 <-- unsigned, 2 bytes
00 EB <-- 235 Volt
End:
83 77 <-- Frame check sequence (FCS) field, CRC-16/X-25, altered
7E <-- Frame end flagFor me the most interesting part of this message is the OBIS for Active Power + (1.1.1.7.0.255) which tells how much power - in Watt - that is currently being used. If you have a house that produces electricity and exports it to the grid (e.g. if you have solar cells) the exported power would appear as the OBIS for Active Power - (1.1.2.7.0.255).
Total usage - every hour
The message appearing hourly is similar to the one that comes every 10 second, but contains a bit more information:
Connected to: /dev/ttyUSB0
Got 302 bytes:
7E A1 2C 2B 21 13 FC 04 E6 E7 00 0F 00 00 00 00 0C 07 E3 07 09 02 14
00 05 FF 80 00 80 02 23 0A 0E 4B 61 6D 73 74 72 75 70 5F 56 30 30 30
31 09 06 01 01 00 00 05 FF 0A 10 32 32 30 30 35 36 37 32 32 33 31 39
37 37 31 34 09 06 01 01 60 01 01 FF 0A 12 36 38 34 31 31 33 31 42 4E
32 34 33 31 30 31 30 34 30 09 06 01 01 01 07 00 FF 06 00 00 01 6C 09
06 01 01 02 07 00 FF 06 00 00 00 00 09 06 01 01 03 07 00 FF 06 00 00
00 00 09 06 01 01 04 07 00 FF 06 00 00 01 42 09 06 01 01 1F 07 00 FF
06 00 00 00 85 09 06 01 01 33 07 00 FF 06 00 00 00 5C 09 06 01 01 47
07 00 FF 06 00 00 00 3F 09 06 01 01 20 07 00 FF 12 00 EB 09 06 01 01
34 07 00 FF 12 00 EB 09 06 01 01 48 07 00 FF 12 00 EB 09 06 00 01 01
00 00 FF 09 0C 07 E3 07 09 02 14 00 05 FF 80 00 80 09 06 01 01 01 08
00 FF 06 00 38 DE 2A 09 06 01 01 02 08 00 FF 06 00 00 00 00 09 06 01
01 03 08 00 FF 06 00 00 00 1F 09 06 01 01 04 08 00 FF 06 00 09 00 85
83 77 7EThe only part that I really care about is the Active energy A+ (OBIS code 1.1.1.8.0.255), which is total power usage - in kilowatt hour (kWh) - since the installation of the smart meter. Keeping track of this value one knows the hourly power consumption. This is the value you have to pay for. If you produce and exports power it would appear as Active energy A- (1.1.2.8.0.255).
Jonas (35) er CTO for en norsk undervannsdrone
Ukas Koder Jonas Follesø er ansvarlig for teknologien som ble brukt under hevingen av KNM Helge Ingstad.
Error detection
Error detection is supported through cyclic redundancy check (CRC) in both the header and footer of the frame. In the start there is a header check sequence (HCS), and in the end there is a frame check sequence (FCS). The checksum algorithm used is the CRC-16/X-25. There are libraries for all kinds of programming languages implementing all sorts of checksum calculations. I have used the Python library crccheck which provides the class CrcX25 which takes care of this.
Links to more information
Here are some of the sources of information I've used for deciphering the messages. I'll leave them here for anyone wanting to dive deeper:
- https://www.dlms.com/files/Green-Book-Ed-83-Excerpt.pdf
- https://www.dlms.com/files/Blue-Book-Ed-122-Excerpt.pdf
- https://github.com/roarfred/AmsToMqttBridge/blob/master/Samples/Kaifa/readme.md
- https://github.com/roarfred/AmsToMqttBridge/blob/master/Samples/Kamstrup/obisdata.md
- http://www.interfacebus.com/HDLC_Protocol_Description.html
- https://www.hjemmeautomasjon.no/forums/topic/390-ny-strømmåler-med-han-interface/?page=4
- https://www.nek.no/wp-content/uploads/2018/10/Kamstrup-HAN-NVE-interface-description_rev_3_1.pdf
- https://byggebolig.no/imageoriginals/88b3d1774ecb41e6a3fe067ae9e6a893.pdf
Rounding it up
That's it. In this post I've shown how I connected to the HAN port of my smart meter, how I read the data and how to transform the byte arrays into meaningful information.
In the next post I'll discuss how I store the data and calculate the price of the electricity usage.
Jeg fikk en LED til å blinke, følte meg som Gud, og la det i skuffen
Hvorfor kjøper jeg alt dette tullet, gang på gang? 🤦
Kjør kodedebatt!
Velkommen til kode24-debatt. Vi synes kodere bør bli flinkere til å prate sammen! Både deling av erfaringer, tips, triks og ikke minst dine meninger. Så kjør debatt! 👨💻⚡️👩💻