Parsing information from a Dutch Smart Meter

In the Netherlands and increasing number of homes have a smart energy meter, that communicates with the energy supplier and automatically provides them with numbers of the usage. In earlier times, you had to read of the reader yourself, fill in a registration form and mail it in to the energy supplier. The newer meters have an accessory port that can be used by the home owner, to read out the meter data. In this short article I will explain how I did this, but there are plenty of examples available online for those that want to DIY it or want to further integrate it into their home automation environment.

If you do not want to implement this yourself, you can just buy a commercial of the shelf P1 meter and use the included web environment and/or app to get access to the meter information. For a comparison of commercial of the shelf options, have a look at P1Meter.nl [1]. Keep reading if you are interested in putting something together based on a Raspberry Pi or similar single board computer.

Dutch Smart Meter Requirements

The collective of dutch network operators Netbeheer NL publishes the requirements for Smart Meters that are intended for the Dutch market. Most of this documentation is only interesting to meter manufacturers, but it for our purposes the requirements for the P1 port are of particular interest. Each meter that meets the DSMR specifications has a RJ11 connector somewhere on the device that is labeled P1 and that is accessible to the user. The meter publishes a telegram message to this port at set intervals of time (once per second for DSMRv5 meters, once every 10 seconds for older versions) that contain information on the voltages and currents per phase, the instant power consumption and delivery, the total energy consumption per tariff and other information on for example the number of power outages. The technical specification of the P1 Companion Port is available from Netbeheer NL [1].

Required hardware for the P1 reader

In principle, all you need is a computer and a cable to connect it to the P1 port of the meter. I used a Raspberry Pi Zero 2 W as a single board computer that consumes little power and can be left inside the fuse box.

Parsing the DSMRv5 telegrams

In this article, I will focus on the DSMRv5 standard, simply because our meter uses that specification. Older versions are available online as well, but they follow the same principles. If you can follow the rest of this article, you should be capable of adapting it to your own meter as well.

The protocol is described in Chapter 6 of the Companion Standard [2]. It is based on the NEN-EN-IEC 62056-21 Mode D standard, but does not conform to it in its entirety. The data is transmitted at a fixed rate of 115200 baud and is defined as 8N1: 1 start bit, 8 data bits, no parity bit and 1 stop bit. Each message or telegram consists of several lines.

When parsing the telegram, not all information will be useful. For example, the first line of each telegram contains identification information. The most important bit is that the first line of a telegram starts with a forward slash (/), and that the last line ends with an exclamation mark (!). After that, every line that contains useful and interesting information about energy consumption starts with a reference number to identify the information contained in that line. For example, the actual power delivered to the customer is item 1-0:1.7.0, and the actual power delivered back to the energy supplier is item 1-0:2.7.0.

Because each of the lines follows a well defined format, we can use Regular Expressions to check if the line matches the pattern and if so, extract the values of interest. Instead of explaining how it works, I am giving you some dressed down Python 3.x code, that verbosely walks your through the process.

import serial    # Import the serial library for serial communication to P1 port of the meter
import re        # Import the regular expression (re) engine for python

def p1_reader():
  # Initialize the serial connection
  serial_connection = serial.Serial()
  
  # Load the DSMRv5 serial connection settings, see [2]
  serial_connection.baudrate = 115200
  serial_connection.bytesize = serial.EIGHTBITS
  serial_connection.parity = serial.PARITY_NONE
  serial_connection.stopbits = serial.STOPBITS_ONE
  serial_connection.xonxoff = 0
  serial_connection.rtscts = 0
  serial_connection.timeout = 12
  
  # Assign the address of the P1 cable. 
  # With the suggested cable, the addres on linux/unix like 
  # environments is /dev/ttyUSB0
  serial_connection.port = "/dev/ttyUSB0"
  
  # Close the connection to avoid issues
  serial_connection.close()
  
  # Loop forever
  while True:
    # Open the connection
    serial_connection.open()
  
    # Keep track of whether we have encountered the start 
    # or the end of the transmission.
    start_of_transmission = False
    end_of_transmission = False

    # Wait until the start of the transmission
    while not start_of_transmission:
      # Fetch data from serial connection
      try:
        serial_data = serial_connection.readline()
        serial_data = serial_data.decode('ascii').strip()
      except Exception as e:
        print(str(e))
        pass
            
      # Check if we find the / that marks the start of the telegram.
      if re.match(r'(?=/)', serial_data, 0):
        start_of_transmission = True
      else:
        start_of_transmission = False
        # We have not encountered the start of the telegram yet.
        # Skip the rest of this iteration of the loop. 
        continue
    
    while not end_of_transmission:
      # Fetch data from serial connection
      try:
        serial_data = serial_connection.readline()
        serial_data = serial_data.decode('ascii').strip()
      except Exception as e:
        print(str(e))
        pass
    
      # We have encountered the start of the message.
      # and we have not encountered the end of the message yet.
      
      # Try to match the line against the pattern.
      # We use the Regular Expression feature of a positive lookahead: 
      # see e.g. https://www.regular-expressions.info/lookaround.html for details 
      
      # Actual power delivered to the customer in Watt
      if re.match(r'(?=1-0:1.7.0)', serial_data):
        actual_usage = float(serial_data[10:-4]*1000
      
      # Actual power delivered back to the energy supplier in Watt
      if re.match(r'(?=1-0:2.7.0)', serial_data): 
        actual_delivered = float(serial_data[10:-4]*1000
      
      # ... et cetera
      
      # Check if we have reached the end of telegram
      if re.match(r'(?=!)', serial_data, 0):
        end_of_transmission = True
    
    # We have reached the end of the Telegram. 
    serial_connection.close()
    
# If the program starts and this is the main process, we execute the p1_reader
if __name__ == "__main__":
  p1_reader()

Bibliography

[1] P1Meter.nl, “De beste P1 Meter voor het meten van energieverbruik”, URL: https://www.p1meter.nl/beste-p1-meter/, Accessed: 29 june 2024

[2] Netbeheer NL, “DSMR 5.0.2 P1 Companion Standard – Dutch Smart Meter Requirements”, (2016), URL: netbeheernederland.nl/publicatie/dsmr-502-p1-companion-standard, Accessed: 29 june 2024