Monitoring electricity usage at the meter

Updated on .

I’d like to know how much energy I’m using to ensure that the baseline load for my activities is reasonable. Since there’s already a meter attached to my electrical service for the purposes of billing, it made sense to use that instead of adding monitoring devices to my electrical circuits.

Unfortunately, it’s difficult to correlate instantaneous usage with the inherent delay involved with the data showing up on my utility’s (Pacific Gas & Electric) website. PG&E’s smart meters can use Zigbee to stream data, which seems relatively rare in the US 1 Double-check that your electricity meter and utility support wireless access before taking this approach and buying any products.. I bought a Rainforest Automation EMU-2 and connected it to a Raspberry Pi that sends realtime data to a self-hosted Grafana dashboard. When that device stopped working, I also registered for self-access PG&E’s Share My Data service and wrote a service to send historical data to InfluxDB.

Rainforest Automation EMU-2

The EMU-2 transmits usage data over a USB serial port at a prescribed rate, about once every 10 seconds. The instantaneous demand can be queried with a request packet, and even settings like the data rate changed, but these features don’t work well in practice. The EMU-2 doesn’t queue writes to the serial port and an interrupt on the device to emit a scheduled response will clobber whatever was in the process of being written. All of the serial interfaces I found rejected invalid XML data to account for this and then retried the request. This is most apparent when first connecting to the device.

The company has an abandoned Emu-Serial-API project on GitHub to parse the XML output with Python 2. Despite the shortcomings of the interface, emu2influx uses it to push metrics to InfluxDB. This is what I use in my home network to visualize energy usage. The most promising 3rd-party interface is Emu-Api (known as emu-power on PyPi), written in Python 3. The RAEdataset project also has a very simple EMU2_reader.py script, if full control isn’t necessary.

Luckily, PG&E supports this device, despite not being listed as a HAN product.

PG&E Share My Data

PG&E allows commercial vendors to request access to their customer’s data with a service called Share My Data. The documentation is centered around the corporate use case, but it does include references to self-service usage under Access Your Own Data. The process is a bit complex and might require setting up a server for any bulk asynchronous data requests to be delivered, but I was able to at least register for the service by completing their testing.

The steps outlined by PG&E might make sense to someone well-versed in web technologies, but I had to navigate it by feel and many web searches. My first obstacle was generating a public/private key pair that PG&E’s endpoint expects for Mutual TLS (mTLS). It seems like this is a common approach for business-to-business communications, but I had never run across it before. I followed whatever online tutorial I could find to generate the key pair, which also required setting up a root certificate authority to sign the certificate. I ended up with key pairs for the server itself and the certificate authority, along with a certificate signing request, and some serial file that was generated during the signing process. The only part that’s needed to communicate with PG&E is the server key pair – the rest are artifacts of the process (which you probably should still keep around somewhere safe).

Authentication

When signing up to access the data, PG&E will also grant you an OAuth client ID and client secret that are used to bootstrap each session using HTTP basic authentication. I wrote my data access service in Go due to the built-in HTTP client and general attitude of doing the simplest thing possible. The first step is being able to authenticate and acquire a bearer token, which can be used in subsequent requests. 2 It should be abundantly clear that I don’t understand the constraints or goals of OAuth. I guess the bearer token allows other servers that handle business logic to operate without access to any means of permanently authenticating themselves.

Right off the bat, I ran into trouble getting the standard library to actually send the client certificate. To rule out my code, I tried the request another client framework and ended up with the following curl command:

curl --verbose --request POST \
	--cert <path-to-crt> --key <path-to-key> \
	--basic --user <client-id>:<client-secret> \
	'https://api.pge.com/datacustodian/test/oauth/v2/token?grant_type=client_credentials'

That did work, and when I removed the --cert and --key options, I saw the same behavior as my Go program. All the tutorials on mTLS in Go recommend setting a certificates list in the HTTP client object when it’s created, like this:

http.Client{
	Transport: &http.Transport{
		TLSClientConfig: &tls.Config{
			Certificates: []tls.Certificate{cert},
		},
	},
}

Where cert is a certificate created from LoadX509KeyPair or X509KeyPair from the crypto/tls library. This doesn’t work for self-signed certificates because the Go standard library ensures the certificate being provided is acceptable to the server before sending it. 3 Go’s client implementation calls SupportsCertificate on the Certificates list in the TLSClientConfig, which uses the CertificateRequestInfo’s AcceptableCAs to ensure at least one of them is in the signing chain. It makes total sense, though it would be nice if there was a field that could augment the server-provided list of acceptable CAs. The correct way to do this when a server doesn’t claim to accept a certificate authority in the signing chain of the client certificate is to use the GetClientCertificate field of the tls.Config. Putting all of the authentication together, but eliding any error handling:

import (
	"crypto/tls"
	"encoding/json"
	"log"
	"net/http"
	"time"
)

type tokenCredentials struct {
	ClientAccessToken string `json:"client_access_token"`
	ExpiresIn int `json:"expires_in"`
}

func authenticateWithPGE(cert tls.Certificate,
		clientID string, clientSecret string) (
		token string, expiration time.Time) {
	client, _ := &http.Client{
		Transport: &http.Transport{
			TLSClientConfig: &tls.Config{
				GetClientCertificate: func(
						info *tls.CertificateRequestInfo) (
						*tls.Certificate, error) {
					return &cert, nil
				},
			},
		},
	}
	req, _ := http.NewRequest(http.MethodPost,
			"https://api.pge.com/datacustodian/test/oauth/v2/token?grant_type=client_credentials",
			nil)
	req.SetBasicAuth(clientID, clientSecret)
	now := time.Now()
	resp, _ := client.Do(req)
	defer resp.Body.Close()
	if resp.StatusCode != 200 {
		errorBody, _ := ioutil.ReadAll(resp.Body)
		log.Fatalf("PG&E returned %d %s response, body: %s",
				resp.StatusCode, resp.Status, errorBody)
	}
	dec := json.NewDecoder(resp.Body)
	data := tokenCredentials{}
	_ = dec.Decode(&data)

	token = data.ClientAccessToken
	expiration = now.Add(time.Duration(data.ExpiresIn) * time.Second)
}

Any further requests until the expiration date just use the same client certificate setup for mTLS and an Authorization header field with a Bearer <token> value.

Other wireless devices

PGE only lists a few certified “HAN” products but other utilities have more up-to-date lists, like the one from SDGE.

I haven’t seriously considered building my own hardware interface to the meter. The ecosystem around the Smart Energy Protocol in Zigbee seems locked down tight, with “production” certificates required to communicate. When setting up the reading device with my utility, they asked for a MAC address that presumably is configured with the meter. It seems like this protocol was built around centralized cryptography to prevent third-parties from snooping on energy usage data.

I was able to find a few resources on Zigbee and Smart Energy, though. It’s unlikely a board based on the Nordic nRF52840, despite it supporting Zigbee, would be able to communicate with the smart meter, since it lacks certification. Texas Instruments has a guide on using one of their modules to monitor Smart Energy packets.

Some smart meters can be read using software-defined radios. The most up-to-date software package I found was rtlamr for Itron ERT meters but I have no experience with it. The Home Assistant Community seem full of people willing to tinker with this sort of thing. The RECESSIM wiki’s page on Advanced Metering Infrastructure also seems useful.

Discontinued

On the meter

I found a couple of devices that need physical access to the meter to monitor it, but I treated these as last resorts if the wireless options didn’t pan out.