This has been brewing for a while, but I finally put it into words during this pandemic: if you're an embedded software engineer, the ability to work remotely (without the hardware next to you) is in fact a sign that you have reached a certain level of software engineering maturity.

Automation

Being able to automate your setup, opens the door to many things: first, automation frees the mind of the menial tasks. I've worked on projects where we had dedicated reset button on the board. Very fast to reboot, especially when you're writing bootloaders or debugging crashing kernel drivers. And very practical. But I couldn't be more wrong.

What's needed is to control the power supply. A reset button is just a bonus, but you need to be able to control the power supply. And if the device does not power on automatically, you need a way to control that, too. On all the recent projects I worked on (STB and GW), the board would turn on automatically on electrical power on. This means you can go cheap and order off-the-shelf USB-controlled power switches; otherwise, it's always possible to build more complex setups, but I like that this uses a standard power supply, up to the wall socket.

I'm assuming you always have a serial port plugged to your hardware; if not you might need one, or a similar facility.

Automation means you can track a hard to reproduce bug that only happens after an electrical reboot. It means you can try to reproduce those hangs easily. It means you can work around bugs that only happen in developer mode when the workaround is easy to automate and much cheaper than the fix.

And it means that in case of a pandemic you can continue working from home, with your hardware in a remote lab or at the office, as if (almost) nothing changed.

sispmctl

The energenie power switch is controlled with the readily-available sispmctl package in most distros. To be able to use it as user from the dialout group, I use the following udev rule:

SUBSYSTEM=="usb", ATTR{idVendor}=="04b4", ATTR{idProduct}=="fd13", GROUP="dialout", MODE="660"

You can substitute the appropriate vendor and product ids.

I use a simple script to power-cycle port 1 by default, or the port passed in argument:

#!/bin/bash
PORT=${1:-1}
#off
sispmctl -q -f "$PORT"
sleep 2
#on
sispmctl -q -o "$PORT"

For maximum efficiency and saving a few keystrokes I usually have a keyboard shortcut to reboot the port I'm currently working on, mimicking a reset button, but without moving from your keyboard.

ser2net: serial port automation

Why do specific automation of the serial port, isn't reading from a tty trivial ? Well, almost. ser2net's initial goal was to make a serial port available over the network, meaning you can remotely connect to a board that is plugged to a different lab computer, without the need for local access on said computer. But this isn't the killer feature of ser2net. Since version 3.2 (now packaged in all distros), ser2net allows multiple clients to connect to a single device.

This means you can have a program that does trivial socket read/write (in telnet mode) to automate a task, and at the same time use the device from your terminal.

Here is how I configure it in /etc/ser2net.conf:

localhost,2000:telnet:0:/dev/blueserial:115200 8DATABITS NONE 1STOPBIT banner max-connections=5

Here I can use the "blue serial port" — I always give my serial ports a name: it usually refers to the color of the cable or the board I'm using — and it will listen in "telnet mode" on port 2000 on localhost only. It can support at most 5 connections in this configuration. You can then connect to it with telnet localhost 2000.

As a parenthesis, here is the udev rule to give the serial port a name:

SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", ATTRS{serial}=="FTA6371G", SYMLINK+="blueserial"

I often have multiple serial adapters with the same usb ids, hence the use of the serial number to differentiate them.

Testing

Improving the everyday quality of life is only half the story. Now that you have superpowers, you can use those to enable another superpower: do automated testing.

You might have a testsuite that needs to power-cycle the board ? Very simple to do. You need to check the output of the serial port ? Very simple to automate with a socket.

Continuous Integration

And once you plug this to your continuous build infrastructure, you can start running your integration tests directly on hardware using the latest version of your software. I won't explain how CI works, but it's probably something you want to have.

LAVA can be useful if you want to build a virtual lab, as part of the chain that manages your board farm, and inside your CI loop; it can integrate with ser2net and your power switch, take your Jenkins artifacts and load them on boards to run your testsuites. You can find many articles and talks on why and how to use LAVA, but I recommend Bootlin's articles as starting point. You can even combine it with lavabo to add board reservation for lab shared with a team.

Not everything can, or should be done remotely (yet)

Working on embedded means you're always close to the hardware. There are things that just can't be automated without paying a very high cost/benefit ratio. But that ratio is much higher than most people think. For example, if you want to go into mass production and care about your yield, you might need to setup a testing bench for a sample (or even all) of your output. Any automation that you do upfront can be reused in the factory, since it's something that will need to be done anyway. And any setup giving you a view of a production board (test points, measurements, etc.) can be reused for a remote setup.

Sure, you won't be doing a board bringup with the hardware in remote lab. Nor would you be doing certain types of hardware enablement. But that does not mean that you shouldn't at least try to get the low hanging fruit.

And if you want higher software quality, and higher development velocity, investing in tooling can help you get there.