So I was in Edinburgh this year, and I took notes as I usually do. These are intended for personal consumption (do no expect LWN-style reports), but as more people were asking me to share them, I thought why not do it in public ?

Embedded Linux timeline

by Chris Simmonds

Busybox was started by Bruce Perens to solve the floppy installation problem. The first Linux Router was described in "Arlan Wireless Howto".

Linux gained portability to other architectures over time:

• 1995: MIPS
• 1996: m68k, ppc
• 1998: m68k Dragon Ball Palm Pilot : creation of uClinux (no mmu)
• 1999: ARM

Flash memory support was added by David Woodhouse in 1999 (MTD layer), then JFFS by AXIS for their IP cameras.

Devices

Things really started in 1999: AXIS IP camera, TiVo DVR, Kerbango Internet Radio(Threecom). Lot of media coverage at that time.

Companies sprung out to service embedded linux: Timesys, MontaVista, Lineo, Denx.

The handhelds.org project aimed at porting of linux to Compaq iPaq H3600. Cross-compiling being a pain in the arse, they had a cluster of ~16 iPaqs to compile code.

In 2001, there was the infamous Unobtainium : handset prototype at Compaq based on iPaq hardware with GSM/CDMA/Wifi/Bluetooth, camera, accelerometer, 1GiB of storage: it was really the first smartphone prototype. Never shipped.

At the same time, Sharp made the Zaurus running Linux 2.4.10 (software made by Lineo).

In 2003, Motorola made the A760 handset, first Linux handset (MontaVista) In 2005, the Nokia 770, the first Internet Tablet running Maemo Linux.

Buildtools and software

In 2001 buildroot was created from the needs of the uClinux project: it's still the oldest and simplest build system. Then OpenEmbedded in 2003. Then in 2004 Poky Linux based on OE by OpenedHand, then Yocto. In my opinion Chris has a narrow view of the build systems choice (what about Debian?)

Real-time was at first achieved with sub-kernels, like Xenomai. Then Native Real-time : Linux/RT 1.0 by Timesys in 2000. Then the voluntary preempt patch (Ingo Molnar & Andrew Morton). Robert Love kernel preemption patch in 2001. In 2003 Linux 2.6 includes voluntary preempt. In 2005 PREEMPT_RT was started, in 2013, not all of it is merged yet.

In the end: Linux is the "default" embedded OS.

How not to write x86 platform drivers

by Darren Hart

This talk was mostly a feedback around getting the Minnowboard mainlined properly.

At Intel a platform is CPU + chipset, or a SoC. In Linux, it represent things that are not on a real bus, or things that cannont not be enumerated, leading to board fils drivers.

The Minnowboard uses a 32bit UEFI Firmware. One of the first designs to make use of all Queensbay(Intel SoC) GPIOs. The UART clock is special (50mhz). Low-cost Ethernet phy with no EEPROM for macs. The Minnowboard is a dynamic baseboard, which is very different from what Intel usually does: it supports daughter cards.

There are three main sources of GPIOs on this board (5 core, 8 suspend, 12 pch), 4 user buttons, 2 user LEDs, phy reset, then expansion GPIOs.

Board files

MinnowBoard used board files at first because they are simple to use.

Those were rejected. Why ?

• not automatically enumerated and loaded
• adds maintenance
• independent drivers had to be board aware

All this leads to "evil" vendor trees.

UART clock is firmware dependent. Previous code used DMI detection, which isn't nice.

Ethernet is complicated: aggressive power saving meaning you must wake it up open. How to id the PHY ? You could use SMBIOS/DMI, DT, ACPI; in the end PCI subsystem ID were used. Initialized with platform_data.

The MAC: no EEPROM, so had to solve how to get a MAC. Was done in firmware in the end: read the SPI flash, then write PCI registers.

To preserve the platform on should not create vendor trees. The complexity of core kernel vs drivers is inverted: core kernel has simple primitives, but complex algorithms. Drivers are the opposite: simple to understand, but hard to organize, how they fit together.

GPIO, take 2: ACPI 5.0

A lot of things can be done with the new ACPI standard, like identify GPIO resources. You can't do keybindings, default trigger, etc. Some vendors (like Apple) do already, but with their own proprietary additions.

One needs to write ASL for the DSDT. You might want to have dictionaries to describe your hardware, which needs to be standardized. Right now ACPI reserved method are used (_PRP).

Device trees for dummies

by Thomas Pettazoni (Slides)

Before DT, all the information was inside the kernel. Little information in ATAGS. Now all information is in DT.

Device Tree is a tree data structure to describe hardware that cannot be enumerated. You compile a DTS (source) into a DTB(binary). In arm, all DTS are in arch/arm/boot/dts and automatically compiled for your board. Device Tree bindings is the "standard" for how you should describe the hardware using the DT language, and what the driver understands. All bindings should be documented and reviewed. DT should not describe configuration, just hardware layout. The problem isn't solved yet for configuration.

The talk had a lot of nice syntax examples to learn how to write DTS. For example, Thomas explained the importance of compatible string, which are used to match DTS node device with a driver.

Should DT be an ABI ? Hard question. While it was the original idea, maybe it shouldn't. Current discussions seem to want to relax the stable ABI rule.

Use case power management

by Patrick Titiano (Slides)

First rule about PM: shutdown anything not used. You need to track running power resources: it starts with the clock tree.

Things to monitor:

• C-states/idle states stats.
• Operating point statistics
• CPU & HW load
• memory bandwidth : most often a bottleneck

You need to instrument both software and hardware. It means you need resistors points in the PCB, temp sensors. You need to automate everything, otherwise you're not comparing apple to apples. All measurements should be automated to be easily reproduced.

You need to have power model of your raw soc consumption and characteristics, then you need to assess this model to verify that the target is realistic.

Voltage is more important than frequency. It's easier to reduce consumption than to find better way to dissipate energy.

Battery is king. You need a full system view, because you should optimize the biggest offenders first. Take care of inter-dependent stuff.

Android debugging tools

by Karim Yaghmour (Slides)

Android usually runs on an SoC. It uses bionic, a different libc, it has a Hardware Abstraction Layer that allows proprietary drivers in userspace. Toolbox is lesser busybox clone in BSD.

Binder is a standard component, object IPC, that almost defines system android. Every system services uses that.

To debug, it's handy to load AOSP in eclipse. You can load all the OS classes and apps in the editor to be able to trace anything and browse AOSP while seeing classes, call chains, etc. It's more powerful for browsing and live debugging than common editors. You still have to build AOSP by hand (type make/lunch) to generate an image.

A few tools:

• latencytop, schedtop, etc.
• dumpsys
• service
• logcat
• dumpstate (root app), bugreport : dumps system state (in /proc, etc.)
• watchprop

Logging goes through the logger driver.

Interfaces with the system:

• start/stop : stops zygote, which means you can shutdown/start the interface and all the java stuff
• service call statusbar {1,2,5 s16 alarm_clock i32 0} : you can call methods directly that are defined in an aidl file, by using their implicit sequence number. It's useful to bypass the java framework and call the services directly. see in android/os/IPowerManager.aidl for example, or android/internal/statusbar/IStatusBar.aidl for the previous example
• am : tool to call intents. e.g am start -a android.intent.action.VIEW -d http://webpage.com . Very powerful tool to call intents
• pm : calls to the package manager
• wm : calls to the windows manager

When working with AOSP sources, source build/envsetup.sh, it has very handy functions, like:

• godir: jumps to a dir a file is in
• mm : rebuilds the tree
• jgrep/cgrep/resgrep : grep for specific files (java/c/resource)
• croot : jump up to aosp root

Take care when working it AOSP, it's BIG (about 8GB)

When debugging, you have to use a different debugger depending on the use case:

• ddms for dalvik level stuff
• gdb/gdbserver for HAL stuff
• JTAG for kernel

DDMS talks JDWP (Java Debug Wire Protocol). Use the one from AOSP, not eclipse. It's very powerful to debug (java) system processes live.

gdbserver: you have to configure your app's Android.mk to have -ggdb, and disable stripping. You also have to do port forwarding with adb in order to access gdbserver:

adb forward tcp:2345 tcp:2345

You can use the prebuilts arm-eabi-gdb, but Multi-thread might not be supported.

logging

• logcat works
• ftrace is supported through systrace
• atrace (device dependent ?)
• perf is not well supported on ARM

Embbedded build systems showdown

A nice panel, where we had representatives of different build systems: - DIY : Tim Bird, Sony - Android : Karim Yaghmour, Opersys - Buildroot : Thomas Petazzoni, Free Electrons - Yocto : Jeff Osier-Mixon, Intel

It was very friendly. The take-out is that each system addresses different use cases. Yocto is big-company friendly, because it has metadata and licence management built-in. Tim Bird said he had a personal preference for Buildroot as a developer, although a division of his company recently switch to Yocto for its projects.

Karim's opinion was that although Android wasn't community friendly, that it couldn't integrate with anything external, it was king in term of market traction, and that it might be the most used system in any kind of embedded device in 4 to 5 years.

Best practices for long-term support and security of the device-tree

by Alison Chaiken (Slides at Author's)

DT make life a bit easier, although there are pitfalls. Best practices could help with that matter.

Updates are hard without DT. How about with DT ? Should you update the DTB ?

DTs are supposed to be for HW description, but there already many configuration items in DT: MTD partition tables, boot device selection, pinmux. Alison gave example about automotive and battery technology that's evolving that would allow updating electric car's battery. Cars have a lot of processors; e.g 2014 Mercedes S-Class will have 200MCUs on Ethernet, CAN-FD and LIN.

One thing to be careful about is Kconfig and DTS matching.

One pitfall you might have, is unintentionally breaking DT or device by changing something in a driver or another device. Example about Koen Kooi's post who said you might blow an HDMI transceiver on some board if you boot with micro SD, because micro SD uses a higher voltage by default.

You can use .its to bundle DTS, kernel and other blobs in one .itb file. Support was added in u-boot to sign .itbs by ChromeOS engineers.

One option floating around, presented by Pantelis, is to use DTS runtime overlays as an update method, similar to unionfs.

DTS schema validator looks like a good thing to have, like Stephen Warren's very recent proposal.

Android on a non-mobile embedded system

by Arnout Vandecapelle

The main motivation is the reduced time to market, and the wealth of available app developers.

It's interesting because it's still linux, but there are few differences (bionic libc, special build system)

My own impressions: lots of generic stuff, from someone who just recently went into android. Like most of this stuff, doesn't come from Google, so it has little "new" information in it. It was a nice conference if you've never heard of AOSP and have only been using other embedded distros/build systems.

BuildRoot : What's new ?

by Peter Korsgaard

BuildRoot

BuildRoot is an Embedded Linux build system. It's one of the oldest out there, and is fairly well documented. It has an active community. It's relatively simple, and that's a focus of the project to Keep It Simple (Stupid). For example, there's no binary package management.

It's Kconfig-based for configuration, and uses make for building.

Buildroot is package-based, and a build step just runs every package build. It's therefore a meta build system. A package is composed of :

• a config (in kconfig format) for dependencies, description, etc. You need to include this config under the parent config option.
• a makefile (Package.mk) with the build steps

Buildroot is using git for its source code, patches are posted on ML and managed in Patchwork.

Buildroot activity has been growing over the years: more emails on ML(~1000/month), more contributors (30-40 each month). Developer days are held 2 times a year (this year at FOSDEM and ELCE).

Buildroot is used in many products(Barco, Google fiber), and SDKs (Atmel, Synopsys, Cadence, Imagination…)

What's new ?

It supports more architectures (ARC, Blackfin, Microblaze, Xtensa…), and the variant support has been improved (ARM softfp/hardfp/neon…), as well as the nommu support.

Buildroot now supports more toolchains: C library (glibc, eglibc, uclibc), and external toolchains.

Buildroot has 30% more packages than last year. A lot of stuff has been added (gstreamer, EFL, wayland, systemd, perf, python3, nodejs…). A GSoC student worked on adding ARM proprietary GPU and video drivers support.

QA has been improved as well, with continuous integration/regression testing. The development cycle is now 3 months, with 1 month of stabilization.

License compliance has been added: every package should have a license, and "make legal_info" generates all the necessary stuff.

There's a new Eclipse CDT plugin. Popular boards got their own defconfigs to ease starting.

A lot of configuration options were added to the menuconfig. New options to add a rootfs overlay, or last-second hook scripts.

Upcoming work includes external packages overlays, SELinux support, updated systemd/udev, and whatever else gets submitted.