Round-Up of the Yocto Project Summit 2024.12

Episode 57

Dear Reader,

What makes Japanese people distinctively Japanese? For me, the key lies in what we are taught during our elementary school years.

Children as young as 6 are given the responsibility to clean their own classrooms and serve one another lunch. Schools are structured like mini societies, where everyone has a role and is expected to contribute to the community. There’s a strong focus on nonacademic education intended to teach teamwork, work ethic and a sense of accomplishment.

Ema Ryan Yamazaki, What a School Performance Shows Us About Japanese Education, The New York Times, 2024-12-22 (emphasis mine)

Over the last year, I got convinced that the embedded ecosystem could use a big dose of teamwork. I implemented an OTA update solution and secure boot. As a system integrator, I hooked up the application with the operating-system layer. The target device sports an iMX8M Plus with internal eMMC storage.

None of the OTA-update vendors provides a ready-made solution for the iMX8M Plus, which is one of the most common SoCs together with all the other iMX SoCs. I had to make sense of dozens of pages of documentation and find out why the client or server don’t work as expected and why they don’t work together. Of course, every vendor has its own Yocto layer. No vendor provides a proper API to perform updates. I had to create the API for my customer.

By ready-made solution, I mean that I specify the update client and server (e.g., SwUpdate/Memfault, RAUC/QBee, Mender or Torizon Cloud), the partitioning of the eMMC storage and the update strategy. Furthermore, all vendors should implement the same API, which customer applications would use to perform OTA updates. Vendors could provide OTA updates as a microservice.

The situation with secure boot is similarly bad. The SoC makers leave it to the SoM makers to provide a solution. Many SoM makers offer nothing, some a partial solution with some issues, and one or two a fairly complete solution. All solutions have in common that they only work for the SoMs of one manufacturer. Migrating such a solution from one SoM maker to another is a tough exercise with a high chance to fail. So, you are locked in to a specific SoM vendor.

The secure-boot solutions have something else in common. None of them helps you with storing and installing the private and public keys on the build server and on the device. I had to create a good-enough solution for my customer.

How can teamwork improve the situation? Teamwork would mean that all the players upstream their work. This would force these players to separate machine-dependent parts cleanly from machine-independent parts. The players in the embedded industry do not have to reinvent the wheel over and over again. Instead, they can incrementally improve the work of others.

For SoC makers like NXP and TI, it should be paramount to make the SoM, SBC and terminal makers give back their extensions and improvements to the upstream BSP layers or to the more general distro layers. The SoC makers should play an active role in integrating work into upstream layers to avoid fragmentation and lock-in.

ARM has created a company with Linaro that strives to make it easier and quicker for OEMs to bring embedded systems with ARM chips to the market. It is a huge task and Linaro could definitely use some help. We, the embedded industry, would benefit significantly from better teamwork!

Now enjoy my summary of five talks from the Yocto Project Summit 2024.12. The presenters have understood teamwork. These experts give you a lot of actionable advice for free. This advice will save you and your employers a lot of time and money. Please honour their generosity. Don’t try to squeeze even more free advice out of them, but think about how you can make the embedded industry a little bit better.

I wish you all a merry Christmas and a happy new year!
💜 Burkhard 💜

My Post

Embedded Devices Covered by EU Cyber Resilience Act (CRA)

The EU Cyber Resilience Act (CRA) entered into force on 11 December 2024. It covers all embedded devices. A self-assessment will be enough for most embedded devices. The more security-critical functionality a device contains the more onerous the conformity assessment will be.

The EU CRA concedes manufacturers a three-year transition period. Hence, manufacturers will face hefty penalties for violations of the EU CRA from 11 December 2027. One important obligation starts even earlier. Manufacturers must notify the relevant authorities about exploitable and severe vulnerabilities from 11 September 2026.

Round-Up of the Yocto Project Summit 2024.12

Iain Menzies-Runciman (known as “Ming”): Creating and Maintaining a Yocto Project in the Real World

As a long-time independent consultant for embedded Linux, Ming “spends his days building applications and systems for clients with the Yocto project”. He is working with multiple customers, boards and distros in parallel. The systems need updates in the field. When he leaves a project, the customer team must be able to maintain and update the system on its own.

This sounds a lot like what I’m doing day in and day out. So, I hoped for a couple of useful tips. And - Ming didn’t disappointed.

Ming uses the following project layout.

MyProject
build
layers
project
meta-my-bsp
meta-my-distro
meta-my-software
third-party
poky
meta-something
scripts

It is important to distinguish strictly between project layers that you can change and third-party layers that you must not change. Ming defines the project layers as follows.

• The layer meta-my-bsp contains “anything to do with board bring-up” like machine definitions and kernel and u-boot customisations.

• The layer meta-my-distro contains “anything to do with Linux user space” - including distro configurations (replacing Poky), users and system configurations. This layer consists of a lot of .bbappend files.

• The layer meta-my-software contains “anything created in-house”. It provides the recipes to install the manufacturer’s software.

Ming recommends to keep absolute paths out of bblayers.conf. So, this is very bad:

BBLAYERS = “ \
/home/burkhard/VeryBad/layers/meta-freescale \

Instead, you should give the layer directories as relative paths to the build directory ${TOPDIR}.

YOCTOROOT = ${TOPDIR}/../layers
BBLAYERS = “ \
${YOCTOROOT}/third-party/meta-freescale \
${YOCTOROOT}/project/meta-my-software \

Relative paths make it easy to share the Yocto setup with other developers.

Interestingly, kas creates bblayers.conf files with relative paths. If you use kas like I do (see my post Setting Up Yocto Projects with kas to get started), you don’t have to do anything.

For each project, you typically build a production and development image. The production image normally has a read-only root filesystem to enable delta updates, integrity checks with dm-verity and other security features. The development image better has a read-write root filesystem and includes extra tools for debugging, profiling, testing and more. Ming recommends a development image recipe similar to this.

require images/my-production-image.bb
IMAGE_FEATURES:remove = “read-only-rootfs”
EXTRA_DEV_PACKAGES = “valgrind gdbserver iperf3 vim ...”
IMAGE_INSTALL += “ ${EXTRA_DEV_PACKAGES}”

The recipe for the development image forces the inclusion of the recipe for the production image. It makes the image read-write by removing the image feature read-only-rootfs. Finally, it adds to the image some packages that must only be used during development. This recipe would also be the right place to add some image features like allow-empty-password, allow-root-login and debug-tweaks, which should never make it into the production image.

Anna-Lena Marx: Patching unpatchable File

With .bbappend, BitBake provides a mechanism to patch recipes (.bb files). But how do you patch .inc and .bbclass files? Anna-Lena answer’s will save you a lot of time.

If a recipe includes a .inc file with include or require, you simply create .bbappend file for the recipe. The .inc file is inserted verbatim into the recipe.

This leaves you with the hard case: .bbclass files. You have two options to patch a .bbclass file.

Bad Option - Copying the class file

You can copy the .bbclass from the original layer to your own layer and modify it. Of course, you are responsible to merge changes in the original class into your duplicated class. This works, if the .bbclass file is not used in the original layer. Otherwise, you must reorder the layer directories in the bblayers.conf file: The layer with the modified .bbclass file must precede the layer with the modified .bbclass file. Note that layer priorities are ignored.

Instead of copying the whole class file, you could create a class that inherits the original class, copy the function that needs changing and modify it. As you use Bash or Python for writing functions in recipes, the second function definition in the new class overwrites the first definition in the original class.

Especially long-term, this option is terrible. Hence, you should try to upstream your solution as quickly as possible - not only in the Yocto version you use but also in the Yocto LTS versions.

Good Option - Patching the class file with kas

What you probably would like to have is something like the SRC_URI mechanism for recipes and class files. By the way, you can’t use SRC_URI, as Anna-Lena explains in detail.

Fortunately, kas can patch metadata files in repositories. After kas has checked out a layer repository, it can patch the metadata files (see the entry for repos/patches in the section Configuration Reference). If, for example, you wanted to change something in swupdate.bbclass, you could extend the `meta-swupdate` entry in the kas project file with a patches entry:

repos:
meta-swupdate:
url: "https://github.com/sbabic/meta-swupdate"
commit: "91de63c006e886eba8d9c263c9aa55d4c7632072"
path: "layers/meta-swupdate"
patches:
swupdate-replace-placeholders-in-all-src-uri-files:
repo: meta-b4-ota-update
path: patches/meta-swupdate/001-swupdate-replace-placeholders-in-all-src-uri-files.patch

Kas applies the patch patches/meta-swupdate/001-swupdate-replace-placeholders-in-all-src-uri-files.patch from the repository meta-b4-ota-update to the file classes/swupdate.bbclass in the layer meta-swupdate.

This solution is a lot better than copying the class file. But it is not an excellent solution. The patch is in a different repository than the class file: a long-distance dependency too easy to overlook. You should still try to upstream your change.

Sergio Prado: Honey, I shrunk the rootfs!

You have two ways to reduce the size of the rootfs. First, you can remove unnecessary packages from the distro, machine and image. Second, you can reduce the size of the files provided by each package - by stripping binaries and by optimising the compilation for size.

The starting point is a build of the image core-image-minimal, the machine qemux86-64 and the distro poky with Yocto 5.0 LTS (Scarthgap). The resulting image has a size of 28 MB and contains 35 packages. After the size reduction, the rootfs must still boot successfully, provide access to the console and allow running common Linux commands.

Sergio starts by reducing the distro size by replacing the INIT_MANAGER sysvinit by mdev-busybox, by setting DISTRO_EXTRA_RDEPENDS, DISTRO_EXTRA_RRECOMMENDS and DISTRO_FEATURES to empty lists. This brings the size down to 26 MB.

The next target is the machine size. Sergio sets the variables MACHINE_ESSENTIAL_EXTRA_RDEPENDS, MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS, MACHINE_EXTRA_RDEPENDS, MACHINE_EXTRA_RRECOMMENDS and MACHINE_FEATURES to the empty list. The image still boots. The image size is now 15 MB.

You might not always get away with such a radical approach. Some of the packages listed in the MACHINE and DISTRO variables may be required. Sergio checks the variables values before the build and then decides which packages to remove. You can check the values of DISTRO and MACHINE variables with the utility bitbake-getvar. For example:

$ bitbake-getvar —value DISTRO_FEATURES
acl alsa bluetooth … waylaid vulkan sysvinit
$ bitbake-getvar —value MACHINE_FEATURES
alsa bluetooth … x86 pci

For reducing the size of the image, Sergio uses a more measured approach. He checks the relevant image variables IMAGE_INSTALL, CORE_IMAGE_EXTRA_INSTALL, IMAGE_FEATURES, EXTRA_IMAGE_FEATURES and IMAGE_LINGUAS with bitbake-getvar. As the values of these variables are specific to the recipe core-image-minimal, you must specify the recipe name in the bitbake-getvar call.

$ bitbake-getvar -r core-image-minimal —value IMAGE_INSTALL
packagegroup-core-boot

You cannot simply reduce packagegroup-core-boot, but only the unnecessary packages from this group. bitbake-getvar can help you with this.

$ bitbake-getvar -r packagegroup-core-boot —value RDEPENDS::packagegroup-core-boot
base-files base-passwd busybox netbase udev update-alternatives-opkg

Sergio decided to remove the packages udev and update-alternatives-opkg from IMAGE_INSTALL and keep the others. This reduces the image size to 6.6 MB.

Interestingly, both IMAGE_FEATURES and EXTRA_IMAGE_FEATURES have the value debug-tweaks. I would have expected Sergio to remove the image feature debug-tweaks, which must not be included in production images. I don’t know whether Sergio overlooked it or whether it doesn’t reduce the size any further.

By disabling all recommended packages with NO_RECOMMENDATIONS = “1” in a configuration file, the image size goes down to 5.7 MB.

bitbake-getvar is a pretty cool utility, isn’t it? Sergio introduces two other very useful utilities: oe-pkgdata-util and buildhistory-diff. Check them out!

The last reduction in size comes from software optimisation. You can replace glibc by the much smaller musl by setting TCLIB = “musl”. You can tell the compiler to optimise the size by setting FULL_OPTIMIZATION = “-Os -pipe ${DEBUG_FLAGS}”. The variable FULL_OPTIMIZATION is passed to CFLAGS for target builds.

Marta Rybczynska: Security improvements in Yocto 5.1 (styhead)

Marta is the security expert of the Yocto project. She is building a tool yocto-vex-check to create a list of CVEs (Common Vulnerabilities and Exposures) for each package in a Linux image built with Yocto and for the image itself. This list is the basis for conducting an analysis of the cyber security risks of a device - as required, for example, by the EU Cyber Resilience Act (CRA). For each unpatched CVE, you must decide whether the CVE is exploitable for your device. If a CVE is exploitable, you must deploy countermeasures and provide an update for your device.

Before you can run the tool yocto-vex-check, you must add INHERIT += “vex” to local.conf file and run a build. The build generates an SBoM in SPDX 2.2 format and a metadata JSON file in cve-check format. The vex class is a drop-in replacement for the cve-check class (available since Kirkstone), which enables the build to generate the resulting CVE list in VEX format (Vulnerability Exploitability eXchange). You must also download the National Vulnerability Database (NVD) or the CVE raw database.

Now, you are ready to run the yocto-vex-check tool with the SBoM. The tool takes the SBoM, the metadata JSON file and the one of the two databases as its input and generates an updated metadata JSON file and a VEX document as its output. The VEX document is the list of CVEs for your risk analysis. After the first run of the tool, the metadata JSON file contains a mapping between the Yocto package names and the names used in the CVEs.

The vex class is already part of Yocto 5.1 (Styhead). However, the tool yocto-vex-check is not yet part of an official Yocto release. This is on Marta’s to-do list.

At the end of her talk, Marta presents some of her own ideas how to improve security for embedded systems. I especially like her suggestion that “someone” should provide best-practice blueprints for secure boot and system updates. I would go a step further.

• The SoC and SoM manufacturers should provide a ready-made solution for secure boot including the secure generation and storage of cryptographic keys in hardware-security modules (HSMs).

• The vendors of system updates should provide ready-made and secure OTA update solutions for the most common SoCs and SoMs.

And dozens of pages of documentation how to implement secure boot or OTA update don’t count as a ready-made solution. These companies don’t care that each of their customers must spend the time and money to implement secure boot and OTA updates over and over again. That is not exactly a good customer experience! Linaro is trying to improve the situation, as the next talk by Mikko Rapeli shows.

Mikko Rapeli: Secure boot all the way to userspace and upstream

Mikko summarises what he and his colleagues at Linaro have been doing to get secure boot working “all the way to userspace and upstream”. They have got a lot working but not everything. And they weren’t able to upstream everything. But they won’t give up as Mikko’s conclusion shows.

The dream lives on…
Hopefully in 2025 there will be a working example of secure boot all the way to user space and TPM backed secure storage in an upstream maintained Yocto layer.

Mikko Rapeli

Linaro work on two projects: the Trusted Substrate (TS) and the Trusted Reference Stack (TRS) project. Here is a simplified description of the TS and TRS.

TS provides a reference implementation of an UEFI-compliant bootloader like U-Boot - supporting secure boot. The TS firmware can securely store and access keys and other secrets in a Trusted Platform Module (TPM). On power-up, a special boot hardware checks - with a public key retrieved from the TPM - if the signature of the bootloader is correct. Then, the bootloader checks whether the signature of the unified kernel image (e.g., the FIT image) containing kernel and initramfs is correct.

At this point, the job of the TS is done and the TRS takes over. The TRS extends the chain of trust into user space. The unified kernel image contains a dm-verity hash of the rootfs from the build. The initramfs calculates the rootfs hash at runtime and compares it against the build-time hash. If the hashes are the same, initramfs mounts the rootfs. Otherwise, it stops the boot process. The TPM can be used to manage an encrypted read-write partition like a data partition for customer data.

TS and TRS support devices based on the NXP iMX8M Plus, Raspberry Pi 4 (no TPM), Qualcomm RB1/2/3/5 robotics boards, qemu for ARMv7 and ARMv8 (with software TPM) and some more. You can build TS with the Yocto layer meta-ts and TRS with meta-trs. Note that Linaro weren’t able to upstream all their changes yet.

Mikko identifies one big unsolved problem: How shall device manufacturers handle private keys and other secrets at build time? When you run a Yocto build, you need the private keys to sign the hashes of the U-Boot, kernel and rootfs images. You better keep private keys secrete. The standard solution is an HSM (hardware-security module). Plugging the HSM into the compute server for every build or just keeping it plugged is cumbersome and insecure. Software APIs for HSMs could be an alternative during development.

I solved this problem differently for a customer this year. The keys and secrets are stored on an encrypted USB drive. Two people know the password for the USB drive. Before they run a build, they unlock the USB drive. The build picks up the keys and secrets from the USB drive. This method is not super-secure and not fully automatic, but it is secure enough for my customer. It waits for improvement.

It shouldn’t come as a surprise that SoC and SoM manufacturers provide absolutely no support for storing the keys needed during build and for installing keys and secrets on the device. They leave their customers high and dry. One of these customers may be you!