Embedded Online Conference 2024: My Summary

Episode 53: Better Built By Burkhard

Dear Reader,

The Embedded Online Conference 2024 took place last month. For the price of $295, all the videos and live Q&A sessions are now available. In this newsletter, I want to give you a taste of the conference: I summarised the talks by Jacob Beningo, Steve Branam and Kate Stewart for you. I learned a lot by watching them and even more by summarising them.

I am pleased that I contributed a talk to the conference as well: The Ports-and-Adapters Architecture for Embedded HMIs. The live Q&A session moderated by Jacob Beningo was fun. My blog post Ports-and-Adapters Architecture: The Pattern from last summer covers large parts of my talk for free.

The conference features many more great talks. My watchlist includes the following talks:

• James Grenning, SOLID Design for Embedded C. James explains how to write better C code by following the Single Responsibility, Open-Closed, Liskov Substitution, Interface Segregation and Dependency Inversion Principle.

• Andreas Jarosch, Build your Own Embedded Middleware. Learning about design patterns for (automotive) middleware is just my cup of tea and should be a good complement to my talk about the ports-and-adapters architecture.

• Ben Saks, Modern C++ Interface Design. Ben improves a C++ interface predating the modern era with modern C++ features like scope enumerations, attributes, structured bindings, std::optional and std::expected.

• Sergio Prado, Power Management on Linux: From the Hardware to the Kernel and User Space Interfaces. Sergio takes on a tour of how to reduce the power consumption of embedded linux devices with frequency scaling, idle states, wake-up sources, sleep states, suspend-to-RAM, hibernation and other features.

• Marian Petre, Beyond Coding: Toward Software Development Expertise. Marian lets us in on her research what distinguishes expert developers from the rest and which practices lead to better software.

• And more…

My OTA update project came to an end last month. I implemented the last puzzle piece: OTA updates of u-boot. I wrote about the fundamentals in the post Updating U-Boot with an A/B Strategy. This update method should work for all SoCs that implement their NAND flash with version 4.1 or newer of the eMMC specification. As version 4.1 is from 2007, most SoCs released over the last five years should have two eMMC boot partitions. But I can’t quantify it yet.

Happy reading,
Burkhard

Embedded Online Conference 2024: My Summary

Jacob Beningo: The Embedded Frontier: Modern Trends and Practices

In his keynote, Jacob takes about six trends for embedded software development: programming language trends, build systems and configuration, AI & ML development techniques, lifecycle automation with DevOps, testing and simulation, modern software architecture. I could have easily written my whole newsletter about Jacob’s keynote and Q&A session. But I also want to give other talks their due. So, I’ll limit myself to three key take-aways.

Modern Software Architecture: Bottom-Up vs. Top-Down Approach (39:45 - 44:22)

We all develop layered embedded software architectures for the most part today. The way we think about software architecture is still too coupled.

Jacob Beningo

Jacob identifies the traditional bottom-up approach as the reason. Developers tend to think first about how to talk to peripherals over I2C, SPI or UART, or which protocol to use for the communication with the IoT cloud. Some teams spend months on implementing drivers just to find out that their customers want to do things differently.

Instead, developers should use a top-down approach. This gives them the chance to get early feedback from their customers and avoid spending lots of money for the wrong features. Good abstraction layers separating the application and the hardware layer are a must. You can simulate the hardware not only to get early feedback but also to test large parts of your code right from the beginning.

The software folks always get blamed for being late. The hardware guys get all the time they need to develop the hardware. If they deliver late to us, guess whose fault it is that the product is late. It’s the software guys’.

Jacob Beningo

Starting top-down, the “software guys” can avoid the blame, because most of the software is already in place once the hardware arrives. Some hardware-dependent parts may be missing, but they can be added quickly without changing the rest of the software.

I couldn’t agree more! Coupling is the bane of (embedded) software! The dependency inversion principle (DIP) is your best tool to achieve decoupled software with well-defined abstractions.

Code should depend on things that are at the same or higher level of abstraction

Brett L. Schuchert, DIP in the Wild

The application layer must not depend on the hardware layer directly. Instead, it must depend on the hardware abstraction layer (HAL), which is on the same abstraction level. The hardware layer depends on the higher-level HAL. If the HAL just forwards the functions and structure of the hardware layer and hence is on the same level as the HAL, this is a violation of the DIP. Your code will become harder to change.

Modern Software Architecture: Microservices-Like Tasks (44:23 - 45:52)

In the embedded space, microservices are implemented as tasks to leverage the capabilities of the RTOS.

Jacob Beningo

Thinking of tasks as microservices gives tasks well-defined interfaces. Your software becomes a system of decoupled and cohesive tasks. Another positive consequence is that you can reuse tasks more easily. Jacob emphasises that tasks should not be as tiny as microservices. Tasks are a bit bigger - more like centi- or milli-services.

In my talk The Ports-and-Adapters Architecture for Embedded HMIs, I come to the same conclusion. I suggest to extract the adapter into a service running in its own process. The main application and the service use inter-process communication like DBus, gRPC or QtRemoteObjects to talk to each other over a well-defined interface. Such a service could, for example, implement OTA updates or J19393/CAN message handling. You could even move some of these services from the application processor to the on-board microcontroller.

AI & ML Development Techniques (22:15 - 31:33)

AI and ML are going to be a key driver, a game changer for our industry. A lot of embedded software developers don’t have the time they need to fully develop the products they have been asked to develop - within the [allotted] time frames and within the budgets. In order to help close that gap, we are going to have to leverage […] AI and ML.

Jacob Beningo

From a conversation with Jacob and from reading his newsletter, I know that Jacob is seriously experimenting how to develop more productively using generative AI. In his talk (29:07), he lists ten (!) tasks for AI assistance. I find the following most promising:

• Writing […] documentation for specific hardware components.

• Providing code snippets for common embedded system tasks.

• Generating skeleton code for communication protocols (SPI, I2C, UART).

• Reviewing existing code and suggesting improvements or refactoring.

• Simulating responses from different hardware components for testing.

• …

Jacob Beningo

Jacob gives an example (29:15) using GitHub Copilot: how to create a Dockerfile that allows you to build firmware written in C/C++ for a Cortex-M4. The result isn’t error-free, but can be fixed easily.

This example motivates me to try out something more elaborate: “Dear AI, how can I set up Docker so that I can clone private GitHub repositories inside the Docker container?” - It took me a couple of days to find the solution: Maybe, the AI is clever enough to find my article Accessing Private Git Repositories from Docker Containers.

My conclusion: AI can best help you with problems that have been solved many times before. If AI reduces the time spent on tedious tasks, if it suggests solutions that just need few corrections, or if it provides search on steroids, that would be fantastic.

Steve Branam: Example of BDD Style TDD For Embedded System Software

Dragons (6:37-11:34)

In the past, I associated BDD (behaviour-driven development) with acceptance tests, heavy theoretical frameworks like Domain-Driven Design (DDD) and heavy technical frameworks like FITNesse and Cucumber. This is not entirely wrong, if you look at common definitions of BDD (see here and here, for example). However, this is not the essence of BDD. Steve’s definition of BDD cuts to the chase.

BDD is good TDD avoiding the dragons of poor TDD.

Steve Branam

If you are doing TDD already, you can move to BDD or “good TDD” by avoiding bad TDD practices (the “dragons”) and by adding some discipline to writing tests (the BDD structure of given-when-then). That’s good news for TDD practitioners. Heeding Steve’s recommendations, you can step up your game and write BDD tests that non-developers like product owners, customers and managers can understand and that can serve as both as a human-readable and executable specification.

What are “the dragons of poor TDD”? Steve lists four:

• Dragon #1: Tests are written to the implementation. If you change the implementation, the tests will break.
  Avoid by writing tests to the interface, where the tests describe the behaviour.

• Dragon #2: High code coverage doesn’t imply high quality.
  Avoid by covering all behaviours of the code under test (CUT), which implies coverage near or at 100%.

• Dragon #3: Developers shouldn’t write tests for their own code. Testers should do. This leads to test-last development (TLD) or debug-later programming (DLP), which both waste lots of time.
  Avoid by coding to intention: “Your code should simply do what the tests say.” Without a clear objective, the tests, you would be writing code by trial and error. This reminds me of the thought experiment that a monkey typing on a computer keyboard will eventually produce a work of Shakespeare - given an infinite amount of time (see The Mathematical Case for Monkeys Producing Shakespeare—Eventually for the odds).

• Dragon #4: TDD is the only testing method used in a project.
  Avoid by considering TDD as the enabler of testing (remember: “TDD is a development tool, not a testing tool.”). TDD makes your code observable and controllable. Hence, TDD makes it easier to test other aspects like concurrency, performance, resource consumption and security. TDD tests should not cover these other aspects.

BDD Structure (23:26 - 34:15)

At this point, I thought: “Well, I know these dragons. Sometimes I avoided them, sometimes not. I could certainly use some help to reduce the number of scars.” - And, The three-part BDD structure for test cases provide this help.

• Given initial conditions,

• When CUT performs behavior,

• Then verify expected results.

Steve Branam

BDD forces you to think before you code. What is the starting point? What is the trigger for the CUT to do something? What is the expected result? If the result is unexpected, either your expectation is wrong (sometimes) or your implementation is wrong (much more often). Anyway, you get very early feedback, whether you are on the track. Your mistakes are cheap and quick to fix this early.

Focusing on exactly one clearly defined use case lets you cut through complexity. You can solve one use case after the other in an incremental fashion. You don’t risk the cognitive overload of juggling multiple use cases in your mind at the same time. You have a much higher chance to come up with a working solution and to avoid long debugging sessions later (think DLP).

A test case (43:02) - following the given-when-then structure - for bit-banging bytes over UART to a computer using the 8N1 serial line protocol could look like this:

TEST_F(Given_OpenBBSerialTx, When_WritingByte_Then_WriteStartAndStopBits)
{
BBSerialTx_WriteByte(0u); // When
EXPECT_TRUE(is_valid_framing()); // Then
}

The arguments of the GoogleTest macro TEST_F - the test fixture Given_OpenBBSerialTx and the test name When_WritingByte_Then_WriteStartAndStopBits - feel a bit verbose, but actually give an easily understandable description of what the test case is doing: Given that a serial connection is open, when the CUT writes a byte, then a valid frame has 8 data bits, no parity, 1 start bit and 1 stop bit. You find a lot more examples in the section Test Suite Walkthrough (34:15 - 44:29).

The test fixture is a setup function, which drives the test case into a certain state (here: the serial connection is open). The Given part or fixture should only use API functions of the CUT. It should avoid the use of “back-door” or “under-the-cover” functions. Similarly, the Then part should not use special access function to verify the results. Using special non-API functions in the Given and Then parts is a sure sign that your interface is not quite right.

BDD’s Given and Then parts correspond to the preconditions and postconditions known from programming by contract. One module’s precondition (Given part) is another module’s postcondition (Then part). If the postconditions of module A meet the preconditions of module B, then A and B work together fine. When you write the tests for B, you should make sure that B handles the results of A’s Then parts correctly. This insight enables you to test the modules of your software in isolation - without suffering from exponential explosion as system tests do.

The BDD structure is well-suited to specify the acceptance criteria of user stories. So, BDD tests are acceptance tests. Some TDD purists might object that acceptance are not unit tests and not suited for TDD. But, what exactly is a unit? A unit is a fuzzy thing that nobody can define properly but many people discuss fervently. As outlined in Doing TDD with I/O-Free Tests, I prefer a clearer definition: I/O-free tests are fine for TDD and I/O-based tests are not. As long as acceptance tests are I/O-free, I am more than happy to use them for BDD and TDD.

BDD-style test cases function both as a human-readable specification (even for non-technical people) and as an executable specification. They transform human intention into executable code.

[BDD-style test cases ensure that] you don’t inflict crap code on your co-workers and the rest of the world.

[…] BDD is a disciplined approach that will work for your projects.

Steve Branam

Amen to that!

Note: You can find the complete example, the Bit-Banged Serial Transmitter, on the BBSerialTx GitHub repository and a detailed post on EmbeddedRelated.com.

Kate Stewart: Architecting for Safe Embedded Systems that Integrate Open Source Components

“Ingredients of Modern Embedded Systems (1:34 - 10:30)

Teslas driving on Autopilot were involved in 736 crashes with 17 fatalities. Autopilot ignored speed limits and the stop signs of school buses or didn’t recognise motorcycles and pedestrians. Autopilot’s AI models were not trained on these situations or the decision making component interpreted the available data wrong.

This examples illustrates that cars and embedded systems in general are much more than hardware and software. Embedded systems additionally rely on AI/ML models, training data sets and remote services. What makes safety analysis especially difficult is that all these components interact in myriad and often surprising ways.

System thinkers have long known that a system is not defined by the sum of its parts but by the product of its interactions. Taming and understanding this product of interactions is a formidable task. Kate gives an overview how we can do this to build safe embedded systems. Most ideas also apply to secure embedded systems.

I’d like to add that building safe systems is very much a leadership problem. If the leaders of a company have a don’t-care attitude towards safety, their products won’t be safe. Tesla proves this point. If a company like Volvo seriously pursues the goal of building cars that don’t seriously injure or kill people, it will build much safer cars.

System Engineering Needed When Safety Elements Present (10:31 - 18:25)

Currently, the methods for performing a risk or root-cause analysis are ad-hoc and the data for the analysis is provided in many different format. How can we do this more efficiently and eventually automate it?

Automating such analyses is still a good deal away. But collecting the metadata in a standardised way and format is pretty much available. As the bill of materials for hardware (BoM) and software (SBoM) is not enough for today’s systems, Kate introduces the system BoM (SysBoM). The SysBoM documents the dependencies or interactions between hardware, software, AI models, training data sets, remote services and other components of the system.

The SysBoM answers the following and many more questions.

• What software component versions are executing on which specific hardware devices?

• What software components’ direct and transitive dependencies should be monitored for vulnerabilities?

• What is the provenance of how a model was trained? What datasets were used for testing and validation?

• How were the datasets used for training created? Are there known biases?

• How were the software components and models integrated and tested?

• What APIs are used to manage updates through remote services?

• What remote services does the running software and trained models depend on? What happens when the service is not available?

Kate Stewart

The SysBoM must be able to answer these questions for any point over the lifetime of the system. The SysBoM is a bit like a time machine. It must always be up-to-date. So, it’s best to create the SysBoM right away for any phase of the development process, that is, for the design, sources, build, deploy and runtime phase. Creating the SysBoM as an afterthought will reduce its usefulness considerably.

Connecting Metadata Information for Risk Analysis (18:26 - 28:09)

The SysBoM is stored in the SPDX format, which is formalised in the ISO/IEC 5692:2021 standard. SPDX offers profiles for security, licensing, AI, data (e.g., AI training data sets) and build. Profiles for safety, hardware, services and operations are in progress for SPDX 3.1.

For a risk or root-cause analysis, you create a knowledge graph from the SBoM in SPDX format with tools like sbom2dot (see the video from 23:27 for example graphs). At the moment, you must look at the graphs and do the analyses manually. The challenge is how to perform these analyses automatically in the future.

Evolving Open Source Project to Support Safety Analysis (28:10 - 35:43)

Kate mentions four open-source projects that are working towards support for functional safety analysis.

• ELISA (Enabling Linux in Safety Applications). The goal is to make it easier for companies to build and certify Linux-based safety-critical applications,

• Zephyr was “built with safety and security in mind right from the start”. The project is working on certification according to IEC 61508 SIL 3.

• Xen provides a hypervisor to enable safety and security in virtualisation solutions. The project is working towards getting IEC 61508 and ISO 26262 safety certification.

• Yocto “generates SPDX 2.3 SBOMs for the build toolchain and all components built by this toolchain, to source level today, by a single configuration change”. Support for SPDX 3.0 is in progress. See Sergio Prado’s article Introduction to SBOM management on embedded Linux for how to generate the SBOM with Linux.