Episode 36: Better Built By Burkhard
A Basic Continuous Delivery Pipeline for Embedded HMIs
A Basic Continuous Delivery Pipeline for Embedded HMIs
Continuous Delivery is achieved by working so that our software is always in a releasable state.
Dave Farley, Continuous Delivery Pipelines - How to Build Better Software Faster, p. 5
A Continuous Delivery (CD) pipeline tells you when your software is not in a releasable state. The CD pipeline for embedded HMIs is divided into three stages: the Commit Stage, the Acceptance Stage and the System Stage. The Commit Stage takes less than 5 minutes, the Acceptance Stage less than 1 hour and the System Stage less than 3 hours to decide whether the software is releasable. You should strive to halve the running times of the three stages.
Continuous delivery is all about getting fast and frequent feedback for all our activities. That’s why it works. The earlier we get feedback the quicker we can fix problems and the less rework we will have.
Burkhard Stubert, The Key Principles of Continuous Delivery
The CD pipeline is the engine for Continuous Delivery. It tells you quickly that something is wrong with your software and gives you a good idea where this “something” is. In Farley’s words, a CD pipeline is a “falsification machine”. The throughput metrics (deployment frequency and lead time for changes) tell you how you are doing.
If the Commit Stage passes, you should be 85% sure that your software will work as expected. The Acceptance Stage gives you 90% confidence and the System Stage 95%. Again, you should strive to increase the confidence for each stage. If you find a bug in a later stage, push its test to an earlier stage - best to the Commit Stage. The stability metrics (change failure rate and failure recovery time) are your guide.
The best time to start with a CD pipeline is now. You start with a bare minimum pipeline and adapt it small step by small step. You may even start without the System Stage.
My focus is on embedded HMIs such as infotainment systems for cars, driver terminals for agricultural and construction machines, and operator terminals for industrial machines. You typically cross-build the applications, services and the Linux system on a workstation with a 64-bit Intel architecture for a target device with a 32-bit or 64-bit ARM architecture. Hence, the CD pipeline for enterprise software as described in Farley’s book Continuous Delivery Pipelines needs some adaptations - like an extra System Stage for building the Linux system and SDK.
Commit Stage
When a developer pushes a change to the main branch of a git repository, the Commit Stage starts running. A minimal Commit Stage performs the following steps in less than 5 minutes.
(C1) It builds and runs all unit tests on the host workstation.
(C2) It cross-builds the applications, services and libraries against an SDK for the target device and creates a binary archive for the standard update procedure, the release candidate.
The unit tests from step C1 are best organised into many small executables that the stage can build and run in parallel. A unit test is a small executable with as few dependencies as possible and with most dependencies replaced by test doubles. You can use ccache or small fast-to-build libraries to avoid multiple compilations of source files. The time limit of 2.5-5 minutes guides your choice.
TDD produces such unit tests. TDD makes your code more easily testable, which helps with the tests in the Acceptance and System Stage (see also Why TDD is a Main Driver for High Performance). Qt Test encourages you to organise unit tests into small independent executables, as it doesn’t know test suites. CMake and CTest enable you to build and run these executables in parallel.
Cross-building all applications, services and libraries in step C2 in less than 5 minutes, let alone in less than 2.5 minutes, is a challenge - even more so for legacy systems. There are two remedies that you will both apply. First, you run the build on more powerful workstations with faster drives either on premises or in the cloud. Second, you minimise the compile-time and link-time dependencies, which leads to a less tightly coupled architecture.
The Commit Stage runs steps C1 and C2 in parallel. This can happen on the same workstation, if the workstation is powerful enough. Don’t listen to the people in the IT department who inevitably want to save some euros on computer hardware. These cost savings are dwarfed by the costs that development teams spend on debugging and rework due to late feedback.
If both step C1 and C2 pass, the Commit Stage passes and the code changes are integrated into the main branch of the repository. If one of the two steps fails, the Commit Stage fails and the code change is rejected.
C1 and C2 are the mandatory steps for the Commit Stage. Here are some examples of optional steps that you can add over time.
(c3) This stage runs undefined behaviour sanitisers (UBSan) and address sanitisers (ASan) with the unit tests of step C1. If a sanitiser reports an issue, the stage fails. For example, UBSan would find casts from a negative float to an unsigned integer and ASan deletions of C++ objects by the QML engine.
Note: As UBSan and ASan instrument the code differently, the stage must build the unit tests once for UBSan and once for ASan. Whether UBSan and ASan run on ARM architectures and, if so, how reliably, is unclear. That’s why I would run them on the host workstation only.(c4) This stage runs step C2 with increased warning level and with warnings turned into errors. The stage fails if the cross-build emits a warning.
(c5) This stage runs analysis tools like TeamScale to find out whether code changes are sufficiently covered with tests. If some parts of the code change are not covered by tests, the stage fails.
And many more optional steps.
Acceptance Stage
The CD pipeline runs the Acceptance Stage, if the Acceptance Stage isn’t running already and if there was a successful run of the Commit Stage since the last run. This stage cannot be run on every code change, because it takes too long. Running the Acceptance Stage must not block the Commit Stage. You best run the Acceptance and Commit Stage on different computers.
A minimal Acceptance Stage performs the following steps in less than 1 hour.
(A1) It deploys the release candidate from the Commit Stage on the target device using the standard OTA update procedure.
(A2) It runs all acceptance tests on the target device in a production-like environment.
The Acceptance Stage runs each steps on one application or on one service (called containers in C4 modeling). A1 installs the latest version of an application on the device. A2 runs the acceptance tests on this applications. The stage repeats steps A1 and A2 for each application and for each service - in parallel if possible.
This stage does not run the steps on multiple applications or services at the same time. Doing this is reserved for the System Stage.
Using the standard update procedure to deploy the applications, services and libraries of your embedded HMI on the target device ensures that you have executed hundreds if not thousands of updates. Then, updates will hardly ever fail when done by customers.
Acceptance tests emulate how users interact with the embedded HMI and checks whether the HMI responds as expected. For example, if the driver of a forage harvester changes the cutting length, the machine must receive this change and confirm it. The test checks that the message with the new cutting length is written to the CAN bus and that the confirmation message is read from the CAN bus. The CAN bus is a mock that records received messages and responds with messages pre-configured for this test.
You should turn all acceptance criteria from user stories into acceptance tests. Each bug should be reproduced as an acceptance test. Test scripts that you walk through manually for every release should be automated as acceptance tests.
Squish makes it easy to write acceptance tests and run them automatically. Squish can run tests on the target device while controlling the test execution from the host computer. This remote test execution works similar to remote debugging. As Squish can simulate mouse clicks, touch gestures and key presses, you can easily emulate the user interaction in the tests. This is extremely useful for legacy systems, where business logic and user interface are intertwined.
If your embedded HMI is structured according to the ports-and-adapters architecture (see slide #8), there is an alternative to Squish. You can replace the GUI adapter by a test adapter, which has functions for each user action. In the harvester example above, the adapter would provide a function for changing the cutting length. If you start duplicating code from the GUI adapter in the test adapter, business logic has seeped into the GUI adapter. This needs fixing.
You can extend the minimum Acceptance Stage with other types of tests.
(a3) This stage runs integrations tests.
(a4) This stage runs performance tests.
And many more types of tests - especially tests for non-functional requirements.
In the ports-and-adapters architecture, integration tests can validate the interfaces between the business logic and the adapters. They can check the interfaces between classes. The outputs of one class must be admissible inputs for other classes. If that’s guaranteed, classes can interact smoothly with each other.
Integration tests can corroborate that the synchronisation between multiple threads works properly. Running ThreadSanitizer with these tests helps detect tricky synchronisation problems.
Performance tests could check that the terminal application can decode 1200 messages per second, that at most 200 are forwarded to the GUI and that the GUI displays 60 frames per second. The database must be able to save 500 data points per second.
System Stage
The CD pipeline runs the System Stage, if the System Stage isn’t running already and if there was a successful run of the Acceptance Stage since the last run. Running the System Stage must not block the Acceptance Stage. You best run the System, Acceptance and Commit Stage on different computers.
The System Stage runs the following steps in less than 3 hours.
(S1) It builds the Linux image and SDK using Yocto or Buildroot.
(S2) It installs the Linux image and the release candidate from a successful Acceptance stage on the target device - using the standard OTA update procedure.
(S3) It runs system tests on the embedded HMI system on the target device. The embedded HMI system consists of applications, services and the Linux system. The system tests are executed in a production-like environment.
The system tests check the interaction between multiple applications and services. Here are some examples:
After powering on the system, certain applications and services must be running.
When the user turns off the ignition, say, of an excavator, the touch display and some other electronic components are switched off. The main application keeps running. When the user turns on the ignition again, the user can instantly interact with the main application again.
Video playback from the rear-view camera must resume within 1s, when the terminal wakes up from sleep mode. The user must be able to interact with the HMI again.
While an update is running, the system must ignore the ignition being switched off.
Automating such tests is not easy. Your test code must, for example, be able to switch the power and ignition on or off. In other words, you are able to control the power and ignition of the target device from software.
Asserting the expected result of a system test is rarely straightforward. For waking up video playback, you could check that the dmesg log doesn’t contain any errors about the VPU (video processing unit). For power management, you could check that the dmesg log contains certain messages about disabling or enabling components like the backlight, ethernet interface or CAN interface.
My Content
Compatibility between Yocto and Qt 6 Versions
The Yocto layer meta-qt6 allows you to build every Qt 6 version (currently, Qt 6.2 - 6.5) against every Yocto version 3.1 or newer (currently, Yocto 3.1 - 4.1). You only have to change the revision of meta-qt6 in your kas configuration file or repo manifest file.
Building Qt 6.2 For Old Yocto Versions
With meta-qt5, there was no quick way to update the Qt version for a given Yocto version. The Qt version was determined by the Yocto version. Yocto 2.7, for example, comes with Qt 5.12. There are two ways to update to a newer Qt version while staying on the same Yocto version.
You update the Yocto recipes in the meta-qt5 layer to work with a newer Qt version. This is feasible if you upgrade to a new Qt minor version, say, from Qt 5.12 to 5.15.
You build Qt against the SDK for your BSP - outside the Yocto build. This is the preferred option when you upgrade to a new major Qt version, say, from Qt 5.12 to Qt 6.2. That’s the topic of the linked post.
The new meta-qt6 layer relieves you from this work. It uses the first option to enable you to build any Qt 6 version against any Yocto version from 3.1.
Around the Web
The Top 100 QML Resources by KDAB
When you struggle with developing QML applications (and who doesn’t from time to time?!), KDAB’s page about QML resources will be a huge time saver. You do not only find video tutorials about (nearly) every aspect of QML, but also invaluable tips about Qt Creator, debugging, profiling and many other topics. Try it out!
When I scanned the page, I found two issues that had lately become a bit of a nuisance in my daily development work. The next two items explain how to fix these issues.
Jesper Pedersen: Why doesn’t my Qt Creator find my files anymore
When I moved to Qt Creator 8 last year, I noticed that renaming variables, functions and classes with Ctrl + Shift + R forgets the occurrences in the header files. The same is true for replacing strings in the project. I learned to make sure that all the header files in the search pane were checked.
Jesper - a.k.a. the cavalry - comes to the rescue in his video tutorial. You must add the header files to the list of source files in CMake’s add_library and add_executable commands. Then renaming and replacing over the complete code base starts working again.
Jesper Pedersen: Understanding qAsConst and std::as_const
When you write range-based for-loops over Qt containers, Qt Creator warns you that the loop may detach the container (that is, make a deep copy of the container) and suggest that you use qAsConst.
for (const QString &s : m_stringList)
doSomething(s);
Jesper explains what triggers the warning and suggests three ways to get rid of it. Apply the fixes in the given order:
Make the function containing the for-loop const.
Wrap qAsConst around the container.
Store the container returned by a function in a const value (no deep copy for implicitly shared Qt containers) or pass it as a const reference to the function containing the for-loop.