Round-Up of Embedded World 2024
Episode 52: Better Built By Burkhard
Dear Reader,
As loyal readers of my newsletter, you know that I am currently working on an OTA-update solution with an SwUpdate client and a Memfault server. Last month, I ran into a problem with delta updates. SwUpdate requires the upload of two update archives to the OTA-update server: one archive with the update configuration and the checksums of all blocks, and another archive with all the blocks of the ext4 image.
The Memfault server, however, only supports the upload of one archive. So, my customer would have had to set up their own server for hosting the second archive. They explicitly chose a hosted solution, because they wanted to avoid the hassle of running their own server. I came up with an application update as a workaround.
Given a list of application packages, a Yocto task packs all the files of these packages into a tarball and creates an SwUpdate archive with the tarball. The archive is a lot smaller than the archive for a full rootfs update (22 MB versus 350 MB for my project). Installing an application update on a device is faster and requires less bandwidth than a full rootfs update.
If only the applications but not their dependencies changed, the application update is a good workaround. Otherwise, you must ensure that all the modified dependencies are included in the tarball as well. I have used application updates for driver terminals of agricultural and construction machines. These updates even included new versions of the Qt libraries built against an unchanged SDK.
Of course, I was curious whether I could find proper delta-update solutions at EW24. I did with the RAUC client and the QBee server. Read more in the section Two More OTA-Update Solutions.
My next task on the project will be to enable secure boot for the iMX8M Plus. Manufacturers cryptographically sign the boot loader, Linux kernel and applications with their private key. On start-up, the device checks the signature of the software with the corresponding public key (see What is Code Signing? for more details). Manufacturers can guarantee that the software running on the device is the software they provided.
Keeping their private keys secret is paramount for manufacturers. If evildoers get their hands on private keys, they can make users install malicious software on their devices. As this should never happen, manufacturers need a secure and effective way to manage cryptographic keys. They need a so-called public key infrastructure (PKI).
At EW24, I found one company, Crypto Quantique, that offers an OTA update server with built-in PKI. Read more in the section Two More OTA-Update Solutions.
Now, enjoy my round-up of the Embedded World 2024.
Happy reading,
Burkhard
My Upcoming Talk at the Embedded Online Conference 2024
I am pleased to announce that I am giving a talk The Ports-and-Adapters Architecture for Embedded HMIs at the Embedded Online Conference 2024 (EOC24). My talk is based on my post Ports-and-Adapters Architecture: The Pattern.
If you register by April 26, you’ll get a $100 discount with the promo code SPEAKERDEAL. The EOC24 is a virtual conference with an amazing line-up of speakers including Elicia White, Kate Stewart, Philip Koopman, James Grenning, Jacob Beningo and Sergio Prado.
I had to pre-record my talk for the EOC24. Up to now, I had used the built-in screen-recording capabilities of my Macbook. This time, I upped my game and used Camtasia+Audiate - a professional video and audio editing tool. After watching two 5-minute tutorials, I was ready to go.
Audio editing is a breeze. You can edit the transcript, which is automatically synchronised with the audio and video. You can quickly get rid of hesitations, partial sentences and unwanted repetitions. This enables a different way of recording talks. If you are not happy with a section, you go on and repeat the section slightly differently - without starting over. Then, you simply cut out the parts you don’t like.
Camtasia+Audiate are a good example how Usage-defined or UX-defined software should work. It should simplify the user’s work or even make new things possible. You find this as Guideline 1: Focus on What Customers Want to Buy of My Mission.
Round-Up of Embedded World 2024
The Elephant in the Room
The elephant in the halls of the Embedded World 2024 (EW24) was the EU Cyber Resilience Act (EU CRA), as it most likely becomes law this year. At least for me, it should have been the most important topic for EW24, but exhibitors ignored it and went on with “same ol’, same ol’”.
The EU CRA is the highest risk with the most severe consequence that device and machine manufacturers are facing over the next years. While big players may have the resources and knowledge to deal with the EU CRA, most small and medium-sized manufacturers (SMMs) lack both. Most SMMs have no idea or only a very vague idea what kind of tsunami is going to hit them and how to avoid or mitigate the impact. And, they will struggle to find help.
In an ideal world, someone would offer an EU-CRA-ready embedded Linux system. Such a system would comprise a minimal Linux distro, an end-to-end secure OTA update solution and a web-based dashboard for managing and provisioning cryptographic keys, that is, a public key infrastructure or PKI. The OTA client would be well integrated with the application over a de-facto standard interface. Different manufacturers could implement different update strategies based on the interface.
With such a solution, manufacturers can quickly and securely update their devices, when a security issue is published. This is what the EU CRA demands.
Back in the real world. When I discuss this idea with SoM, SoC or terminal makers or with OTA solution providers, their answer is something like: “This is a nice idea. But it’s impossible to realise, because our customers’ systems differ far too much. Anyway, we have some documentation how to do these things” (see my newsletter episode The Board Support Package is Not Enough for such a conversation). Really?
What a lame excuse! These “solution” providers expect their customers to develop an EU-CRA-ready solution on their own, that is, to reinvent the wheel over and over again. What a total disrespect for their customers!
The three magic words for such a solution are: focus, focus, focus!
Focus on a single target market.
Focus on the SoC family dominating your target market.
Focus on a minimal custom-built Linux system.
Focus on a single update client that performs delta and full updates, online (OTA) and offline updates without any workarounds.
Focus on a single update server that supports fleet management, crypto-key management (PKI) and staged rollouts, and that can show crash reports, logs and metrics (e.g., battery level, RAM usage, reboot reasons) from devices.
Focus on a price model best suited to your target market.
Current “solution” providers lack focus and differentiation, because they are afraid of missing out on some business. . They want to cover the complete market of all embedded devices. They want to cover all SoC and SoM makers. They want to cover all operating systems from bare metal over RTOSs to Linux and Windows. They want to build the egg-laying wool milk pig. As such a pig doesn’t exist, such a “solution” doesn’t exist either. It provides everything and nothing - and creates unhappy customers.
The focused solution sketched above is a good starting point for an EU-CRA-ready embedded Linux system. Solution providers can improve it incrementally in many small steps, which add to pretty progress in the mid term. They could add other SoC families like the ones from TI, STM or NVIDIA or other OTA-update clients and servers. The selection criterion should be: Which extension has the most value and the lowest costs?
Two More OTA-Update Solutions
QBee
One of my first stops on Tuesday morning was at the QBee booth. QBee is a Norwegian startup providing a fleet management server for secure OTA updates of IoT devices. I know one of QBee’s founder and CEO, Carsten Lehbrink, from his time at Trolltech and Nokia Qt. Carsten kept me updated about QBee’s progress over the last two years.
In their demo, QBee had integrated a RAUC client on a Raspberry Pi with their QBee server. The RAUC client was performing a delta update (“adaptive streaming” in RAUC lingo) of the rootfs using an A/B update strategy. In contrast to my experience with SwUpdate, delta updates were running without any problems. Seeing this, I would now choose RAUC over SwUpdate as the update client.
The QBee server allows the installation of configuration files on devices. You could remotely enable additional features of a device, as soon as the user has paid for them. For example, you could enable the lane assistant of your car two years after you bought the car.
Another promising QBee feature is remote device access via VPN. Support engineers can log in to a device via ssh or mirror the device screen via VNC to their local desktop. They can do all this in a secure way. By the way, VNC mirroring is another feature I’ll implement on my current project - after secure boot is working.
My first impression is that QBee could be an alternative to Memfault and Mender as a fleet management server. The crucial question that QBee must answer is: Where is their focus? They can only compete with the bigger players, if they get their focus right. I am curious where QBee’s is heading. I have them on my shortlist now.
QuarkLink
When roaming the halls of EW24, I was diligently looking for any mention of the EU Cyber Resilience Act (EU CRA). One hour before closing time, I found one exhibitor, the British company Crypto Quantique, explicitly referring to the EU CRA: “Get EU Cyber Resilience Act Ready”. The irony wasn’t lost on me that a British company helps EU companies deal with EU regulations.
Their main product, QuarkLink, is a fleet management server for secure OTA updates - with a full PKI. They started their demo with generating a certificate for a device from the root certificate stored in QuarkLink. When they installed an unsigned application on the demo device, the system refused to execute the application and showed a warning. The device runs a correctly signed application without any complaints.
The whole demo took less than five minutes - including the explanations for me. The only input QuarkLink needs is a public key generated by the device. QuarkLink hides all other cryptographic actions behind a simple web UI. Users need not be experts in security and cryptography.
QuarkLink can also register devices with IoT cloud services like AWS IoT or Azure IoT. Devices can send device metrics into the cloud using MQTT or CoAP (see this demo).
Having a built-in PKI clearly differentiates QuarkLink from other fleet management servers like Mender, Memfault and QBee. However, I still need to put QuarkLink through its paces to reach a final verdict.
Qualcomm acquires Foundries.io
During EW24, I learned that Qualcomm had acquired Foundries.io a week earlier. You might rightly ask: Who is Foundries.io?
Their product, FoundriesFactory, is a platform-as-a-service (PaaS) offering that provides a “comprehensive, secure and over-the-air, updatable Linux-based platform for development, deployment, and lifetime maintenance of IoT and edge devices.” FoundriesFactories is “a very thin platform at the bottom” of IoT devices], on which the manufacturers build their industry-specific devices. The quotes are from the article IoT Is Broken: It Won’t Scale by Junko Yoshida (The Ojo Yoshida Report).
FoundriesFactory turns many isolated vertical solutions into a single horizontal platform with many applications running on top of it.
If correctly implemented, a solution to one IoT problem could be automatically fed to other connected devices. The approach would divert from current practice, in which a solution is exclusive to one segment or, worse, a problem is solved by an OEM that shares it with no one else.
Junko Yoshida, IoT Is Broken: It Won’t Scale, The Ojo Yoshida Report
FoundriesFactory provides the Yocto layers and recipes for embedded Linux systems in the cloud. Developers adapt the recipes to their needs and trigger builds on the servers of Foundries.io. All the work happens in the cloud. FoundriesFactory comes with fleet management, secure OTA updates and with 20+ years of maintenance.
The pricing has disappeared from their web site. I remember that the monthly subscription fee was $6,000 six months ago. When the subscription ends, developers can keep their system but have to move it to their local servers.
In short, FoundriesFactory is an implementation of the EU-CRA-ready embedded Linux platform that I am arguing for in the section The Elephant in the Room above. It’s actually quite a bit more. It comes close to what I am demanding in my newsletter The BSP is not enough!
The most important player of the system is still missing: the customer. [Their] core business […] is building the HMI or controller software for machines and devices [- and] certainly not building a custom Linux system with Yocto. They best buy the custom Linux system for a fixed price or for a premium on the [terminal] price, spend the time saved on their core business and get their products to the market faster.
Yours truly, The BSP is not enough!
It is unclear whether Foundries.io can operate in the same way as before under their new owner or whether Qualcomm will use FoundriesFactory only in-house. The acquisition is a clever move to gain an advantage over their competition. All the SoM and SoC makers are struggling to find developers building not just a BSP but a Linux platform. Qualcomm just bought the experts for this.