Episode 19: Burkhard on Qt Embedded Systems

Welcome to Episode 19 of my newsletter on Qt Embedded Systems!

My focus is on the Forrester study "Smarter Products Need Smarter Development" paid for by The Qt Company. The key recommendation is to focus on customer experience.  The recommended way to achieve this focus is to use cross-platform tools and frameworks: cross-platform as silver bullet. I beg to differ. I think that smarter development needs smarter people with more profound knowledge.

Enjoy reading and stay safe - Burkhard 💜

My Blog Posts

Architecture of Qt Embedded Systems: Single vs. Multiple GUI Applications

When is a system with a single GUI app without a window manager good enough? When is a system with multiple apps with a window manager the better choice? What are the requirements on a window manager? Shall we use a Wayland compositor or X11? The answers for a concrete Qt embedded system have considerable influence on the system architecture.

Events

Qt Day Italy 2021

Qt Day Italy 2021 is a series of webinars spread all over the year. My talk
How to Get the Architecture of Qt Embedded Systems Right (registration required!) is on 8 July 2021, 18:00 - 19:30. I'll talk for 45-60 minutes and then answer your questions for 30-45 minutes.

First International Test Driven Development (TDD) Conference, 10 July 2021

The speaker list is lead by the doyens of TDD: James W. Grenning and GeePaw Hill. There are many other excellent speakers including some who are sceptical of TDD. The conference is free. It starts at 16:00 (UTC+2) on 10 July and ends at 04:00 (UTC+2) on 11 July.

My Thoughts on the Forrester Study "Smarter Products Need Smarter Development"

The Qt Company commissioned Forrester Consulting to conduct a study Smarter Products Need Smarter Development. Forrester interviewed "262 embedded device and connected product development decision-makers at global enterprises". Here are the results with my commentary.

Challenges

This main part of the survey lists the organisational, process, people and technological challenges of embedded product development (see Figures 1, 2, 3 and 5, respectively). "Overall, decision-makers in our study are less challenged by individual technology concerns compared to organizational, process, or people concerns."
Qt has considerable influence to overcome the technological challenges and the chip shortage. The Qt Company has little influence, however, on organisational, process and people challenges. These challenges are best solved by the decision makers of Qt's customers. Improving the organisation, processes and people gives a much bigger leverage than improving technology. It is the difference between effective (doing the right thing) and efficient (doing a thing right).
Organisational Challenges

1. For 37%, making decisions and coordinating development teams is difficult, because teams are scattered all over the globe.

2. For 36%, teams are too much "focused on individual metrics instead of end-customer experience".

3. 34% regard the lack of skilled people as a problem.

I find it slightly ironic that the most individualistic region, North America, complains about a "lack of shared objectives" (item 2 above) nearly twice as often as the least individualistic region, Europe.
The lack of skilled people would be less of a problem, if societies integrated well-educated women and immigrants better into their workforces and if managers supported developers better in improving their skills (during work hours, of course).
Process Challenges

1. 36% see difficulties when teams from different departments must work together. This seems to be a generalisation of the first organisational challenge. 

2. 36% think that development teams spend too much time on maintaining existing products. This smells of quality problems, a.k.a. too many bugs.

3. For 35%, "different ways of working in software teams" is a problem. The 4th challenge adds 30% lamenting a "lack of iterations in the development process".

4. For 26%, "difficult/inconsistent user experience" of their products poses a problem. 

The Agile Manifesto celebrates its 20th anniversary this year. Decision makers should familiarise themselves with the four values and twelve principles and ensure that their development teams apply them.  Using test-driven development (TDD), refactoring and continuous integration (CI) rigorously would considerably mitigate the problems of items 2 and 3. 
The research results presented by Nicole Forsgren in the book Accelerate – Building and Scaling High Performing Technology Organizations (see my review) provide the empirical evidence why the Agile principles work. Decision makers have no excuses to ignore these values and principles any longer.
People Challenges

1. 35% complain about "complex development processes that create backlogs". That's a variant of the third process challenge. 

2. 32% acknowledge the high pressure of very short development cycles on the teams. 

3. 30% state that they have "difficulties finding skillful resources" and 26% that there is a "lack of available resources to innovate". These two challenges are a variation of the third organisational challenge.

Item 1 emphasises how complex processes can drain the energy out of developers. Developers try to deliver good work despite the process being an obstacle. The pressure of very short development cycles (item 2) is felt most in North America: 15% more than in Europe and Asia.
Finding innovative people (item 3) is especially hard, because the economic mantra for the last 100 years has been efficiency, efficiency and efficiency. If you look for innovation, you better look for effective people (doing the right things) instead of efficient people (doing things right). Being effective yields much higher profits in the long-term than being efficient.
One more thing: I don't like to work for companies who regard me as a resource. I'd assume that you are not much different. So, companies may be more successful in finding skilful people.
Technological Challenges

1. 29% have difficulties "scaling and maintaining software across multiple platforms/target hardware".

2. 29% face "quality issues that reduce customer experience or require expensive updates".

3. 24% think that there is "lack of efficient development tools".

Building cross-platform software (item 1) is where Qt truly shines. Qt supports all desktop platforms (Windows, macOS, Linux), all mobile platforms (iOS, Android) and all important embedded platforms (Linux, QNX, VxWorks, Integrity, FreeRTOS, bare-metal). Qt also supports the SoCs from NXP, NVIDIA, Broadcom, Qualcomm, Renesas, Intel, AMD and more. And don't forget Qt for MCU supporting microcontrollers from NXP, STM, Renesas and Infineon.
With QML, The Qt Company provides the best technology in the market to create compelling HMIs (item 2). The Qt libraries with more than thousand classes lift application development on a higher abstraction level. Developers write a lot less code and can make a lot less errors. However, nothing prevents designers and developers from producing a bad UX with many flaws that requires many expensive updates in the fields. Garbage in, garbage out. This partially turns this technological challenge into a people and process challenge.
The Qt Company has a good offering of development tools (item 3) with QtCreator, QtDesigner (including the Photoshop, Sketch and Figma bridges) and Squish. Nevertheless, I think that all tools would benefit from an improved UX. This starts with little things like higher colour contrasts between texts/icons and backgrounds. Of course, there are bigger things. I'd like to have the refactoring and CI/CD features from CLion in QtCreator. I'd also like the design-tool bridges to generate reusable and high-quality QML code more similar to how a good developer writes it.
A Special Market Challenge: Chip Shortage
"Until the global semiconductor shortage eases, [...] decision-makers will [...] use what can be sourced, when it can be sourced. Investing in flexible software tools and platforms that support a wide variety of silicon can reduce the impact of critical supply chain shortages." - Qt is exactly such a platform as it runs on so many SoCs. Migrating Qt applications from one supported SoC to another is fairly straightforward and quick.

Recommendation: Focus on Customer Experience

The old buzzword User Experience (UX) has been replaced by the new buzzword Customer Experience (CX). 20 years ago, UX and CX were called usability. Don't ask me what the difference is. It must be a fashion thing.
The study found that decision makers regard CX as something very important.

1. 48% consider "an improved customer experience as their biggest strategic priority" compared to 40% for better collaboration, 39% for reducing costs and 19% for faster time-to-market. 

2. 55% think that "compelling user experiences [have] become more important over the past year".

3. 70% want to use CX "as a measure of success".

4. Roughly 65% look for "technologies that can improve CX". The study emphasises "the many benefits from cross-platform design and development tools". This is an excellent opening for Qt.

Forrester recommends to put customer experience front and centre in embedded product development. Unsurprisingly, I fully agree with focusing on CX, UX or usability.
I am a bit skeptical whether decision makers walk their talk (item 1). My customers often want an iPhone-like UX but they rarely want to pay for it. They often don't even bother to hire professional UX designers. Whenever I ask for some extra data from the machine to lift the UX to a new level, I am rejected.
UX has become noticeably more important (item 2) since the introduction of the iPhone (see my post Why Use Qt for Embedded Systems). The UX of infotainment systems has, for example, become an important factor in buying cars.
Even if managers can figure out how to measure CX (item 3), I am sure that profit is and will be the ultimate measure of success for a product. A famous example: the Macintosh computers in the 80s had a much better UX than Windows computers, but they were a commercial flop.
Forrester makes four more recommendations that help improve the CX (underlining mine):

1. "Plan for longer, more engaging product lifecycles. [...] the ability to support a long, ever more valuable product experience will increasingly define successful digital product organizations."

2. "Invest in technologies that support product, skill, and supply chain flexibility. [...] That’s why 80% of smart product decision-makers are investing in cross-platform design and development tools."

3. "Blunt the effects of the semiconductor shortage with cross-platform frameworks."

4. "Break up the skill silos. [...] Cross-platform frameworks and tools make it easier for developers to work across different embedded platforms and fill talent needs."

3 of 5 recommendations mention "cross-platform" explicitly: a clear hint to Qt. 4 of 5 recommendations are of technological nature. The only exception is the main recommendation: focus on CX. This focus on technical solutions comes a bit surprising. The study seems to contradict its own findings: "Overall, decision-makers in our study are less challenged by individual technology concerns compared to organizational, process, or people concerns."
I agree with this statement that organisational, process and people problems are more pressing than technological problems. I think that organisational, process and people problems cannot be solved by technology. The main leverage comes from improving people's knowledge, from removing organisational obstacles and from using the right development and deployment processes (e.g., XP, Kanban). 
Smarter development needs more people with profound knowledge. The rest unfolds from there. Only the decision makers can make that happen.

Reading & Watching

Tor Arne Vestbø (The Qt Company): Qt on Apple Silicon

Qt 6.2 will support Apple Silicon, a.k.a. Apple M1 chips. You use the Qt 6.2 preview from the Qt installer. Or, you can build the Qt libraries from source. If you want to build universal binaries, add 
    -DCMAKE_OSX_ARCHITECTURES="x86_64;arm64"
to the configure command. If you are happy with arm64 binaries for the M1 chip, you don't have to do anything. The same CMAKE define is needed for building universal binaries of your applications.
If you build a Qt application from QtCreator on an M1 Mac, you must add the CMAKE define
    -DCMAKE_OSX_ARCHITECTURES="arm64"
to the "Initial CMake parameters". Debugging doesn't work for arm64 binaries at the moment, but will be available in the final Qt 6.2 release.

Sergio Prado (Toradex): Designing OSTree based embedded Linux systems with the Yocto Project

OSTree enables OTA updates of the bootloader and the root filesystem on embedded Linux systems. It works delta-based, that is, it only updates the files that have changed between two Linux versions. It performs updates atomically so that the new Linux version is guaranteed to be complete and bootable. Toradex uses OSTree to update its own TorizonCore, the base Linux system of its containerised operating system, Torizon.
OSTree works similar to Git. It stores different versions of the rootfs in a repository. Each version is the unpacked rootfs tarball from a Yocto build. OSTree only stores the deltas between these versions. The deltas could be the files from added or removed packages.
Updating the Linux system on an embedded device amounts to performing an OSTree-Pull and an OSTree-Deploy. The pull command downloads the deltas of the new version compared to the currently active version. The deploy command changes the boot environment variables (uEnv.txt). Both commands leave the active system untouched. The new system is started after reboot.
Sergio explains the workings of OSTree in an easy-to-understand way in his talk and even  shows the update of an Apalis i.MX6 system live.
In 2016, Qt announced a Qt OTA module based on OSTree as part of Qt for Device Creation. The module's documentation was archived as a technology preview. Hence, I suspect that Qt OTA never made it into a product. Nevertheless, Qt OTA could still be a good starting point to integrate OSTree-based OTA updates into your Qt application.

Norihiko Shirouzu: How Toyota thrives when the chips are down

Unlike VW, GM, Ford, Honda and Stellantis, Toyota hasn't suffered from the chip shortage yet. On first glance, this is a bit surprising, as Toyota is the champion of just-in-time supply chains. On second glance, it is not surprising, because Toyota learns from its past mistakes.
The Fukushima catastrophe in 2011 seriously disrupted Toyota's supply chain for six months. Toyota analysed the problem and implemented a fix. Toyota put in place so-called business continuity plans that require their suppliers to stockpile critical parts like chips for 2-6 months. 

Matt Greenberg, Keya Patel: Tech debt is not a burden, it's a strategic lever for success

The common advice is that developers should keep technical debt at zero. If technical debt occurs, developers should eliminate it as quickly and radically as possible - typically by refactoring the code. Business people "view tech debt as a strategic lever for your organization's success over time".
"Companies and products never win by having as little as possible tech debt. Speed to delivery, being first to market, and constantly adding value to the product are things that lead to winning. And, most tech debt can make sense to accumulate as a tradeoff in order to get more of the things that lead to winning."
So, managing technical debt becomes classical risk management. How much value do you get in return for taking on this risk (technical debt)? If the value is considerably higher than the cost, you will accept the technical debt. Otherwise, you will decrease the technical debt to an acceptable level, where value outweighs cost.
The authors start with debunking the wrong but common assumptions behind technical debt.

• Tech debt = bad. That's too black and white. You should "break the tech debt bucket apart by naming and sizing the pain". Tech debt can also be good. For the first 8 years of its existence, Twitter used a publicly available distributed database. Once it was well established, it built its own database.

• All tech debt = complicated. You should try to estimate the costs of technical debt. Breaking down technical debt in smaller chunks helps makes estimation simpler. In the next Reading item, Philippe Bourgau gives some practical tips for estimation.

• Tech debt ≠ Product work. If technical debt negatively influences UX or CX  and hence the product itself, eliminating the debt is product work. If not, the debt may not be important.

• Individual pain = Organisational pain. Developers often mistake their personal pain with a piece of code for a pain that everyone must feel. If you put the pain into a broader, more systemic context, the pain may abate.

The authors advice to debunk the assumptions is to make a business case for eliminating the technical debt. For a business case, you must specify the technical debt, estimate the costs for eliminating it, estimate the value from eliminating it and prioritise it against other features. The authors describe this in detail in the rest of their post.
My summary in one sentence: Technical debt becomes a normal user story in the product backlog.

Philippe Bourgau: Series About How To Get Sponsorship For Large-Scale Refactorings

In a series of 15 posts so far, Philippe answers the question: How can developers convince business people to sponsor a large-scale refactoring? A large-scale refactoring is the consequence of high technical debt.
Presenting a large scale refactoring as a business opportunity
Developers have cried technical debt so often that business people just ignore it. Developers must present large-scale refactorings as business opportunities and work out their value. What does the business get in return for spending good money on the refactoring? 
Cost arguments always resonate with business people. Demonstrating how the refactoring will reduce the cost for fixing bugs and for developing new features most likely gets your refactoring prioritised. While doing a large-scale refactoring, you should be able to deliver new features. Incremental refactoring supports this goal by breaking down the large refactoring into smaller steps.
Making the business case for a large-scale refactoring - Part 1
Just claiming cost reductions may work once but certainly not multiple times. Business people love numbers. So, you give them numbers.

• Refactoring cost is an estimate for how long the large refactoring will take. This is a one-time cost.

• Non-refactoring cost is the extra time you must spend on the bad code. This cost includes time for bug fixes, for support and for working slower because of bad code (productivity loss). In the planning meeting, you estimate each story twice: once for the bad code and once for the refactored code. The difference is the productivity loss. You log the non-refactoring cost for a fixed period of time (e.g., for one month). This is a recurring cost.

• Payback period = refactoring cost / non-refactoring cost. Example: Your refactoring costs are 120h. Per month, you waste 40h as non-refactoring costs. Then, it takes 120 / 40 = 3 months to amortise the refactoring costs.

The payback period gives lets business people (e.g., product owners) make an informed decision whether to do the refactoring or not. 
Making the business case for a large-scale refactoring - Part 2
Philippe first discusses what you should do if the payback period is too long. "The longer the life expectancy of your product, the more refactoring you should invest in!" If the payback period is too high, you can split up the refactoring into "smaller, more focused [...] steps". Back to incremental refactoring.
Philippe then discusses some ideas for improvement. You can convert time into money and add the value new features (delayed or blocked by bad code) can create. This can yield pretty impressive amounts of money that easily convince business people. If the return of investment is meagre, you better abandon the refactoring.

News

Tuukka Turunen (The Qt Company): Qt 6 Reaches Feature Parity with Qt 5 - The Qt 6.2 Alpha Released

Six months ago, I took Qt 6.0 for a spin and migrated a harvester terminal from Qt 5.12 to Qt 6.0. My conclusion was: "When I look back at my last 15 years of architecting and building Qt embedded systems, I couldn’t have built a single one with Qt 6.0. Qt 6 is a technology preview and will stay so until Qt 6.2. Qt 6.2 will be the first feature-complete Qt 6 release. It is planned for the autumn of 2021."
With the alpha release of 6.2, the wait for a feature-complete Qt 6 release is over. We can start building embedded systems with Qt 6.2. A big thank you to the Qt Project to making that happen according to the promised schedule 🙏

Alexa Voice Service (AVS) Overview

If you want to build a device with Alexa inside, don't look any further than here, the documentation for Alexa Voice Services (AVS). You find more than 20 development kits for AVS here. The kits are grouped into kits for smart speakers, sound bars, headphones, headsets, smart home devices and set-top boxes.

Digital Driving Experience: Continental Receives Major Order for Display Solution across Entire Cockpit Width

The digital cockpit is made up of one display stretching from the left A-pillar to the right A-pillar. The instrument cluster is on the left, the infotainment system in the middle and the passenger entertainment system on the right - with enough space for more systems in between. I don't know whether Qt is inside. Nevertheless, it gives a good view into the future of car cockpits. You'll see Continental's digital cockpit in cars in 2024.