Episode 8: Burkhard on Qt Embedded Systems
Welcome to Episode 8 of my newsletter on Qt Embedded Systems!
On my current project, I am re-designing and re-implementing a core part of an embedded system with a pretty high spaghetti factor. So, I decided to re-read John Ousterhout's book about software design to refresh my design skills. His concept of deep classes and my diligent use of TDD help me to find the right abstractions and simple interfaces. As I learn best when explaining things to other people, I wrote a review of John's book for this newsletter.
After the review, you find summaries of deep technical posts. The topics are race conditions, the premises of TDD and undoing almost any mess produced by Git.
Stay safe and enjoy reading - Burkhard 💜
My Thoughts on Yocto Updates
My plan for July was to update the Yocto version used for building my Internet radio system and publish Part 3 in my series on "Qt Embedded Systems" (see Part 1 and Part 2). The current system was built with Yocto 2.6 Thud released in November 2018. I wanted to update first to Yocto 2.7 Warrior, then to Yocto 3.0 Zeus and finally to Yocto 3.1 Dunfell - with as little changes in the recipies as possible.
Building the Linux image and Qt SDK with Warrior worked without any changes to the Thud recipes. I installed the Linux image with the Internet radio application on the SD card, plugged the SD card into my Raspberry Pi 3B and powered on the device. The Linux system booted up and the application was started. However, the application was not displayed and the radio station didn't play. Although I used identical recipes, nothing worked 😢😠
After some minutes of consternation, I ran lsmod to list the kernel modules. The list seems to tell me that both sound and display are sent to a monitor connected over HDMI. However, my 7-inch touch display is connected over LVDS. My next step will be to verify or falsify whether my radio system works with an HDMI touch monitor. The step after that will be to get the 7-inch touch display working again. So, I'll have to spend good time on a "simple" update from Thud to Warrior.
I passed identical recipes into a Thud build and into a Warrior build and the resulting Linux images were totally different. One image is working, the other is not working. One image uses LVDS for display and ALSA for sound, the other uses HDMI for both display and sound. From one minor version to another minor version (2.6 to 2.7), the system changed fundamentally.
For me, this is a nuisance. I'll figure it out eventually. For many companies, this would be a showstopper. They either stick with a working and more and more outdated version or they pay good money to fix the problem. These companies are not the customers I'd want to have as a consultant. Changing things fundamentally behind the back of users and making them pay is a very bad idea.
My Blog Posts
Webinar: Building Embedded Applications from QtCreator with Docker
The Qt Company invited me to present my post Docker Builds from QtCreator as a webinar. My post drew two main points of criticism. First, the motivation for running Docker builds from QtCreator wasn't clear. I address this point in slides 2-5. Second, my setup was too desktop-y, as the target device used an x86_64 architecture. I use a Raspberry Pi 3B as the target device in the webinar, which requires a true cross-compilation. As a bonus, you can pick up some tricks how to set up QtCreator for CMake builds, which never works out of the box. See the webinar link for the video and slides.
Book Review: "A Philosophy of Software Design" by John Ousterhout
I focus my review on the first nine chapters of the book. These chapters provide a strategic view of software design, whereas the remaining twelve chapters give tactical advice. The first nine chapters introduce the concept of deep modules and modular design. Deep classes are loosely coupled with other classes and have methods with strong affinity. John gives an enlightening visualisation of deep modules in Chapter 4. He also gives many examples and guidelines how to design deep modules.
You can buy the book on Amazon as a paperback for 18.95 USD or as a Kindle ebook for 9.99 USD. You find other reviews here and here. John also gave a talk about his book at Google. You find the video here.
Chapter 1 - Introduction (It's All About Complexity)
"Writing computer software is one of the purest creative activities in the history of the human race. Programmers [...] can create virtual worlds with behaviors that could never exist in the real world. [...] All programming requires is a creative mind and the ability to organize your thoughts." - Creativity and organisation are the yin and yang of software design. They are opposites that complement each other.
Good design keeps the complexity of software at a level such that we can extend the software with minimum effort. The book has two goals:
It defines complexity, how to recognise it and what its consequences are.
It presents techniques and design principles to minimise complexity.
Chapter 2 - The Nature of Complexity
"Complexity is anything related to the structure of a software system that makes it hard to understand and modify the system." The author lists three symptoms of complexity:
Change amplification: A change leads to many other changes at different places in the code.
Cognitivie load defines the amount of information we must keep in our head to implement a user story.
Unknown unknowns: It is unclear which piece of code we must change or extend to implement a user story.
Unknown unknowns are the worst form of complexity. We only find out about them when bugs start to appear (often with a delay) after a code change.
Complexity doesn't appear in a single big bang, but it accrues in small chunks over time. The more common term for this kind of complexity is technical debt.
Chapter 3 - Working Code Isn't Enough (Strategic vs. Tactical Programming)
Tactical programmers try to get a feature or a bug fix working as fast as possible. They take shortcuts, don't write tests, don't refactor code to improve the design and hardly think about design. Every shortcut introduces a little more complexity. Short-term, tactical programmers go fast, especially when they work on a greenfield project. Mid-term, they will go slower and slower. Pretty soon they will hardly make any progress. At some point, other developers will have to reimplement their parts of the software.
Strategic programmers understand that "working code isn't enough". They focus on getting the design right such that future extensions become easy. Their designs also happen to work. They avoid technical debt by all means. Initially, they may be slower than tactical programmers. They will be considerably faster mid-term. Long-term, they avoid the huge cost of a reimplementation.
Chapter 4 - Modules Should Be Deep
Modular design decomposes software "into a collection of modules that are relatively independent". Modules can be classes, subsystems or services. The goal of modular design is a loosely coupled collection of modules with minimal dependencies on each other. Separating the interface and the implementation of a module is the key to modular design. "The best modules are those whose interfaces are much simpler than their implementations."
An interface provides "a simplified view [of a module], which omits unimportant details". This simplified view is called an abstraction. "The key to designing abstractions is to understand what is important, and to look for designs that minimize the amount of [important] informaton." A well-designed module is a deep module, which is the core concept of the book.
The author visualises modules as rectangles. The area of the rectangle represents the functionality of the module, the top edge the interface and the height the level of abstraction. A deep module has a small width and a large height. A shallow module has a large width and a small height.
"Module depth is a way of thinking about cost versus benefit. The benefit provided by a module is its functionality. The cost of a module (in terms of system complexity) is its interface. A module's interface represents the complexity that the module imposes on the rest of the system: the smaller and simpler the interface, the less complexity [...] it introduces. The best modules are those with the greatest benefit and the least cost. Interfaces are good, but more, or larger, interfaces are not necessarily better!" - A good software designer finds the right balance between cost and benefit.
Linux file I/O is a good example of a deep module. With the five basic functions open, close, read, write and lseek, we can perform almost all file I/O on Linux. This tiny interface hides a huge amount of implementation details: file representation on hard disks, SSDs, CDs and DVDs, mapping of hierarchical file paths to directories, access permissions, concurrent access, caching, and many more. Five functions hide tens of thousands of lines of implementation. Deep modules "maximize the amount of complexity that is concealed".
A linked list is a good example for a shallow module. The interface is nearly as complex as the implementation. It conceals very little complexity.
Chapter 5 - Information Hiding (and Leakage)
As we saw in the previous chapter, deep modules like Linux file I/O do an excellent job in hiding information. Information hiding reduces the complexity by simplifying the interface of a module and by making the evolution of systems easier. Interfaces cannot expose any dependencies on hidden information.
Information leakage is the opposite of information hiding. It occurs, when changing an interface (that is, when changing a design decision) implies changes in clients of this interface. The author gives three common reasons for information leakage.
The first reason is temporal decomposition. For example, a program first reads information from an XML file, then manipulates the information and finally writes the modified information back to the XML file. If we put the three tasks in three separate classes or in three public functions of the interface, the module leaks information to its clients. If we decide later that we store the information in a JSON file or a SQL database, or request it from a cloud server, clients are forced to change accordingly. We would also find out that the classes for reading and writing the XML files duplicate certain information.
A deep class would have a single public function to read information, manipulate it and write it. The constructor of the class would get all the information (e.g., a URL string) required to figure out from where to read the input information and where to write the output information. Clients of this class wouldn't have any idea how and where the information is stored.
The second reason for information leakage is the provision of getters and setters, which make modules shallower. A pretty bad example would be a function
QMap<QString, qreal> getParameters() const
that returns the map from parameter names to values as it is stored in a member variable of the class. If we decide to use a data structure different to QMap, we force all clients to adapt to the change. We also allow clients to manipulate data outside the class owning the data (the smell of Feature Envy). This leads to tighter coupling and more complexity.
The third reason for information leakage is the omission of default values for function parameters. Missing defaults force clients to provide information that is not needed most of the time. They make common cases more difficult than necessary.
Chapter 6 - General-Purpose Modules are Deeper
When we design modules, we must often decide whether to make the module general-purpose or special-purpose. General-purpose modules tend to be deeper, but they also tend to provide functionality that is never needed in the future (smell of Needless Generality). The author suggests an excellent compromise: Make the interface "somewhat general-purpose" and the implemented functionality special-purpose.
The author explains the compromise with the example of deleting text in a text editor. The text editor is implemented with the model-view design pattern. One important goal is that the model and view class can be developed mostly independent. The user can delete text by pressing the Backspace key, the Delete key or Ctrl+X (Cut) on a selected text area. The view class calls functions of the model class to perform the respective actions on the text document.
A bad design would provide three functions for backspace, delete and cut in the model class. Whenever we added some new way of deleting text to the model class, we would have to change the view class accordingly. This solution is special-purpose, because every function is only used once in the view class.
A good design would provide one function delete(Position start, Position end) in the model class. For each of the three delete actions, the view class could call this somewhat general-purpose delete function. The interface is deeper and will not change, when we add another way of deleting text (e.g., deleting a line) to the view class.
Chapter 7 - Different Layer, Different Abstraction
Programs are typically organised in layers, where higher layers depend on lower layers. Well-designed layers are deep modules. Higher layers are more abstract than lower layers.
Pass-through methods call similar methods from a lower layer and pass the result with little or no modification to the higher layer. Wrapper functions often are pass-through methods. Pass-through methods are tell-tales that layers are shallow and that the responsibilities of two layers are overlapping.
Pass-through variables are handed from the top to the bottom layer through a long chain of function calles - or in the other direction. If we change, delete or add such a variable, we'll have to change the signatures of all the functions using them in all the layers. The author suggests to move all the pass-through variables into a context object. The context object is passed to the constructor of objects using these variables in their functions. This way the context object can be passed up or down the layer chain. It's not an ideal solution but better than the alternatives.
Chapter 8 - Pull Complexity Downwards
We make modules deeper by pulling complexity down into the module, that is, by moving code from outside the module into the module. Pulling down complexity is best applied, when
the functionality to be moved has a strong affinity with the module's existing functionality,
the program is simplified in many places and
the module's interface is simplified.
The author's reason for applying this technique is timeless advice: "Most modules have more users than developers, so it is better for the developers to suffer than the users. As a module developer, you should strive to make life as easy as possible for the users of your module, even if that means extra work for you. [...] it is more important for a module to have a simple interface than a simple implementation."
Chapter 9 - Better Together Or Better Apart?
Although small modules (i.e., small classes or small functions) are often portrayed as a magic bullet to reduce system complexity, they are not. Many small modules imply more dependencies and more interfaces leading to higher complexity. Big modules have gained a bad reputation, although they are not bad per se. As software designers, we should base "the decision to split or join modules [...] on complexity. Pick the structure that results in the best information hiding, the fewest dependencies, and the deepest interfaces."
The author gives some guidelines when to split or join modules.
Bring together if information is shared. A typical example is the encoding and decoding of network packets, which shares information about the structure of packets.
Bring together if it will simplify the interface. Temporal decomposition from Chapter 5 provides a good example.
Bring together to eliminate duplication. Sure!
Separate genral-purpose and special-purpose code. Chapter 6 gives the example of the text editor split into the text view and text model. The text view has three special-purpose function for deleting text, which the text model implements in a single general-purpose function. The author concludes: "In general, the lower layers of a system tend to be more general-purpose and the lower layers more special-purpose."
Splitting and joining methods. Methods should be deep with interfaces that are much simpler than their implementations. Methods with long parameter lists are typically shallow and should be joined with methods calling them. If we must jump around between many small functions to understand the function that calls them (called "conjoined methods"), we should think of joining the methods or introduce better names for the small functions. If a function performs multiple, hardly related tasks, we should split up the function. The golden rule is: "Each method should do one thing and do it completely."
In general, we join modules if they have a strong affinity. We split modules if they have a weak affinity. Affinity is strong, if modules change for the same reasons. It is weak, if they change for different reasons.
Reading
The Anatomy of a Race Condition by Matthew Eshleman.
Matthew defines a race condition as "an assumption in the system that some other event has already occurred when processing a new event". He quips: "And we all know what happens when we assume, right?" He discusses three common causes for race conditions and how to spot them during code review.
Arbitrary Delay or Sleep. I stumbled over this type in legacy code performing a firmware update for ECUs over CAN (in a harvester). The code sent 8-byte messages with update data to an ECU and then waited 20 ms for an acknowledgement from the ECU. Needless to say that the code didn't handle the case when the acknowledgement didn't arrive.
The firmware worked reliable for ECUs of type A. It didn't work at all for ECUs of type B. It turned out that type-B ECUs had a less powerful microcontroller than type-A ECUs. Type-B ECUs needed too much time to process the incoming messages so that the total trip time was longer than 20 ms.
Increasing the delay, say, to 30 ms isn't the solution. Eventually, there will be an even slower ECU in the network. Or, other ECUs send messages while an ECU performs an update so that the trip times are longer. A robust solution would be to wait for the acknowledgement before sending the next message - in an event-driven fashion. Fortunately, arbitrary delays and sleeps are pretty easy to spot in a code review.
Order of Events. Matthew's car decides which person enters through the driver's door by checking the key fob. Then, it adjusts the driver's seat accordingly. Task A receives the message which key fob passed through the door and stores the fob ID in a non-local variable lastKey. Task B asynchronously reads the fob ID and triggers the seat adjustment. If A happens before B (90% of the time), everything is fine. If B happens before A (10% of the time), the seat is not adjusted, because lastKey still holds the value Unknown. One solution is that task B always returns a concrete driver (e.g., the last, main or most frequent driver) when lastKey is still Unknown.
I think that the problem at hand falls into the category of wrongly or not initialised variables - exacerbated by concurrency. Code reviewers should be trained to look for such situations, as they occur fairly often. By the way, mocks are a good means to force unit tests into different event sequences.
Announcing an event before changing the data or state. This race condition illustrates the difference between synchronous and asynchronous Qt signal-slot connections. Here is the Qt code equivalent to Matthew's example.
emit stateChanged();
m_state = MOVING_ARM_UP;
The signal stateChanged is connected to a slot, which does something depending on the value of m_state. If the signal-slot connection is synchronous or direct, the slot still sees the previous value of m_state. If the signal-slot connection is asynchronous or queued, the slot may see the previous value or the next value - depending on the execution order.
A robust solution should emit the state change only after it changed the state. If the slot sometimes runs asynchronously to the signal and sometimes not, I'd advice to mark the signal-slot connection as a queued connection explicitly. You'd see such problems much earlier - typically already in your unit tests.
Five Underplayed Premises of TDD by GeePaw Hill (video and transcript).
GeePaw explains the five hidden premises of TDD. The premises are the reasons why TDD is an excellent way to develop software.
The Money Premise. TDD helps us to ship more value faster. That's the one and only way to make money in software development. TDD is the best way we currently know to develop software.
The Judgement Premise. TDD forces us to break down our work in many small steps. We must make a judgement call for every single step: Is this the right step to reach our goal? As judgement is an inherently human trait, TDD is certainly not an algorithm for coding
The Correlation Premise. External quality is what the user sees. Internal quality is what the developer sees: it's code quality. If internal quality goes up, productivity goes up. If internal quality goes down, productivity goes down as well. In other words, there is a correlation between internal quality and productivity.
The Chaining Premise. If we have a dependency chain A->B->C->D ("->" means "depends on"), we best test it the following way. We test A assuming B works fine, then B assuming C is fine and so on. We test the chain by testing each link in isolation.
The Steering Premise. Tests and testability help us navigate the dangerous and often unknown shallows of software development projects. Tests tell us where we stand and whether we broke anything. Testability gives us simple interfaces and hence easy extensibility of our software.
How to undo (almost) anything with Git by Joshua Wehner.
Joshua's post is a real treasure! Whenever I mess up anything with Git, I go to this post. And, I (almost) always find an answer. Here is the list of things you may want to undo.
Undo a "public" change. You pushed one commit sha1. You undo this commit with git revert <sha1>. You can also pass multiple SHAs to undo multiple commits.
Fix the last commit message. You committed a change, but didn't push it yet. You noticed that you forgot some changes or that the commit message is bad. You stage your changes and call git commit --amend -m "Better message".
Undo local changes. You changed files locally, saved them but didn't stage them yet. You discard these changes with git checkout -- <filenames>.
Reset local changes. You performed several commits, but didn't push them yet. You wipe out these commits without any by git reset <last-good-sha>. The option --hard gets rid of the local changes as well.
Redo after undo local. It turns out that git reset --hard doesn't undo all traces. If you want to undo the hard reset, git reflog is your friend. The fix is tricky with many if's.
Once more, with branching. You made several commits on master, but want them to be on a feature branch. On master, you create the feature branch, perform a hard reset to the SHA before your accidental commit and check out the feature branch.
Branch in time saves nine. You created a feature branch based on master. Then, you noticed that master was far behind origin/master. You get the missing commits from master into your feature branch by checking out the feature branch and by running git rebase master.
Mass undo/redo.You made several commits. You want to keep some of them and remove some others. You do an interactive rebase with git rebase -i <earlier-sha>. Git shows the list of commits up to <earlier-sha> in your default editor. You can remove a commit by removing its line in the editor. You can combine two lines with fixup (new commit message) or squash (message of first commit). You can change commit messages with reword.
Fix an earlier commit. You made a couple of commits, but didn't push them yet. One of the commits but not the last one misses a change. git commit --squash <earlier-sha> and git rebase --autosquash -i <even-earlier-sha> will right the wrongs.
Stop tracking a tracked file. You accidentally added a file like CMakeLists.txt.user or application.log to the repository. As these files change frequently, Git will bother you with irrelvant changes regularly. You untrack these files by adding them to .gitignore and by calling git rm --cached <filename>.