Think Slow, Act Fast

Episode 45: Better Built By Burkhard

Think Slow, Act Fast

Thinking slow and acting fast is the single most important practice to succeed with big and complex projects. This is the conclusion of Bent Flybjerg and Dan Gardner in their book How Big Things Get Done. The authors have collected the data from big projects for decades, have analysed it and have advised big projects. Example projects include the Guggenheim Bilbao museum, the Sydney Opera, the Empire State Building, airports, nuclear power plants, tunnels, high-speed railways and IT projects.

“Think fast, act slow” […] is a hallmark of failed projects. Successful projects, by contrast, tend to follow the opposite pattern and advance quickly to the finish line.

Bent Flybjerg and Dan Gardner, How Big Things Get Done, p. XIII

Does that sound familiar? I bet it does. Companies rush project teams right into coding and make them skip thinking, because otherwise most of the team members would sit around twiddling their thumbs. Agile hardliners despise activities like planning, architecture and risk analysis, because any non-coding activities are a waste of time and eliminating waste is one of the foremost goals of Agile.

These projects almost inevitably end up with a big ball of mud, which is riddled with bugs, where a bug fix produces more bugs due to intricate dependencies, which are hard to maintain and even harder to extend. After some years, these project teams must decide whether to re-implement their software from scratch or to migrate their software to a proper architecture. The first approach almost always fails. The second approach costs a lot of time and money - roughly as much as developing the big ball of mud. The costs for such disasters are immense.

My database revealed that information technology projects have fat tails. To illustrate, 18 percent of IT projects have cost overruns above 50 percent in real terms. And for those projects the average overrun is 447 percent! That’s the average in the tail, meaning that many IT projects in the tail have even higher overruns than this. Information technology is truly fat-tailed.

Bent Flybjerg and Dan Gardner, How Big Things Get Done, p. 10

Cost overrun for software projects does not follow a normal distribution but a fat-tailed distribution - like the distribution of wealth. If such a project is over budget, then it is dramatically over budget. “Moreover, fat-tailed distributions, not normal distributions, are typical within complex systems {…]” (p. 12). Embedded systems are definitely complex systems. Just think of the 50-80 computers with a total of 10 million lines of code in a car these days.

Companies could avoid such disasters by thinking before acting. They could start with a standard architecture like the ports-and-adapters architecture. They could minimise the team and software dependencies right from the project start. They could make the Continuous Delivery principles and practices mandatory for the development teams. By involving an expert, they would find out early that video streaming from rugged cameras doesn’t work with Qt 6, that a Yocto version out of support for 4 years would entail significant extra costs, or that the chosen display would not be readable in daylight.

By planning, Flybjerg and Gardner do not mean big design up front, the stage planning of SAFe or writing requirement specifications of hundreds or even thousands of pages. Such planning is rarely effective. Instead, Flybjerg and Gardner recommend to plan like Pixar and Frank Gehry (see “Chapter 4: Pixar Planning: Plan like Pixar and Frank Gehry do”).

What sets good planning apart from the rest is […] captured by a Latin verb, experiri. Experiri means “to try”, “to test,” or “to prove”. It is the the origin of two wonderful words in English: experiment and experience. Think of how people typically learn: We tinker. We try this. We try that. We see what works and what doesn’t. We iterate. We learn. This is experimentation creating experience.

Bent Flybjerg and Dan Gardner, How Big Things Get Done, p. 62

This doesn’t sound like the meaningless planning that we dislike so much. Quite the contrary: It sounds a lot like eXtreme Programming and Scrum in their early days 25 years ago. Planning requires “supercharged iteration” (p. 65). So, what is Pixar planning?

At Pixar, movies go through the following planning process (p. 70-72):

• Once a director has great idea for a movie, she writes an outline of “roughly twelve pages”. Other Pixar employees - directors, writers, artists and executives - review the outline. The director updates her outline for another review.

• If the outline is compelling enough, the director’s team writes a script of around 120 pages. The script gets reviewed and modified a few times.

• The team produces a detailed storyboard from the script, makes photos of the 2700+ drawings of the storyboard and compiles the drawings into a crude video. Other Pixar employees and filmmakers watch the video and give feedback. The team refines the storyboards and the video. The team goes through eight or more iterations.

At this point, Pixar has “an extremely detailed and rigorously tested proof of concept”. It has only spent a tiny fraction of the 100 million dollars of the final movie. This is the point of no return, as the production gets really expensive from now on. Pixar creates the digital animations with voice-overs by famous actors. While planning moves at a slow pace, shooting the movie moves at a fast pace - with only minor hiccups.

Pixar planning actively looks for early and frequent feedback. Planning is highly iterative and incremental - and not “a static, abstract, bureaucratic exercise” (p. 79). “Planning is cheap” (p. 74) and mistakes can easily be fixed. Hence, planning reduces the risks of complex projects.

Planning, as I see it, is not merely sitting and thinking, much less a rule-based bureaucratic exercise of programming. It is an active process. Planning is doing: Try something, see if it works, and try something else in light of what you’ve learned. Planning is iteration and learning before you deliver at full scale, with careful, demanding, extensive testing producing a plan that increases the odds of the delivery going smoothly and swiftly.

Bent Flybjerg and Dan Gardner, How Big Things Get Done, p. 76

For me, this sounds a lot like the principles of Continuous Delivery: work in small steps, build quality in and improve continuously. If my customers allow, I like to start my projects with a diagnostic, which is similar to Pixar planning.

• We gather architecturally significant requirements (ASRs) for the embedded device built.

• We discuss possible solutions for the ASRs, record them in architecture decision records (ADRs), and choose one solution.

• Alone or together with the customer’s lead developers, I implement the most important ASRs using TDD, acceptance tests and a CI/CD pipeline. The starting point is typically the ports-and-adapters architecture.

We iterate the three steps above until we are reasonably sure that we can build a product in the given time and budget. The diagnostic typically takes 4-6 weeks. We can then extend the software into a minimum viable product (MVP). The costs for the diagnostic are between $25K and $40K. The costs for developing an operator terminal of a machine easily add up to $400K and more. A diagnostic easily saves 25% and more of the $400K. It amortises its cost quickly.

At the end of the book How Big Things Get Done, Flybjerg and Gardner list “Eleven Heuristics for Better Project Leadership” (p. 185). These heuristics are valuable advice for every project. I am planning to write a newsletter or two about the heuristics Hire a Masterbuilder, Get Your Team Right and Build with Lego (modularity).

My Content

Ports-and-Adapters Architecture: The Pattern

I gave talks about the ports-and-adapters architecture at Qt Day Italy 2021 and at Meeting Embedded 2021. Since 2021, I implemented the architecture twice - once in an excavator terminal and once in the HMI of a home audio system. I learned a lot. In the talks, the Application Core depends on the Machine adapters. This is wrong! Both the Application Core and the Machine adapters must depend on the Machine port (see ASR-8 in the post).

Normally, my posts about architecture are slightly neglected compared to more technical posts about topics like licenses, Qt, and Yocto. This time is was different. The reception on LinkedIn was high with some good comments.

Fav part, “The most important responsibility of a port is to hide the technologies used in the adapters from the Application Core.”

Eugene Barker, Solutions Architect for Pacific Island Government

Yes. [The ports-and-adapters architecture] should be standard and widely used.

Alan Mellor, Author Java OOP Done Right, Test-Driven Development in Java

Even better, two of my favourite newsletters featured my post: Interrupt by Memfault and EmbeddedArtistry by Phillip Johnston. Both monthly newsletter are worth subscribing. I regularly find nuggets of wisdom in them.

Burkhard Stubert published an outstanding overview of the Ports-and-Adapters architectural pattern, which is our preferred software architecture.

Phillip Johnston, EmbeddedArtistry, September 2023

A good case for why ports-and-adapters (or hexagonal) architecture should be the standard for HMI applications.

Tyler Hoffman, Interrupt by Memfault, August 2023

And yes, I am planning to implement the architecture for the simple harvester terminal serving as my running example. There will be posts for implementing and testing the Application Core, the Machine adapter, the Accounting adapter and the UI adapter.