Episode 35: Better Built By Burkhard

Architecture Decision Records

When I advised a professional appliance maker last year why and how to introduce Continuous Delivery, I needed a way to discuss proposed decisions and to document them. Not long before the project start, I had come across the article Documenting Architecture Decisions from 2011, where Michael Nygard introduces Architecture Decisions Records (ADRs).

ADRs were exactly what I needed! And even a bit more. I could use them for decisions about system architecture, team structure, development process - and any other decision.

After the interviews with the development teams and with their stakeholders, managers and executives, I would write down the important topics requiring a decision. I would describe the context, possible options, the consequences of the decision and my recommended decision.

The decision records (DRs) were available on the company’s intranet for everyone to read. So, everyone could comment on the recommended decision. In weekly meetings, a team of managers, architects, senior developers and me discussed the recommended decisions. We revised the records, decided some right away and deferred some to gather more information for next week’s meeting.

Normally, everyone should write DRs. However, the two factions in the organisation - a French and a German faction - preferred to blame each other for having no clue about software development and to insist that their way is the right and only way. I wrote the DRs to get the controversial topics on the table and to discuss them in a structured way.

A big advantage of DRs is that they decouple creation, discussion and decision temporally. Everyone can think through the decision asynchronously. That leads to more constructive and respectful discussions than the fights in meetings. Jeff Bezos, Amazon’s founder, has made asynchronous communication and excellent preparation of meetings.

The reason writing a good 4 page memo is harder than “writing” a 20 page powerpoint is because the narrative structure of a good memo forces better thought and better understanding of what’s more important than what, and how things are related.
Powerpoint-style presentations somehow give permission to gloss over ideas, flatten out any sense of relative importance, and ignore the interconnectedness of ideas.

Colin Bryar and Bill Carr. Working Backwards: Insights, Stories and Secrets from Inside Amazon, p. 83 (Communicating - Narratives and the Six-Pager)

That is definitely true for decision records, although they should not be longer than 1-2 pages. “Enough chitchat” - as Po, the Kung Fu Panda, would say. You want to know how ADRs look. Here you go.

---

Title - an expressive one-liner with a unique identifier.

Status - “draft”, “proposed”, “accepted”, “superseded” or “deprecated”.

Context - an appraisal of the situation with all its technical, social and political influences that made this decision necessary.

Options - each option describes a possible decision how the problem described in the Context could be solved. Briefly discuss the pros and cons of each option.

Decision - the decision taken (e.g., a reference to the chosen option).

Consequences - positive, negative and neutral ramifications of the decision.

---

I added the section Options to Nygard’s original definition. The decision-making process typically unearths two or more options: at least the existing and the new option. Including and discussing alternatives values the contributions from other people. In the rare cases where there is only one option, you can skip this section.

Let me now give you an example ADR from my current project: a driver terminal for an excavator.

---

ADR-004: Video Streaming without Delay

Status: Accepted

Context

The excavator terminal displays the video from a heavy-duty camera. The camera can be either a single camera or a bird-view camera system. Both cameras stream their video over Ethernet packing H.264 frames into RTP streams. They expose the stream through a URL.

The terminal uses Qt Multimedia from Qt 6.4 to display the video. It displays the video with a 3-second display. We found out that Qt 5.12 and Qt 5.15 display the video without any noticeable delay. In Qt 5, we can specify a GStreamer pipeline instead of a URL. In Qt 6, this is not possible any more.

We are two weeks away from BAUMA (the world’s biggest trade show of construction machines every three years), and six weeks away from the final release date.

Options

Option 1: Downgrading to Qt 5.15

We know that upgrading a complete harvester terminal from Qt 5.12 via Qt 5.15 to Qt 6.2 takes three days. The excavator terminal is significantly simpler. We estimate that the downgrade will take roughly 2 days.

The customer can show a fully working excavator at BAUMA to prospective buyers. We gain time to to fix the problems ourselves or wait until others do so. Qt 5.15 will see support from The Qt Company for at least another year, probably more.

Option 2: Fixing the GStreamer Plugin in Qt 6.4

By setting the latency directly in the GStreamer plugin, we could fix the delay for the single camera. However, the fix doesn’t work for the bird-view system. Our best idea is to resurrect the ability to specify GStreamer pipelines instead of URLs. We estimate the effort at 3-4 weeks with a very high uncertainty. The Qt Company stated in a comment to our bug report that they will not resurrect this feature in Qt 6.

Option 3: Fixing the FFmpeg Plugin in Qt 6.4

The Qt Company closed another bug report, because it will move from the current GStreamer backend to an FFmpeg backend. Hence, it will not invest any time in fixing issues in the GStreamer backend. For Qt 6.4, the FFmpeg is a technical preview. Qt 6.5 will see its official release.

We tried out video streaming with the FFmpeg backend from Qt 6.5 alpha. The video played with a delay of 2-3 seconds. As for Option 2, we can try to fix the problem ourselves (effort estimate: 3-4 weeks). We can also wait for the release in Q2/2023. In the meantime, we could support The Qt Company in fixing the problem. We could at least test their solutions and give them feedback.

Decision

We will downgrade the system to Qt 5.15 (Option 1).

Consequences

We must spend 2 days to downgrade the terminal from Qt 6.4 to Qt 5.15 now. Later, when video playback works without delay in Qt 6, we must spend another 2 days to upgrade from Qt 5.15 to Qt 6 again. Our customer must upgrade their excavators to the new Qt libraries. They can do an OTA update.

We gain enough time to work with The Qt Company to fix the problem or fix it ourselves. We are not under the pressure of an imminent deadline.

---

We did the experiments and had the discussions as shown in the above ADR in mid October 2022 - two weeks before BAUMA. We should have worked out this decision more than 6 months earlier, when the customer decided which cameras to use and when we got our hands on the cameras. We could have avoided the last-minute stress and a slightly miffed customer.

ADRs are the glue that keeps a structured and asynchronous decision-making process together (more in Scaling the Practice of Architecture, Conversationally below). Discussions and meetings will be less heated and more constructive.

My Content

Architecture of Qt Embedded Systems: Getting Started

The post contains 50+ questions that you should answer when you build a Qt embedded system. Most questions have multiple answers: options in ADR speak. In the Context of your specific system, you choose one answer as your decision. This decision has certain Consequences. In short, you can write down the discussions and answers for each question as an ADR.

My Talk “Hexagonal Architecture: The Standard for Qt Embedded Applications” at Meeting Embedded 2021

Before you start implementing an embedded HMI, you should capture the Architecturally Significant Requirements (ASRs). ASRs are crucial in shaping the architecture. Slide 10 of the talk gives an ASR: The driver sees the current engine speed in the HMI. Neither the business logic nor the GUI adapter should know that the embedded HMI receives the engine speed over CAN. These are important architectural decisions worth recording in ADRs.

For the Machine adapter (slide 16), you need to decide whether the encoding and decoding of CAN messages happens in the main thread, in separate threads for each CAN bus, in a separate process or on the microcontroller of an i.MX8. Again, it’s worth discussing the pros and cons of each option and document the decision in an ADR.

People can read these ADRs and apply the same reasoning to other adapters. For example, the rest of the embedded HMI shouldn’t know that you use an SQL database in the persistent storage adapter.

People can also revisit earlier decisions and change them. For example, you can later replace parts of the SQL database with a time-series database. Or, you change your mind about the Machine adapter. You move the CAN encoding and decoding from the main process into a process of its own. The original ADRs will be marked “superseded” or even “deprecated” and the new ADRs “accepted”.

Around the Web

Joel Parker Henderson: Architecture Decision Record (ADR)

Henderson lists many useful resources about ADRs on his GitHub repository. You find suggestions for good ADRs, tips how to write ADRs with git, many ADR templates, example ADRs and more.

Andrew Harmel-Law: Scaling the Practice of Architecture, Conversationally

I wish I had read Harmel-Law’s article before I embarked on the project with the professional appliance maker in 2022. The article would have saved me inventing my own decision-making process based on ADRs. Harmel-Law introduces the Advice Process, a simple decentralised decision-making process based on ADRs.

The Rule: anyone can make an architectural decision.

The Qualifier: before making the decision, the decision-taker must consult two groups: The first is everyone who will be meaningfully affected by the decision. The second is people with expertise in the area the decision is being taken.

Decision-takers must not agree with the advice from the two groups. However, they must gather the advice, listen to it and record it in ADRs. The Advice Process encourages people to have important conversations and to reach better decisions. As it doesn’t force people to find consensus, the Advice Process avoids the discussions going in cycles forever and getting more heated. And don’t forget: Nothing is cast in stone! Most decisions can be changed.

Harmel-Law extends the definition of ADRs by the section Advice. This section lists the advice given by the two groups consulted. The advice should be reflected in the options considered for the decision.

The Advice Process has three more supporting elements:

• The Architecture Advisory Forum is a weekly meeting, where people have the necessary conversations to move the status of the ADRs to “accepted”.

• The team maintains a list of architectural principles recorded in ADRs. These principles (e.g., the principles of Continuous Delivery) are the guardrails for architecture decisions.

• A technology radar appraises the current trends in techniques, tools, platforms and languages & frameworks - specific to your organisation.