Continuous Integration and Trunk-Based Development (part 1/2): coding patterns
I often describe the advantages of CI and TBD: today, I will make it more concrete by sharing with you some real-world use cases and coding examples from my experience.
📄 📢 Newsletter Update 📢
🎉 Milestones Reached 🎉
🎉 Hey, developers! Welcome back to Learn Agile Practices, the home of continuous improvement and learning for Software Developers.
📌 In case you missed it
Last week in this newsletter, I wrote about Lean Thinking and Lean Software Development: starting from Toyota Production System, I shared with you the impact of applying Lean to software, what principles it suggests, and the positive outcomes of applying them. If you haven’t yet, you can read it in our archive.
And for those of you who speak Italian, last week on my video podcast:
The Weekly Pomodoro #2 - OOP Misconceptions
Watch episodes on YouTube here, or listen on Podcast using Spreaker.
🔥 Personal Coaching
If you want to speed up your growth as a software developer with Agile practices such as TDD and Clean Code, you should consider my coaching service: one-on-one guidance from me, Dan the dev, an experienced Agile developer who will help you develop the skills and knowledge you need to succeed in 12 1-hour sessions.
Take advantage of my First Lesson Free policy: enjoy the first free session and get to know me to see why I am the right person for you to improve your professionalism as a software developer.
Discover more 👉 here.
Thanks again for being a part of our journey! 🙏
📣 Just a friendly reminder, folks! In our welcome email, you received a sweet discount that's valid for any of my products and services. Don't forget to take advantage of it - it doesn’t expire! 😉
Continuous Integration and Trunk-Based Development (part 1/2): coding patterns
Introduction
Hello, developers! 🚀
Since I started this newsletter, I’ve talked of a lot of topics related to programming, Agile practices, and the work of a Software Engineer in general - but for sure, there are some of them that I talked about much more than others, with the top 3 probably composed by Test-Driven Development, Continuous Integration, and Trunk-Based Development.
Today, we talk again about the last two: after a quick recap of what they are about, I will share with you the coding patterns that enable CI and TBD by hiding work in progress and some real-world examples where I applied those practices.
This issue aims to share a concrete guide for those who want to start applying these practices to help you achieve them in your practices first, and then in daily work - the coding patterns tips will be simple but clear, enough to clarify the approach and start practicing it.
In this first part, we will introduce the topics in short, and then deep dive into the coding pattern I want to talk about. Then, next week, in the 2nd part, I will share 2 real-world examples where I applied those patterns in my experience - with my final reflections on the topic.
I wanted to put some code examples, but this kind of pattern requires a lot of context to build a meaningful use case, and it would still be hard to understand only by reading, so I thought it wasn’t worth it and gave up the idea.
The principles
Continuous Integration is one of the most powerful concepts to master in the Software Development World, probably the one that most impacted how I see this job.
CI is the activity of very frequently integrating work to the trunk of version control and verifying that the work is, to the best of our knowledge, releasable.
Implementing Continuous Integration implies some practices that are embedded into it, and one of those practices is Trunk-Based Development.
Trunk-Based Development is a methodology where changes to code are integrated directly in the main branch, without any other branch in the middle, at least once per day.
If you want to implement CI in your workflow, you must start using the set of practices that together make CI, including Trunk-Based Development: in other words, you can’t say you do CI if you don’t do TBD.
Sadly, that’s what most company does, at least in my experience - they use feature branches, typically with a very long life of multiple days, and then think they are doing CI just because they have an automated pipeline.
That’s just wrong.
Anyway, when it comes to the idea of very frequently integrating work to the trunk, where very frequently means daily or even multiple times per day, the typical concern is: how do we hide work in progress if I have to merge unfinished work to master (and even release to production, if I also want CD)?
There is a great quote I agree with, about this topic. The quote comes from Bryan Finster, who as a guest on an issue of the Crafting Tech Teams newsletter once said:
Software Developers have been trained to deliver complete features. That’s the problem. It’s one of the hardest habits to break.
That is so much true!
90% of software developers I know have this bias: we were taught to release fully completed features, and see the release of the software as the last piece of a long process instead of part of the daily work; this habit makes the question about how to hide work in progress seems to be a very hard and complicated question to find an answer to.
The reality is that this is a problem that has already been solved: there are multiple patterns that we can use to keep the work in progress hidden from the user, allowing us to release our code without any issue on their side.
We will dive into the most common patterns in a while, but I want to point out an important thing before; let me share a quick example with the most known pattern to hide work in progress: Feature Flags. A basic implementation could be as simple as a boolean value in the configuration, at a user level, that allows you to decide if a user has access to that feature or not; this has some consequences on code implementation, of course, because you’ll have to handle this configuration value to decide whether to execute the behavior or skip it.
From this example we can highlight a characteristic that is shared by all the “work in progress hiding patterns”: it will probably require some additional work to hide the work in progress; we have to think about how to hide it and make some coding to achieve this purpose.
So the question here is: is it worth it?
Of course, it is, and the reason is quite simple: the investment we take to hide our work in progress and enable the release of our code is completely repaid by the unplanned work we will avoid after the release; it is proved that implementing CI in our daily work will drastically reduce the unplanned work we will face later on during the process of completing the feature.
Maybe you remember it from our Lean Software Development issue, but let’s say it again: unplanned work is one of the biggest sources of waste in software development, therefore reducing it should be one of our targets.
Here, we are reducing unplanned work thanks to a little additional, but plannable, work: it means we are investing 1 today to be sure to not waste 10 tomorrow; we are accepting a bit of additional conscious work, that we are aware of and therefore is plannable, to ensure we avoid some unplanned, unexpected work later that will cause waste, issues, waiting time and context switch to be solved.
As always, if you want to deep dive into such topics, you will find a lot of useful resources in the Go Deeper section at the end of this issue.
Coding patterns
Keep the API Hidden
The pattern
The easier way to hide code to be executed or noticed in production is to hide its interface: just make sure that the path to the feature is either hidden to the user or the last piece you release. In a well-designed system, such interface elements should be minimal and thus simple to add.
Practical Tips
Hide (or release last) the route of the API - I typically like to do this with a stubbed response that respects the real contract
Hide (or release last) the UI that enables the user to interact with the feature (it can even mean hiding an entire component or page)
Feature flags
The pattern
A boolean value, typically based on a constant, a parameter, or a configuration value, that a class can use to decide if we want to execute path A or B: this is basically what Feature Flags are. This is a simple way to be able to activate the new behavior for testing, while it’s still disabled in production.
Practical Tips
Build a Feature Flag class or use an existing library (an example here) - some tips:
prefer Strategy pattern to implement the two different paths
be careful to pick the granularity you need for the flag: if it’s too big it can become useless, if it’s too small it can become too complex to maintain, so make sure it’s worth it
Feature Flags are temporary most of the times (at some point only one of the 2 paths usually survive, whether it was an experiment or a change) so be sure that the implementation is also easy to remove
Make sure that you can enable/disable the flag quickly: if your release process is fast enough, even changing code can be fine - otherwise favor a DB table configuration or something else that allows you to do it in seconds
Parallel change
The pattern
Also known as expand/contract, these patterns suggest expanding the current code instead of changing it: for example, if we have to rename a database field, we should:
create a new one instead
start writing in both fields
copy data from old to new field
start reading from the new field
remove the old field, now unused
This allows us to make each of those steps reversible easily, something impossible if we directly change the name of the column.
Practical Tips
The main objective when doing a change is to make it safe, ensuring we can rollback in case of issues, and avoid the need for synchronous changes with clients.
Another example is creating a new version of a public API:
release the new version first, while still keeping the old one
ensure that no one uses the old one anymore (providing a deprecation date might be good enough in most cases)
remove the old version of the API
The idea of Parallel Change is pretty flexible: in general, stop thinking of software development as changing code, and think more of it in terms of expanding what exists first, and then removing what’s not needed anymore
Dark Launch
The pattern
When you have to add a new behavior, you can make the code execute this behavior without impacting the existing one: this way, it will be invisible from the user's perspective, nothing has changed from his point of view - but in reality, you are executing that code in production to test its impact and monitor things like performances or saved data.
Add your behavior, and just ignore the results from the application point of view - but log or save any info about what has been done when it goes well, or what happened when it broke up.
Practical Tips
This pattern best suits use cases where the new behavior is enhancing something already existing that doesn't require any additional interaction from the user.
Ensure that the “dark” code is under a try/catch that handles all errors: this way, you can save all the info you need from the error (in logs or everywhere else you might need it) and allow the code to move on without breaking
function foo(): void { // do something before try { // execute the new behaviour here } catch (Throwable $exception) { // save the data you need about the error, then let the code move on } }
Branch by abstraction
The pattern
The idea here is that we can introduce a different alternative behavior, but instead of making it used by everyone immediately, we use an abstraction layer to route only part of the requests to it: it can mean that only a specific component uses the new behavior at first, or just a percentage of requests.
Thanks to this approach, we can iteratively roll out the change and increment the usage until it replaces the old behavior completely.
This pattern has some similarities with Parallel Change: the main difference is that here we progressively go from old to new behavior, while in Parallel Change it’s more of a direct switch after the two behaviors live together.
Practical Tips
If the code you need to change is used directly by others, build the abstraction layer first
If you are doing this on some legacy or badly tested code, you can take advantage of the abstraction layer to add tests to that behavior
Until next time, happy coding! 🤓👩💻👨💻
Go Deeper 🔎
📚 Books
Continuous Integration: Improving Software Quality and Reducing Risk - The authors first examine the concept of CI and its practices from the ground up and then move on to explore other effective processes performed by CI systems, such as database integration, testing, inspection, deployment, and feedback.
Trunk-Based Development And Branch By Abstraction - An all you need to know reference book about trunk-based development, Branch by abstraction, and related software development practices. Many diagrams throughout, and a sections on working out how your company can get from where you are to trunk-based development, CI, CD, and all that comes with it.
📩 Newsletter issues
📄 Blog posts
🎙️ Podcasts
Trunk Based Development vs. Gitflow [The Rabbit Hole podcast]
Trunk-Based Development And Feature Flags With EJ & TJ [The CTO Studio]