Continuous Integration and Trunk-Based Development (part 2/2): real-world examples
I often describe the advantages of CI and TBD: today, I will make it more concrete by sharing with you some real-world use cases and coding examples from my experience.
📄 📢 Newsletter Update 📢
🎉 Milestones Reached 🎉
🎉 Hey, developers! Welcome back to Learn Agile Practices, the home of continuous improvement and learning for Software Developers.
📌 In case you missed it
Last week in this newsletter, I sent the first part of this issue about Continuous Integration and Trunk-Based Development: there you can read the introduction to the 2 topics and the main coding patterns that you can use to hide unfinished work and therefore be able to merge and release daily, multiple times per day, with ease. If you haven’t yet, you can read it in our archive.
And for those of you who speak Italian, last week on my video podcast:
The Weekly Pomodoro #3 - Lean Software Development
Watch episodes on YouTube here, or listen on Podcast using Spreaker or Spotify.
🔥 Available for mentoring and coaching
I’m always available for some mentoring and coaching to other software developers! Feel free to reach me: we will know each other and discuss what path can be best for you, from a free monthly mentoring path to a more frequent one or a 1-to-1 coaching service specifically on your needs!
👉 Feel free to reply to this email or reach me on any social media: all the details on my website.
Thanks again for being a part of our journey! 🙏
📣 Just a friendly reminder, folks! In our welcome email, you received a sweet discount that's valid for any of my products and services. Don't forget to take advantage of it - it doesn’t expire! 😉
Continuous Integration and Trunk-Based Development (part 2/2): real-world examples
Introduction
Hello, developers! 🚀
Here is the 2nd part of this big issue related to Continuous Integration and Trunk-Based Development - and today, I will finally share with you 3 real-world examples where I applied the coding pattern I described last week.
This issue aims to share a concrete guide for those who want to start applying these practices to help you achieve them in your practices first, and then in daily work - so hopefully, real-world examples will make it easier to understand and spot use cases in your work to start using the patterns by yourself.
Real-World examples I applied
A food delivery API backend
The context
In a food-tech company, I was working on a Food Delivery B2B product, and I was able to work on that almost from the beginning. We had a couple of customers who were ready to take advantage of our food delivery services to offer them to their employees as an additional lunch benefit option, and we had to build the entire platform where they could place orders for the day.
As I mentioned, it was “almost” the beginning: we had a few months of work already done on a generic set of APIs that could serve any generic e-commerce/food delivery app, with way more API routes and features than we needed - but still no frontend and no clear idea of what the core of the product should be.
The use case
As you can imagine, a lot of work had to be done, and a lot of it was outside of code - but for this issue, we will focus on coding. We had to sustain the business in a better way than was done before, therefore we had three main objectives:
build the frontend MVP
build the missing backend APIs for the MVP
clean up existing code (clean up here means multiple stuff: the code had a lot of code smells, tests were there but it was only integration tests at controller level, they were very brittle and weren’t testing enough, most often only 200 as HTTP response code, and some features weren’t even needed at all)
Of course, the clean-up had to be done sustainably, because we still had to serve the business to achieve the MVP release successfully and less waste possible (zero waste was impossible at that point).
The approach
We applied mainly two patterns here: Parallel Change and Dark Launch.
Since when I joined, I pushed to move into an MVP version of the software for the first release - of course, the backend would have to live with existing useless code somehow, but at least we could build the frontend from an MVP perspective and also use the set of features used in the frontend as the list of “actually required” features.
Taking inspiration from Dark Launch, we moved all the existing routes of the APIs under a useless route group that no user had access to - and then built a clean version of the active group of APIs including only useful routes. The routes were still there in the codebase, but no one could access them. This made it easy to make decisions about how to improve the code, especially when we had to clean up some code that was used in multiple places: every feature that didn’t enter the MVP phase was removed safely unless we were already sure that it was a feature that we had to include in a new release very soon (example: special rules in discount).
In some cases, we had to keep a feature: the behavior was fine but we had to refactor somehow to build a code and test we could trust… (1 ensure tests were good, 2 copy-paste the feature, 3 refactor in parallel change to test the behavior was the same)
This way, we were able to face this issue iteratively and progressively, keeping it sustainable and completely transparent to the users.
Smart Fridges iterative rollout
The context
In the same company and product context as the previous example, we also offered Smart Fridges to our customers, and one specific use case is to use it as a vending machine for healthy, high-quality products for both lunch and snacks. The UX was very smooth: you could just open the fridge via QR code with your phone, take what you wanted, and go away, and we automatically calculate what you have to pay and take it from the wallet.
The use case
After the first version release, we needed more marketing-related features like using coupons, discount codes, special offers, highlights favorites products, etc. From the user perspective, what we wanted to do is add a pre-opening screen: after using the QR code, the fridge wouldn’t open immediately anymore - instead, we show a page dedicated to the fridge, with an “OPEN” button; that page enabled us to new interactions with the customer. While from the user's perspective, it wasn’t a big change, technically it required some changes on the fridge opening and payments, basically the two most delicate pieces of the puzzle.
The approach
We approached this with two patterns in mind: Parallel change and Feature flags; but then also Branch by Abstraction came in handy.
On the backend, we applied Parallel change by creating a completely new API version for both the opening and the payment, starting from a copy of the existing one. Since the opening API became “pre-opening”, we also had to create a new API specifically dedicated to opening. These APIs were also released in production, but no one could use them because the front end wasn’t using them and we didn’t give access to anyone to those routes.
On the front end, we applied Parallel change by creating the new “pre-opening” page, which was a brand new page - it simply wasn’t used by the user flow, so no one could see it. We didn’t hide it completely (meaning that if someone guessed the URL he could have seen it) because it was in any way related to the QR code and auth strategy with the fridge, so it wouldn’t have worked anyway, and that was enough for us. If needed, we could have added a way to hide it completely.
Then, once released in production, we had to test it from a user perspective (we had unit and integration tests of course), so we implemented a feature flag at a fridge level: this allowed us to decide what to do when a user tried to open a fridge based on a variable related specifically to that fridge.
This was also a simple implementation of Branch-by-abstraction: the interface that decided which open strategy to use was based on the feature flag in this case, but still, we had an interface that was taking that decision and redirecting the user to the right strategy - we could also have a more granular approach to send only a percentage of users of a single fridge into the new strategy, but we didn’t have thousands of users so we didn’t need this level of safety.
Thanks to this flag, we were able to roll out first on our test fridge in our office and run the first tests - then, one by one, we activated the new opening on all fridges. The iterative rollout was already simplifying things, but the flag was also helpful to rollback immediately in case something didn’t work well.
Once the new feature was stable enough (in our case, we waited for a couple of weeks), we just removed the old code and the feature flags.
Feature Flag for a progressive release
The context
In a more recent experience, I had to work on a project dedicated to upgrading the database of the historical monolith database from MySql 5 to 8. In case you don’t know, with this upgrade there is an automatic cache feature that is deprecated, and our application was taking so much advantage of it that upgrading was causing a drop in performances on some of our main pages for SEO: we couldn’t afford that.
The use case
From the beginning, I was very concerned about the work to do because the codebase wasn’t covered by enough tests, and we couldn’t trust the existing ones - I also didn’t have confidence in that codebase either because I was new.
What we had to do in practice was replace the (dozens of) queries from the most important pages for SEO with a single GET read operation from REDIS: an async data calculation would have prepared the required JSON structure to replace the queries.
Some problems: no one had enough knowledge to build a JSON sample, so we had to build it step by step, and no one was aware of what data from those queries was used - we were pretty sure that a lot of fields were unnecessary since most queries were getting all the fields from all the tables, but we couldn’t understand which ones were unnecessary upfront (and even if we could do it, it would have taken ages).
The approach
An iterative, exploratory approach was the only safe way here but we also needed a way to test the changes (non-regression kind of testing, since it’s like refactoring work here).
Again, Parallel Change, Dark Launch, and Feature Flags were our safety net here.
We put together those patterns to build a cascade structure:
via Feature Flags at a page level (a simple boolean value on a database configuration table) we could decide for each page if the data was coming from DB or Cache (with a Strategy Pattern implemented on code through the Feature Flag)
via Parallel Change, we handled some shared code that was unsafe to refactor by copy-pasting it in a version dedicated to cache, where we replaced DB queries with data from cache - once it was stabilized as a good solution, we removed the duplication
and, last but not least, in a sort of a Dark Launch approach, we also implemented the Cache strategy in a way that catches any Exception that might happen in the process: if that happens, we log the exception (so that we could collect all the data issues to fix) and then we transparently fallback to the DB strategy so that the page keeps working from the user perspective
Again, some additional work during development put us in a situation where we were able to avoid any real big issues to reach production - we had some small issues, but it was easy to handle them because the user wasn’t seeing anything weird, and we could always turn the flag OFF if needed.
Conclusions
As you might have noticed, one pattern does not appear in the examples I chose: the reason is that I mostly used those in more simple situations.
For example, I used the Keystone interface sometimes to release a new API without setting up the route where the Controller would respond; sometimes it’s even enough if the frontend is built after the backend, so you can just release the new API but no one can use it. This is useful when we work with separated frontend and backend task (I usually share the API contract before, and release a stub respecting that contract, to enable async releases of backend and frontend - just be careful to replace the stub with the real implementation before allowing the users to see this) but it’s also helpful when we work in Pair/Mob Programming (in this case, we can just build the API on the backend and then work on the frontend after that).
Branch by abstraction has a small example, but it’s the best I can provide from real experience - in any case, you can use it together with Parallel change to build a class that behaves as a sort of Load Balancer between old and new logic, and even configure how many requests to direct to the new logic.
In general, such techniques are powerful and pretty useful, and I strongly suggest everyone that who wants to become a Senior become confident in them because they are great ways to avoid issues in production in a safe way.
Until next time, happy coding! 🤓👩💻👨💻
We must avoid unplanned work at any cost: a little planned cost today is far better than a much higher unplanned cost later.
Go Deeper 🔎
📚 Books
Continuous Integration: Improving Software Quality and Reducing Risk - The authors first examine the concept of CI and its practices from the ground up and then move on to explore other effective processes performed by CI systems, such as database integration, testing, inspection, deployment, and feedback.
Trunk-Based Development And Branch By Abstraction - An all you need to know reference book about trunk-based development, Branch by abstraction, and related software development practices. Many diagrams throughout, and a sections on working out how your company can get from where you are to trunk-based development, CI, CD, and all that comes with it.
Feature Flag Best Practices - With this practical book, software engineers will learn eight best practices for using feature flags in production, including how to configure and manage a growing set of feature flags within your product, maintain them over time, manage infrastructure migrations, and more.
📩 Newsletter issues
The Difference Between Design Thinking, Lean Startup, and Agile [Strong Opinions, Weakly Held]
Why are we so afraid of Trunk-Based Development? [Learn Agile Practices]
📄 Blog posts
🎙️ Podcasts
Trunk Based Development vs. Gitflow [The Rabbit Hole podcast]
Trunk-Based Development And Feature Flags With EJ & TJ [The CTO Studio]
Ep. 5: Trunk-Based Development [The Continuous Delivery Podcast]
A conversation about trunk-based development [Tiny DevOps podcast]
🕵️♀️ Others
MinimumCD [Reference website for Continuous Integration and Deployment/Delivery]
Trunk-Based Development [Reference website for TBD]
Just commit to master, please - Hot to actually implement CI [My talk slides]