The Delivery Newsletter #49
#49 JUN 24, 2020
The Delivery Newsletter

Spark Joy by Running Fewer Tests

Shopify had a problem. Their monorepo contained 150k+ tests. At first, this seems like a good problem. But all these tests had to pass, and they took 30-40 minutes to run. The engineers at Shopify implemented a dynamic analysis system to run subsets of tests based on code changes. While this post is light on how they implemented the system, it documents their clear gains. Without sacrificing stability (they caught 99.94% of legitimately failing tests), the team only runs about 60% of the full test suite with the new system.

Observability and the Misleading Promise of AIOps

"Is AI Ready for Ops? Simply put, no." Danyel Fisher, who works for, spells out several reasons why AIOps isn't the silver bullet that vendors claim. Among those reasons, your AI can't detect anomalies if you keep fixing them:

The net effect is that DevOps teams end up working against their AI: every time it figures out what sorts of alerts are important enough to set off a pager, the DevOps team has the audacity to go fix the underlying issue.

Danyel suggests tracking SLOs tied to user experience as an alternative to AIOps.

Speeding up a Git monorepo at Dropbox with <200 lines of code

Dropbox versioned 250k files with git at the end of 2019. Performance issues appear at that scale. This post covers what the Dropbox team did to improve git's performance. While I don't work on large repos, I think their approach is super interesting:

We created a small fork of Git that measured timing of operations like git status and git pull

Chalk this up as an example of the build-measure-learn feedback cycle in action.

Scaling End-to-End User Interface Tests

Slack's Quality Engineering team documents how it created a reusable automation framework. This post is worth a read if flakey e2e tests haunt your dreams. Here's one takeaway that you could implement immediately:

Instead of relying on product-driven class names or element ids, we add a custom "data-qa" attribute to elements that we need to select for testing purposes.

Let's not override the "id" and "class" attributes for testing purposes!

🔊 Podcasts

The Relationship Between Continuous Delivery and Value Stream Management

Too often, I examine CD at the micro-level. I measure cycle time or how long a job takes in CI. In this episode of The Pipeline, Helen Beal ties CD back to the macro-level. The interview talks about lead time and larger: how long does it take feedback from the user to come back to the user as a feature?

🛠️ Tools

Introducing GitHub Super Linter: one linter to rule them all

There's a tier of code issues that are programmatically identifiable and tedious to search for or debate in code reviews. Let the machines decide! GitHub's Super Linter checks so many languages so that you don't have to.