What’s the problem with Flake you say?

When you’re one of 42 developers spread out across 5 timezones waiting to merge work into master, flaky failures in CI can mean the difference between getting 5 tickets done and merged… versus 0. This problem is severely compounded when software work has complex dependency chains — eg: Mary’s frontend PR depends on Joe’s backend PR.

What can be done?

A precondition to correcting a problem or optimizing a system is understanding its points of failure and bottlenecks. This is not fun by the way — you have to look at the data, and in some cases, be willing to do some trial-and-error discovery. That’s what I did with one CI pipeline in 2020. Below is a brief show-and-tell of tools and processes I put into place in the summer and fall of 2020 to quantitatively mitigate our problem of flaky failures.

What were the rewards I saw?

• Sped up the build — from 95 min to 35 min

• Reduced flaky failure rates from .63% to .02% — approaching the probability of being struck by lighting ⚡ (according to the National Lightning Safety Council)

• Made failures a lot more actionable — people were notified in Slack right when a failure happened, with clear prompts to take the next logical action: investigate the logs, talk to the owner of the check, report a flaky build

• Allowed my team to put more code into production each day — as the saying goes, coffee is for closers — and in Software, you can’t close without merging to prod

Details

• In July 2020, we started harvesting metrics from Jenkins about our automated tests on each build – things like…

  • Git branch, build #, and Git commit author

  • Time taken to execute

  • Total tests passed

  • Total tests failed

  • Total tests skipped

  • Overall Status

• We took inventory of our automation

  • Owners of tests were clearly identified

  • Every test was clearly described in plain English so that new developers, designers, and product owners could understand what each test was intended to validate without having to decipher a single line of code. This made it easy for the right people to chime in and say things like… Why the f*** are we spending 7 minutes testing THIS?

  • ☝🏽 This triggered an intense months-long Marie Kondo inspired drive to rid our pipeline of garbage tests that no longer sparked joy

• A Shotgun Webhook integration with CI alerted engineers in a Slack channel to test failures as soon as they occurred – even before the build was complete. Notifications included the test description, the Jenkins build URL, and a button to create a Jira ticket right on the spot so the test owner could quickly fix up the test

• Flaky test failure rates went down

  • Jul 2020: .63%

  • Aug 2020: .06% (* this is an outlier because many tests were simultaneously disabled in late July and August to improve their stability, then re-activated later on)

  • Sep 2020: .23%

  • Oct 2020: .13%

  • Nov 2020: .09%

  • Dec 2020: .05%

  • Jan 2021: .02%

• As an organization, we acquired the new habit of zero tolerance for flaky tests, and a solid expectation of full confidence in our tests

• The quality of our CI/CD pipeline improved precipitously

  • Test stability: 97% improvement

  • Avg time to remediate flaky tests: Down from 11 days - 0 days

  • Avg time flaky test tickets spend in the open state: Down from 8.3 hours - 40 minutes