William of Occam, medieval logician and scholar, famous for being stingy with words. Probably would’ve been a fantastic author of dead simple bug repro steps.
ORF is an acronym for Occam’s razor format. It is not real. There is no chance it will become a W3C standard. I made it up years ago (jokingly at first) when I started managing a backlog. But I enjoy acronyms, so I held on to it. Here’s the quick and dirty:
Some bug tickets are unclear despite containing lots of information. Some are very good, despite being tersely worded, sometimes profanity-laced – even ungrammatical. I wanted to better understand why: what makes a good bug ticket good, and a bad one bad.
This blog post is an informal summary of those findings, and a breezy presentation of some simple patterns that have worked well for me and my current teammates.
Are badly written bug reports really a problem?
Yes. In the U.S. alone, the annual estimated business cost of bad writing is around $400 billion. It's likely that the proportional share of that burden on software companies is even higher, when you consider the difficult-to-measure compounded costs of allowing bad writing to leach into the workflows of people who produce secondary pieces of writing. For example:
Technical writers revising manuals because of users' inability to understand how a piece of software works
Product owners and dev teams huddling to clarify details of poorly worded tickets
Developers conducting technical investigations and code reviews, later rendered moot/unnecessary
Automated processes provisioning servers (and wasting electricity) to run CI builds against code that is later removed
Attorneys convening to draft, polish, and revise EULA's – only to repeat this process because of a misunderstanding
It sounds absurdly obvious, but it’s worth spelling out: engineering teams’ ability to solve problems is crucially dependent on clearly-worded descriptions of problems. Any team member should be able to quickly infer intent from written artifacts with a minimal time investment.
The following recommendations zero in on a tiny fraction of this problem – writing effective bug repro steps for dev teams.
1. Follow a dead-simple template
When writing bug repro steps, you should move toward an organizational standard (the simpler the better!). Here’s the one I’ve been using for years:
Repro Steps (sample)
Assumptions
Your target site is either a local docker setup or a standard CloudOS provisioned site
Your target site is running v8.18.x or higher, and uses the default template
Steps
Login as the default admin user
Navigate to any entity query page (eg: Projects => {Project} => Shots)
Set the page mode to either list, task, or card mode (using the mode switcher)
Toolbar => Fields => Manage Fields
Wait for the first modal dialog to render
Click + Add a new field
Enter a 256+ character-length string into the New Field Name text input
Click Next
Wait for the second modal dialog to render
Click ‘Create Field’
Expected result
Client-side validation catches the excessively long field name before sending it to the server, and after clicking ‘Create Field’.
The UI presents a clearly worded warning, prompts me to correct my mistake, and allows me to resume the field creation workflow where I left off
No server error results
Actual result
The configure field dialog appears momentarily, then goes away, replaced by a server error dialog
The Server error dialog displays this message:
Please pass on the following id for the error so we can trace what happened: etc.The page is in a recoverable state, the Manage Fields dialog is still present on the page, and no modal overlays are visible. There are no JavaScript errors in the console.
2. Explain everything that isn’t obvious
The “teenager disarming a bomb” analogy
When I was in the army, I handled C-4 as part of my job. I was eighteen years old, and up until that point, had only seen explosions in movies. I was still new to the military, scared, and hungry all the time. The EOD specialists at Fort Leonard Wood fully appreciated all of that, and calibrated their instructions accordingly. They kept things simple, gave me (and everyone else) a single instruction at a time, used plain language, spoke in short sentences, and always used the active voice. They understood that people’s lives might one day depend on the things I learned with them. I think of that experience when I write instructions for a general audience – for total strangers whose incoming level of expertise is unknown; people who might be experts, but could also be interns or new hires.
In general, when you’re writing for a person who might be a total stranger to concepts and procedures you take for granted, a person who will likely be reading your words under duress, with a lot of other things on her mind, show empathy for that person. And don’t disrespect her time. You will one day be that person.
2.1. Follow Occam’s Razor
There’s no need to get fancy here: just keep it simple. With respect to bug repro steps, what’s the minimum number of instructions you could provide that would guarantee that a reasonably smart high school student could follow your steps, and reproduce your bug? Whatever that set of instructions is, that’s your goal. That’s the secret sauce. Pretend that your listener is a player piano, and your steps are the punchcard instructions. They should be mechanistic, unswerving, and precise.
2.2 Create awesome Titles
The title of a bug should be to the point and information-dense. Model your’s off of newspaper headlines. Don’t use subjective words like broken, busted, strange, weird, buggy. In particular, don’t write things like this…
Example title: Groups acting weird on this page
“Weird”? This is not a clue? Also, avoid the word “this” like the plague when referring to things. If your bug requires the existence of a snowball that some kid named Vince made last thursday, its reproducibility goes out the window once we hit 40º. In general: This page might not exist “tomorrow”.
Suggested fix: Grouping headers with long strings do not properly truncate on entity query pagesExample title: Bug happens when creating a note while viewing a filtered page of notes
This is circular and unhelpful. When composing a title for a bug ticket, don’t use the word “bug” unless it is a literal string value. Be literal and crystal clear. If clicking a button results in a potato materializing on your desk, say that. Don’t say “totally random thing appears out of nowhere after clicking a button.”
Suggested fix: Creating a note while viewing a filtered page of notes results in a server error
2.2. Use the active voice and keep things short
When it comes to writing repro steps, keep each step as short and atomic as possible. If necessary, break apart complex multi-step procedures into sub tasks.
The following is too long…
- Go to preferences page and find the Big Data setting. It should be in the advanced section. You might have to click around a bit.
- Clear out Big Data, save the prefs, then reload the page and go to a new project.
This is preferable…
- Go to the preferences page
- Scroll down to the accordion section title “Advanced”
- Click “Advanced”
- Locate the text input preference labeled “Big Data”
- Click into the text input and clear out its value
- Scroll all the way up to the top of the page
- Click “Save Preferences”
2.3. Avoid technical jargon
Don’t write stuff like this…
- Up all your services
- In a second shell, tail the production log and grep for “retry”
- After you see 50 or so lines, kill the passenger queue
- Check the DNC for any unicode characters
Bug tickets aren’t invitations to a dork bonanza. Don’t assume that everyone knows how to start and stop docker containers, grep text files, tail a log file, etc. There are dev teams where such things are common knowledge, but not every software engineer on every type of team would know how to carry out instructions like the ones above.
2.4. Use Visuals
Screenshots and/or screencasts (videos) can be a huge boon in cases where a complex series of terminal commands is unavoidable (ie: tail the production log). Sometimes, a 15 second video or animated gif is a real life saver. If it takes you 30 minutes to gin one up, that’s a time investment that could have enormous downstream benefits.
2.5. Provide everything needed to carry out the steps
If code or specific files are required, provide them. For example…
- Go to the uploader at <base url>/upload
- Upload a .dpx file and wait for it to transcode
- The service will fail on 2-frame .dpx files output by Toon Boom Storyboard 6
In the case above, if you don’t provide a 2-frame .dpx file output by Toon Boom Storyboard 6, guess what… this bug will immediately be closed with status = f**k you.
How to fix this up…
- Go to the uploader at <base url>/upload
- Upload the sample file attached in this ticket –toonboom6_bug_repro.dpx
- Expected result: details…
- Actual result: details…
2.6. Don’t give instructions that require backwards time travel
For example:
- Create a polymorphic field on Sequence, with the system name
field_xyzthat points to Asset and Element
- Create a Sequence with the bare minimum of attributes (ie: don’t set itsfield_xyz)
- Delete the Sequence with a CRUD.retire
- Undelete the Sequence with CRUD.revive.
- Login to the ruby console, do an active record search for all event logs created within the last 4 hours
- Make sure your system clock is set to a time with a UTC offset of -5 or lower
That last instruction – the one about setting your system clock – needs to be first. The entire (onerous) set of steps is rendered pointless if that single step is not done first. Never ever waste people’s time like that. If the person reading your repro steps happens to be from New York City, expect to receive a delivery of dog shit at your house or office.
3. Don’t introduce red herrings
3.1 A “red herring”, in case you didn’t know, is…
“A misleading, or false, clue. It is a common literary device used in mysteries and thrillers that can lead readers down a false path or otherwise distract them from what’s really going on in the plot. In creating a red herring, a writer often includes details added to purposefully mislead readers and lay a false trail. This prevents them from predicting an outcome. ”
3.2. tip: Don’t diagnose as you enumerate
Offering theories about what might be causing a bug – as you’re enumerating the steps to reproduce it – is very distracting. Volunteering your own diagnosis is welcome, just tuck it away in a different section of the ticket, apart from the repro steps. It can be a valuable addition to aiding any subsequent efforts at root cause analysis.
3.3. tip: Don’t inject opinions into repro steps
Color commentary is fine, and adds spice to tickets. I actually enjoy reading highly opinionated tickets. But when it comes to repro steps, don’t do this. You can certainly offer some editorializing in the bug description, but keep repro steps mechanical, robotic, and unopinionated. Sprinkling statements like this is so unhelpful!! doesn’t help an engineer reproduce a bug. It just provides a glimpse into the inner workings of your mind.
Examples of this…
- Open the browser console’s network tab
- Click into the grid’s quick filter text input, and type a single underscore character
- Examine the multiple crud requests that are spawned in the network tab
- Notice the JavaScript errors that repeatedly show up!
- OMG!!! It is 2020, can we please have better static code analysis for this kind of stuff?!!