Stencyling Around

Gotcha with the title. But honestly, bugs are good in many ways that aren’t obvious at first glance. This article will talk about how we approach bugs at Stencyl and how this part of experience has panned out for us.

Bugs != Buggy Product

The first lesson about bugs is that for a decent project, there’s not a whole lot of correlation between bug count and how buggy that piece of software is. A high bug count is indicative of how much feedback that software has gotten. Feedback is good because it opens a conversation between you and the bug reporters, whether they are your users or technical peers. Fixing bugs successfully and collaborating with these people will help build mutual respect, and that’s something that’s incredibly difficult to build without a proper process in place.

How I Approach Bugs

I approach bugs as if they were puzzles. Within 10 seconds, I bucket them into one of three categories.

1. Can fix in 5 minutes.
2. Can fix, but it’ll take some time.
3. Don’t know what the heck is going on or need more information.

All of the (1) bugs are immediately dispensed of on the spot, regardless of priority. My simple reasoning is this - I don’t like trivial bugs piling up, and an excessive amount of trivial bugs can mask the more important ones and just make the whole queue look a lot bigger than it is.

The bugs in group (2) are handled on a case to case basis. If I can fix them rather quickly, I might tackle them at the end of the night or in the morning in my spare time. Otherwise, I prioritize, group and schedule them.

Bugs in (3) are the most fun and grueling to fix. Sometimes, I’ll need more information from the reporting user. Sometimes, I just have to set it aside and let the ideas flow in in the shower or car. It really isn’t something you can to brute force - it’ll come to you in time.

Discovering the Root Cause

After accepting a bug and even while I bucket my bugs, I begin formulating a hypotheses about the root cause(s) of the bug. Sometimes, it comes to me right away, and I know exactly what part of the code to look in and what is causing it, a non-trivial task now that the codebase is well above 100,000 lines and 600 files of source code.

If the cause is non-obvious, unlike some others, I don’t reach for my debugger or print statements right away (and to be honest, I rarely use a debugger). Nor do I look at the output or stack traces. I start at the fundamental level and figure out what IS possibly going wrong. This means that I have to formulate a mental model of the process, play out some scenarios in my head and zone in on the code that’s responsible. This way, I don’t rely on tools as a crutch. I let my mind exercise itself and by this method, know my code that much better than if I had jumped right away for the tools. More often than not, I am able to solve a problem this way.

If I still cannot grasp the problem, then I start bringing in the big guns. If it’s a crash, a stack trace is usually good enough. If it’s a logical flaw, I drill down on the problem, using tools to assist. I employ Toyota’s 5 Why’s Method to discover the cause, again to take an analytical approach to the problem.

All in all, bug fixing should be viewed as a critical thinking exercise where you reinforce your knowledge of the code and occasionally learn something new. It should not be viewed as grunt work, and if you diligently approach this in a constructive way, you’ll find yourself fixing bugs quicker, introducing fewer bugs and understanding your code much more. That’s how I was able to pull off a 90 -fixed -bugs boup in 3 weeks on top of several features I was developing.

We have a QA Process

Back in March, I proposed starting a QA (quality assurance) program and appointed Fox, one of our newer testers at the time, to the task, which was to proactively file bugs and verify my fixes for bugs. At the time, it might have sounded like I’d jumped the gun, but in retrospect, it was one of the best decisions I could have made for Stencyl.

For one thing, having a formal QA process in place allowed such a program to organically develop and refine itself over time. It’s better to start things earlier rather than force them in later.

Another benefit is that it developed an engineering-centric subculture in which our more technically oriented testers were bringing their own knowledge into the fray and engaging in meaningful discussions whether it involved asking be questions about the system or proposing their own solutions to problems. Those kinds of discussions reach far beyond the bug fixing realm and contribute to the larger success of the venture

Getting back to the original process, we started with a forum-based approach and had 3 forums: Bugs, Fixed Bugs and Closed Bugs. Like a Bugzilla-ish system, people would file a bug in Bugs.  I would evaluate and fix the bug and move it to Fixed Bugs. The QA contact would then verify the bug using a battery of tests that we joint developed and would finally move it to closed if fixed or back to Bugs if it needed further work. It was a simple model, but it worked for many months until we started expanding.

Today, we have more categories, an additional QA Contact (Greg) and many more issues to track. Since March, 700 issues have been filed and the volume (both fixing and reporting new issues) has picked up substantially. All in all, I am elated with the current status of our QA program, and it’ll only refine further as the software matures and the website part of our venture comes into play.

Conclusion

This QA process stemmed from my industry experience, and it’s really something you can’t come up with on your own if you work on a project with other developers who have yet to work in a real company. While working in a bigger company has its ups and its downs, the power of process, when applied properly, is a real boon, and this is one example of that.

While users will be happy to know that we place a strong emphasis on quality, the real win here is in the intangible stuff. It’s the valuable feedback, the discussions we’ve held and the general environment that count for a lot, and in a way, this culture is just as pivotal to our success as the technology itself. I’ve realized that there’s a whole lot more to software than just developing it - it’s the singular, unified vision you have, the decisions the team makes and all that expert, domain-specific knowledge that count.

- Jon