Back in April, we published a blog post on bug bashes - events where people with different backgrounds (both tech and non-tech) get together to test an app and discover the many bugs that have eluded the project’s Quality Assurance (QA). Now, with around 100 completed bug bashes under belt here at netguru, we have some tips that will hopefully help your bug-bashers come back from the hunt with bugs aplenty.
Bug bashes should represent a diverse crowd of users, so pretty much anybody will be a good addition to your team. Try to get your group as big and varied as possible! However, there are some kinds of people who really shine during bug bashes, or who can really benefit from joining them:
The idea is that you want to have as many guys and gals as possible during a bug bash since everyone adds their own approach and strange new perspectives, increasing your chances of discovering bugs. The problem is that people have their own jobs, and if you're working in small teams it would be unfair to ask developers from other projects to stop their work help you bash “your” bugs.
Netguru has developed the following flow:
Bug bashes are fun to organize, and beneficial for everyone – but some times are better for holding them than others. The best time for a bug bash is:
On a day-to-day basis, you want to stay rational at work. When testing, you focus on likely user paths. You choose high coverage over depth. You might not have time to delve deep into things that look strange, but innocuous enough. I tend to note them down, though – notes can come in handy during a bug bash, which is a regular part of our testing flow.
During a bug bash - you do the opposite. Keep your eyes peeled for anything looking off. When you find something suspicious, strike against it with all your might! Exploit every weakness you see. Double-check the strange things you previously took note of. Try to hack into the app, even if you know almost nothing about hacking. Do stupid things: copy sources of whole websites into post titles; give yourself admin powers, then ban yourself; nothing is too ridiculous!
This is just my approach - there’s nothing preventing you from acting rationally. But since the goal of the bug bash is to try out as many user perspectives as possible, I believe that a small degree of reckless bravado is appropriate.
Most bug bashes don’t take long - ours typically last 1-2 hours. To ensure that as much time as possible is spent on the actual bug bashing, you can try the following:
With many people using the app at once, bug bashes are a great opportunity for testing as many ways of interacting with the app as possible. You can assign a specific browser, device, account type, etc. to each bug-basher before the event starts and write what’s assigned to whom in the bug bash docs. It dramatically increases coverage, and makes it easier to debug and record the bugs later on.
For added effect, you can test more unusual, but still used, equipment – maybe a smartphone model with a unique screen resolution to mess up mobile apps and websites? Or else, you can try old versions of Internet Explorer (possibly with the help of Browserstack) – notorious for their poor handling of, well, everything.
Always keep in mind how the target users tend to interact with the app. If it’s a mobile project, it should have compatible devices listed. If it’s a website, what browsers are users most likely to choose? Are they high-tech specialists with top-tier equipment at hand, or do they have little knowledge of IT and/or depend on outdated devices? Likely, the owner of the app has all the info you need about the target users’ software and hardware. And even without this info, sites like Wikipedia and w3schools.com are a good start.
If nobody wants to sign up for a bug bash, and your desperate pleas for help are answered with tumbleweed GIFs – offer some cookies for the bashers. It works!
What are your techniques for running a bug bash? Share them with us in the comments below!