Many years ago I was performing a QA test suite on some software. Everything seemed to be going swimmingly well until I started to test “outside the box.” I had completed the standard test fare and had begun what I like to call “adversarial testing.” More accurately it could be described as “what if” testing.
Following this process I found a bug that seemed pretty severe to me. If a user entered a valid username but an invalid password the application would crash and burn. The same held true for valid password and invalid username. But when you entered gibberish in both fields you received a lovely pop up stating that your username and/or password were incorrect and to try again. Huh? What’s that all about?
I reported this as a Severity 1 bug (since it stops a user from using the application, at least temporarily) and went about my business trying to break something else. It wasn’t until our weekly bug review that the humor really started. During our review call the engineering team said they didn’t accept my bug as a valid test case and therefore were closing it as invalid. I protested vehemently. “You can’t close that bug, you need to fix it!” Since this was my first time to this rodeo, I was a bit shocked by the reply.
The lead engineer simply said, “That’s stupid. Just tell the customer not to do that.” He went one step further asking, “Why would the customer want to enter the right username and wrong password anyway?” Had he never heard of a typo? What about fat-finger syndrome on a keyboard? Was he kidding?
I quickly learned he was not kidding and had no clue where I was coming from. To be completely fair, I had no clue where he was coming from either, but assumed he was ignorant of what happens to his work when it hits the real world. He wasn’t moved by impassioned speech about customer experience and our reputation, but fortunately our engineering VP was and ordered them to fix it (prior to delivery).
What’s the point?
When it comes to your mobile project you need to use your User Acceptance Testing (UAT) process as your opportunity to mitigate your risk once your solution hits your full user community. Start by testing the proper way to use the application. Ensure it does exactly what it needs to do (and you should know these requirements from your detailed requirements sessions that I have pounded on you about in previous blog posts). Once you have validated that the solution works as promised start the adversarial testing. Test like a user. Find out what happens when somebody enters a typo. If the user is impatient and tries to hit a button 27 times because it doesn’t react fast enough does the app crash? What happens if you remove the device battery in the middle of your test suite? Does the application recover elegantly or is it an ugly scene? Use your imagination. Make mistakes you would never make but real users might, and see what happens.
All of these scenarios will occur with frightening frequency in the field and knowing what happens BEFORE you send the app out to the masses will allow you to either correct the bug and prevent it from happening or prepare your support team for the troubleshooting task. And, remember, “Just don’t do that” is not an acceptable response to any user no matter how silly the issue may appear on the surface.
Tags: Brian Philbin, Business Process, Mobile Apps, Mobile Solution Design, Mobility - General
