A developer’s biggest fear

An app or a program which doesn’t work properly is probably the greatest fear of any software developer. To avoid this, there are various ways of testing the software for bugs. Find out how these methods work and what you have to consider by reading on.

Worst case: the deadline for a new application is looming closer and the last lines of code are being written. Finally, the app is ready just on time but there was no time left for testing. At the first start, the app crashes immediately. The next time you get to the log-in screen but suddenly an error occurs which states: “The application doesn’t respond”.

Nearly everyone has experienced something like this on their smartphone or PC. Sure, this is extremely annoying for the user, but it’s even worse for the programmer, since it often means bad ratings, a bad reputation and lots of additional work (on top of potentially dealing with an angry customer, of course). Such things tend to happen a lot in gaming software, when users function as beta testers, but more on that later. Clearly, software testing is a pretty useful thing, so let’s see which different kinds of tests you can use.

Testing methods and terms

Basically, every testing process should be defined before the real coding starts. Otherwise, it’s possible that the testing process is made to fit the final application (which obviously won’t give you very good or useful results). That’s also why such tests are infinitely more valuable when they are performed by someone other than the actual developer. But even if you did not develop the app yourself, but work in close contact with the people who did, you know the functions of the software from the beginning, meaning that you can decide what you want to test at the end. But there are also some testing methods which don’t require any previous knowledge.

Functional Testing

Functional tests usually examine the basic functionality of the software. These don’t have to be performed at the end of the development process, but can be done as early as during the alpha phase of an app. For example, after finishing the log-in function, you can test whether the software recognizes different letters or, if allowed, different symbols (with international apps and programs, umlauts and other unique features are a point to consider, for instance). You can test it manually or with an automatic algorithm. But don’t forget to test the way all the individual software components function as a whole at the end.

Usability Testing

In this case, you don’t watch for functional errors but consider the usability of the application. The testing process looks at how easily people who are entirely new to the app or program can learn its functions and whether these functions are perhaps unnecessarily complicated. The testers are often asked to “think out loud” while using the software, so the observers can reconstruct their actions and reproduce the resulting problems. These tests can be highly useful, because often build functions which seem totally logical to them (due to their technical knowledge and background knowledge of the application), but which are not all that clear to the users. With usability testing, you can avoid mistakes like that ending up in the final software.  There are situations, however, where extensive usability testing is not necessary. For instance, if the software is only used “in-house” or by a very limited user group on the client side, making it as accessible as possible to a large and diverse group of potential users would be overkill.

Monkey Testing

This testing method is the easiest one of the bunch. When testing, you simply execute random events, sometimes in a fast sequence. This is supposed to test whether the application will crash when trying to execute many different or repeatedly the same inputs at once. The name stems from the idea of a monkey randomly pressing buttons. There is, however, a distinction between “intelligent” and “dumb” monkey testing.

Intelligent monkey testing comes close to functional testing, where you set a condition and see if the app responds correctly or according to your expectations. By repeatedly forcing it to execute various commands, you can test whether this causes the app to crash or not. You can even use some of the known issues to your benefit in order to deliberately crash the app (see for example the problems with umlauts).

“Dumb” monkey testing doesn’t require any previous knowledge. You just randomly press buttons and execute commands in order to make the app crash (and you basically get to behave like a monkey, if that’s your thing). A drawback of this is that sometimes you cannot retrace the steps that led to the. You can avoid this, however, by automatically recording the inputs and sending a crash report.

Beta testing

Beta tests are often used in the gaming industry. You give out the application to either a closed group of testers with access codes or even to the whole community for an open beta test. The users may receive full or only restricted access to the software and can test whether it works properly. If not, they can send crash reports or bug reports. When you use a closed group, there is always the risk of someone giving the key away so anybody can see the unfinished software, think it is the final version and leave a bad review.

Today’s gaming industry is often accused of selling supposedly finished products, but effectively using the buyers as beta testers. The user pays for a normal game but gets what is basically an unfinished (because buggy) version. The companies then use the users’ bug reports in order to fix any bugs and errors and continuously keep repairing the game with updates and patches. In this context, the word “bananaware” is often used, because the software only ripens in the hands of the users.

Automated testing

This is not really an independent method but is often used with any of the methods mentioned above. With automated testing, you can program repeated inputs of any command, in order to check whether the application reacts in a normal way or crashes.

Test environment

What you also have to watch for is the testing environment. Sometimes all a program or app needs in order to crash, is a small change in any setting or to any function. I’m not talking about the app’s or program’s own settings here either. A crash can also be caused by other software components on the device. Maybe it happened because Bluetooth was activated or maybe it was a bad internet connection. Even a battery that’s not fully charged, or one that’s currently being charged, could cause a crash. By recording the environment, you can more easily deduce what might have produced the crash.

Recording the data

So, to sum up, in order to fully eradicate an error, you have to know what led to the issue. I always fill out a form where I record the complete environment and the sequence of events that led to the error. It shows which hardware I used, at what time I did the testing and which operating system I used. I also sometimes add screenshots where I feel they are needed for the developers to fully see what’s going on. Ideally, my test records enable them to retrace my steps and find the error cause.

protocol  used for software testing
Above: Picture of a testing form

In the above example, here you can see the that I recorded the test date, the version of the app I used and the time at which I did it. You can also see which smartphone model I used and which functions were enabled or disabled. On the bottom left, you can see the exact steps I took and on the right side you can see the errors which occurred.

All bug-free, then?

Well, you can imagine that testers are not always the developers’ favourite people. But they often find major issues which could affect the final software, so whatever. And we can learn from these problems and maybe find a solution to an error more efficiently next time, where, otherwise, we wouldn’t have known what to do or that it was even an issue. And lastly, it’s not always the programmers’ fault if an application doesn’t run well. You simply cannot test for every possible hardware/software combination or for any combination of functionalities that might theoretically occur. You know, “let he who has bug-free software cast the first stone” … or something like that. 😉