Doing, Not Testing
When it comes to manual testing of an application, we tend to differentiate between functional testing and workflow testing. Functional…
When it comes to manual testing of an application, we tend to differentiate between functional testing and workflow testing. Functional testing in this context can be thought of as pressing all of the buttons and making sure that the expected behavior occurs, essentially basic smoke tests and trivial bug hunting. In contrast, workflow testing means that a person walks through a set of real, scripted workflows using the application, for example, creating a new case record and updating it with notional data. Both procedures are good, but they fail to capture important signal needed by a development team that comes from using an application as a user would, and I don’t mean in a simulated fashion.
When a product manager, quality engineer, or test engineer kicks the tires, there is no substitute for regularly testing by doing. I want to be clear about the subtle distinction here. Testing by doing means the tester is using the product for its real benefit or to accomplish it’s intended function, not for the sake of finding bugs. It must be personal for the tester and he/she needs to do work that matters to him/her. Finding bugs is a byproduct of this process, and even more importantly we learn about the most important features that are missing or that detract from the user’s experience.
Let’s use a concrete example. Suppose a team is building a new map product and one of the new features is the ability to associate a home and work location on the map and notify the user of promotions that are located in between them. A traditional approach is to do exactly that, find two random locations, click the button that connects them, and move on if nothing breaks. When we test by doing, we manufacture our own personal, real-life workflows and then perform them regularly. In this example, we want to choose a real home and work location, associate those locations, and then monitor the activity between them.
This methodology will reveal numerous, subtle distinctions that over time add up to the prevention of many bugs and the introduction of a smoother experience and new feature ideas. For example, if we choose the two locations arbitrarily, we may not realize that when choosing a home location inside an apartment building, the user may need to specify the floor or the location within a large building, and our application may not have that option from the get-go. Furthermore, the algorithm for promotion searching may draw a straight line between home and work and in practice, geography or natural highway contours could render this strategy substantially lower signal. Finally, real work is good, but real, consistent work is even better. If we check once or twice with a functional test and we receive promotional offers, we may conclude that everything works. Checking regularly as if we are a user of the product and looking for deals that matter will prompt us to notice things like relevance, density of offers, or other critical factors that would make the difference between a product that flops and one that sings.
Originally published at http://adamjudelson.com on September 8, 2016.