Wow. What a response to this challenge! I am overwhelmed with all the energy people have put into this friendly little contest. Thank you all who have contributed both here in the comments section and you who chose to stay on Twitter to discuss the crappy little datagenerator! I won´t be able to cover all the awesome contributions people have made and I can only do a summary of my own thoughts and feelings. But I’ll gladly continue discussing with anyone who is interested!
If you haven’t tried the challenge you can read about it here: https://www.houseoftest.net/2016/02/testers-contest-crappy-little-datagenerator/. Challenge yourself before reading anymore of this post!
This program was designed as a practice tool for testers by removing most error checks, unoptimizing the code and introducing a bug here and there. Several issues with the program was expected by me to found out by people but what I had not expected was people decompiling my program and uncovering my dirty code that wasn’t intended to be a part of the challenge. I had great fun (and a bit of horror) seeing people debugging the code 😀 This also reveals how deep technical knowledge can have an impact on testing and reveal issues that are not seen on the surface of the product.
I believe all the major bugs I knew of where found. The most crucial bugs are, according to me, the missing international characters, the bad performance of long strings and the generated e-mails that do not conform fully to the specification (or the lack of documentation of what can be expected by the e-mail generator). A lot of people reacted on the last part of an email is not always being generated but it surprised me that people reporting this did not read the specification for what how a valid e-mail address can look like: https://en.wikipedia.org/wiki/Email_address#Domain_part. In the article (I am of course aware that Wikipedia might be wrong but I am assuming the information in this case is correct) the addresses user@[IPv6:2001:db8::1], user@localserver and user@com are described as valid according to the specification. But just because the name itself is according to spec doesn’t mean it is possible to use everywhere. I am assuming that such mails are only used locally on a server or inside a specific network.
Relatively few people that took on this challenge actually asked for more information about the tool. This should be a guiding rule for all testers to first learn about the context of the product; what is it meant to do, who is the intended user and so on. There is often more information available if one is only looking and asking for it. I believe that the testing performed would have been radically different if I would have stated that the goal was to create a practice tool for testers. In that case a bug would’t have been a bug but a feature, and a perfectly working functionality might have been undesirable. In this case though the tool was designed for testers and therefore important for the tester to be able to understand and trust what data is generated. But not all characters where actually available (the international characters) and could fool a user and there was a lack of information about what rules the e-mail generator followed. It might for example be ok if the generated e-mail addresses are not according to spec if the tester knows about this. I also expect that performance might be important for a tester because it might be handy to generate a big string of data quickly.
One thing that surprised me was that I almost ignored reports from some people who had done great testing and very thorough reporting because of the amount of text I received. When I saw a big chunk of text my gut reaction was to skip the reading. This was an important insight about myself; just because the right information is present it might be packaged and presented in such a way that it is in danger of being ignored. For the past week I have been under a high workload and I really wanted to have shorter summarized reports that I could quickly read. I did not realize that I wanted this before I started to get reports and therefore I did not communicate this in the challenge. The lesson I’ve learned here is that people might not know how they want the information packaged until it is too late and as with a lot of things communication and building rapport can make this problem easier to solve.
Another thing I will bring with me from this is how stupid it was to make a contest out of this with an award. There has been so much great testing done in different ways from different people that I cannot say that someone has tested the product better than someone else. But I did however promise an award in the form of a goodie bag from House of Test to a winner. In the end, what I believe where the most helpful test reports were made by Fredrik and Sorina.
Fredrik found the performance issue, the characters missing and decompile the whole program to find the root cause. While not actually asking for it he did show he understood the importance of asking for information about what is important with the program for the creator or stakeholder. Sorina also did excellent testing but what stood out was that she used an oracle to check if the generated e-mail addresses are correct according to spec or not. Together the reports from Fredrik and Sorina told me everything about the product that I needed to know in a short an efficient way making it easy for me to read.
No shadow on the rest of you testers though! There was a lot of great testing being done and I had to make a very subjective decision that might have been different if I was someone else or in a different context. In the end I hope you had some fun with this tool and I will soon update with the sharp version. You found some bugs that had actually slipped by me into the real program so got a little bit of coding left 🙂
Lastly I’m going to read Alan Richardsons lessons learned from testing this app: http://blog.eviltester.com/2016/02/lessons-learned-testing-house-of-test.html. I have been told it is a very good reading.