Summary – The Crappy Little Datagenerator Contest

Wow. What a response to this challenge! I am overwhelmed with all the energy people have put into this friendly little contest. Thank you all who have contributed both here in the comments section and you who chose to stay on Twitter to discuss the crappy little datagenerator! I won´t be able to cover all the awesome contributions people have made and I can only do a summary of my own thoughts and feelings. But I’ll gladly continue discussing with anyone who is interested!

If you haven’t tried the challenge you can read about it here: https://www.houseoftest.net/2016/02/testers-contest-crappy-little-datagenerator/. Challenge yourself before reading anymore of this post!

This program was designed as a practice tool for testers by removing most error checks, unoptimizing the code and introducing a bug here and there. Several issues with the program was expected by me to found out by people but what I had not expected was people decompiling my program and uncovering my dirty code that wasn’t intended to be a part of the challenge. I had great fun (and a bit of horror) seeing people debugging the code 😀 This also reveals how deep technical knowledge can have an impact on testing and reveal issues that are not seen on the surface of the product.
I believe all the major bugs I knew of where found. The most crucial bugs are, according to me, the missing international characters, the bad performance of long strings and the generated e-mails that do not conform fully to the specification (or the lack of documentation of what can be expected by the e-mail generator). A lot of people reacted on the last part of an email is not always being generated but it surprised me that people reporting this did not read the specification for what how a valid e-mail address can look like: https://en.wikipedia.org/wiki/Email_address#Domain_part. In the article (I am of course aware that Wikipedia might be wrong but I am assuming the information in this case is correct) the addresses user@[IPv6:2001:db8::1], user@localserver and user@com are described as valid according to the specification. But just because the name itself is according to spec doesn’t mean it is possible to use everywhere. I am assuming that such mails are only used locally on a server or inside a specific network.
Relatively few people that took on this challenge actually asked for more information about the tool. This should be a guiding rule for all testers to first learn about the context of the product; what is it meant to do, who is the intended user and so on. There is often more information available if one is only looking and asking for it. I believe that the testing performed would have been radically different if I would have stated that the goal was to create a practice tool for testers. In that case a bug would’t have been a bug but a feature, and a perfectly working functionality might have been undesirable. In this case though the tool was designed for testers and therefore important for the tester to be able to understand and trust what data is generated. But not all characters where actually available (the international characters) and could fool a user and there was a lack of information about what rules the e-mail generator followed. It might for example be ok if the generated e-mail addresses are not according to spec if the tester knows about this. I also expect that performance might be important for a tester because it might be handy to generate a big string of data quickly.

One thing that surprised me was that I almost ignored reports from some people who had done great testing and very thorough reporting because of the amount of text I received. When I saw a big chunk of text my gut reaction was to skip the reading. This was an important insight about myself; just because the right information is present it might be packaged and presented in such a way that it is in danger of being ignored. For the past week I have been under a high workload and I really wanted to have shorter summarized reports that I could quickly read. I did not realize that I wanted this before I started to get reports and therefore I did not communicate this in the challenge. The lesson I’ve learned here is that people might not know how they want the information packaged until it is too late and as with a lot of things communication and building rapport can make this problem easier to solve.

Another thing I will bring with me from this is how stupid it was to make a contest out of this with an award. There has been so much great testing done in different ways from different people that I cannot say that someone has tested the product better than someone else. But I did however promise an award in the form of a goodie bag from House of Test to a winner. In the end, what I believe where the most helpful test reports were made by Fredrik and Sorina.
Fredrik found the performance issue, the characters missing and decompile the whole program to find the root cause. While not actually asking for it he did show he understood the importance of asking for information about what is important with the program for the creator or stakeholder. Sorina also did excellent testing but what stood out was that she used an oracle to check if the generated e-mail addresses are correct according to spec or not. Together the reports from Fredrik and Sorina told me everything about the product that I needed to know in a short an efficient way making it easy for me to read.
No shadow on the rest of you testers though! There was a lot of great testing being done and I had to make a very subjective decision that might have been different if I was someone else or in a different context. In the end I hope you had some fun with this tool and I will soon update with the sharp version. You found some bugs that had actually slipped by me into the real program so got a little bit of coding left 🙂

Lastly I’m going to read Alan Richardsons lessons learned from testing this app: http://blog.eviltester.com/2016/02/lessons-learned-testing-house-of-test.html. I have been told it is a very good reading.


A Testers Contest – The Crappy Little DataGenerator


At House of Test a couple of us have been developing a simple little tool for generating test data. We call it the “HoTies Little DataGenerator” and we want to share it with the testing community. But before we go public we are releasing the “Crappy Little DataGenerator” as a challenge to you testers. Besides the honor and glory the winner will receive a cool HoT goodie bag! 

The challenge is to test the tool and report your findings in the comments field of this post. I, Martin, will be the almighty judge of what is the most interesting finding or result that will earn a winner the prize. This is a friendly challenge to the testing community but take it seriously and use this as a practice opportunity to train your testing skills. In other words, don´t let your testing be sloppy but do the ground work.

The tool is created in java and as long as you have java installed on your system you should be able to just double click on the .jar-file and start the GUI. You can find the crappy little datagenerator here: https://s3.eu-central-1.amazonaws.com/houseoftest/Exercises/crappy_little_datagenerator_v_1.0.jar
If you like the tool but dislike the bugs, then you will be able to download the real version after the contest is over.
Ladies and Gentlemen, start your testing. You have to the end of the week, the Sunday 28th of February, to send in your test results.


Superhero Personas

, ,

Using personas in your testing can be a powerful tool to try and broaden your perspective and trigger new test ideas.

Thinking and acting like a certain persona can find bugs you normally wouldn’t find since it is easy to get stuck in your own way of thinking and doing things.

There are many different personas you can use but often they are created as a possible end user. Though another way to help you think a little more outside the box is to base them on different superheroes.

Say you have some scenarios you have to do every month to look out for regression, switching it up with superhero personas might make it more fun and help find new problems. Then you can switch e.g. every month to a new theme like Star Trek, Game of Thrones, Lord of the rings, Days of our lives, MacGyver or any other theme you can think of.

Every time you switch theme it forces you to think about what the characters imply and how it could effect your product and hopefully in the process give you new test ideas.

The plan is to trigger thoughts about how they could be used in your context. Maybe the superheroes mean something different to you, maybe they imply other uses of your product or maybe you need a completely different set of superheroes / theme to get your mind triggered.

Below are a couple of superheroes and some quick examples what they might represent.

What test ideas comes to mind when you think about them?

The Flash

  • Do everything as quickly as possible e.g. go through a step by step extremely fast.
  • Focus on response speed and shortcuts.


  • Do many heavy calls (upload/download large files, post big texts, …).
  • Use Big data.
  • Stress test to focus on seeing how much the system can handle.
  • Handle the product with rage, can it handle it?

Mr Fantastic

  • Re-size items (e.g. window size, screen resolutions …).
  • Use different size of strings in input fields.
  • Stretch the product as far as possible.


  • Just as Rouge can absorb the power of anyone she touches you can try and absorb the knowledge and skills of the people around you that you think are good at what they do. Find out what it is that you think they do good and copy it, try and get some pointers or even get a coaching session.

Jessica Jones

  • She is a super strong, depressed, drunk with good detective skills. So how would a smart angry drunk handle your product? Is the usability so good that even a really drunk person could handle it?


  • Jump back and forth between pages (Web swinger).
  • Are passwords saved? How does page cache work (sticky web)?
  • What does your intuition tell you to focus on (spider sense)?


  • Jump in into the middle of a state flow or do things in wrong order (teleport).


  • Well this guy can do almost everything so if you only want to use one, this might be your persona.
  • Go into the code, is there anything obvious wrong, something that is complicated and hence might contain bugs or some comment giving you a hint that the developer is unsure of that part of the code (X-ray vision).
  • What is the one worst thing that can happen to your product, try to trigger this (kryptonite).

Black Widow

  • Black Widow doesn’t possess any superpowers her abilities comes from extensive training. So even if you don’t possess a superpower for something doesn’t mean you can’t be good at it, you just need to practice.


  • Not only a persona but also your role as a tester. You need to take on the job as the world’s best detective to figure out what can be wrong, where to look and so on.
  • Do things in odd hours (nights, weekends).
  • Make use of all your skills and tools you have in your belt to attack your product in many different ways (utility belt).

Nick Fury

  • Don’t be afraid to collect a team of awesome people to help you out with the testing, collecting ideas or to understand a problem.
  • Pair testing.

Green Arrow

  • Identify and hit important/critical areas with high precision.


  • Finding some bugs can sometimes appear as magic when it is the result of good testing. It may be a good risk analysis, an understanding of how customer thinks or by being observant to the details. Although compared to a magician it is usually good to reveal your tricks to other testers to share knowledge.


  • Focus on the hardware.
  • If there are things a computer can help you with, let it.
  • If there is a tool that would help you but doesn’t exist, create it.

Mr. Mxyzptlk

  • What happens if you do everything in reverse? Start at the end.
  • Break the 4th wall (do pair testing with a developer).
  • The 5th dimension is imagination. What would you pull out of there to help you? What if you had no boundaries?

Invisible Woman

  • What happens when your app / program is minimized or runs in the background?

Wolverine / Deadpool

  • How does the system repair itself after a failure or error (regeneration)?
  • How does the product work during a long time (slow aging)?

Ant man

  • How does your product work with small values, zero, null?
  • Swarm your product with many small test scripts (army of ants).
  • How does your product handle dust or small grains of sand?

Wonder Woman

  • Use your lasso of truth on your developers to make them reveal part of the product they are worried/ashamed/oblivious of.


  • Pause/Freeze the system in different states.
  • Slow down the system (slow internet, slow computer).
  • How does your product handle cold weather (battery life, sluggish response).

Scarlet Witch

  • She has the power to alter reality, often so do you in your test environment. Create an reality that shouldn’t happen (because if might happen anyway). What can you learn from this “unrealistic” reality.

Green Lantern

  • No idea, but you can’t skip Green lantern when having a list of superheroes.


  • How does your product handle water?
  • How does your product communicate with other products (telepathy)?

Ms Marvel

  • She has a limited precognitive sixth sense, so do you. Use your sixth sense to try and feel which areas are high risk areas and focus you testing there.

Captain America

  • How does your product handle different time zones, currency and units (maybe far fetched but he is from America and I’m Swedish).
  • Are you protected from attacks (shield)?
  • What does your product do better than all competitors, how will it achieve world domination? (well you know, USA, USA, USA …).


  • How does your product handle different time zones, currency and units (maybe far fetched but he is from America and I’m Swedish).
  • Are you protected from attacks (shield)?
  • What does your product do better than all competitors, how will it achieve world domination? (well you know, USA, USA, USA …).


  • Don’t only use your eyes, what does your other senses tell you? Is there a strange sound, do you smell something, does the product get hot and so on.
  • How would you defend your product in court (lawyer)?

And don’t forget the super villains. Sometimes you need to treat your product really harsh.

Want to know more?

For more about about superheroes see this great video about “10 lessons entrepreneurs can learn from superheroes”.

All shown superheroes are owned by http://www.dccomics.com/ and http://marvel.com/


Basic security testing of website input fields – Look for lack of input limitations

, ,

If a website has an input field that lacks any kind of limitations on the input then you don’t need to know how a hacker might use this to gain access to the system, you only need to know that a hacker can.

Years ago I was testing a web application and found that for a particular input-field, used to enter a name of an object, I could send in at least a gigabyte of data. I knew that this could potentially be a problem and shared my concerns with the developer sitting next to me. He lighted up and suggested me to send in a simple popup-script to see if the product was possibly vulnerable to script or XSS-attacks but when I tried it in the input field nothing happened. But when I tried it in the input field below the first one I got a popup showing that the system was wide open for attack. This incident taught me that I don´t need to know exactly how hackers can gain access to a system to learn how to spot the security holes.

The lesson was reinforced when I was performing a lecture on security testing. I shared a simple XSS script with my students together with the experience I mentioned above. One of my students started to google “Order here” and went to the first web shops on the list. and entered the script into the first search field he could find. Ten minutes after having been presented with the script he had found out that one of the biggest sites for purchasing academic literature in Sweden was vulnerable for at least Reflective XSS attacks but the possibility for more serious Stored XSS or other types of script attacks should be investigated. I tested the search field some more; Did they have any limitation on the length of the book name to search for? Well, at least no limitations up to a million characters (I simply stopped testing longer strings at that point). Conclusion: The site that did not have any limitation on the input was also the one which had an obvious security risk and potentially open to XSS attacks.

I reported this to the site by sending the information to their customer support (they lacked any other way of getting in touch with them). After a couple of months, the site was patched and I could no longer execute that script. Today, however, over a year after I had reported the security hole, I tried another variant of a script intended to reveal XSS vulnerabilities and managed to get a pop-up alerting me that the site was still open for attacks. Even though there are lists of these scripts that one can simply follow and copy paste into an input field, far from every site bothers to tests this and thus expose themselves for attacks. This taught me that if an input field lacks basic limitations on input then I should look deeper for more issues that can affect security.

So what basic security tests can you perform on a Web site input field?

1)   Test how many characters you can enter and check if there is a reasonable limitation. For example, if you have an input field where you can enter your first name you probably do not need more than 16 characters (for a lot of other types of input fields that a number of chars can, of course, be too limiting). If you are allowed to enter a million you should raise concerns. A very useful tool to check the possible number of characters that can be entered is using a counterstring (you can read more about it here: http://www.satisfice.com/blog/archives/22). In a counterstring you can see exactly the position of any given character has inside the string.

Lacking input restriction might be a sign of other security checks missing but can also be specifically used by hackers. An intruder might try to send in great amounts of data and hope that the system crashes and either grant administrator/root access or reveals unintended information. If the system is not properly setup it might reveal information such as server or database names or ip:s that the hacker can connect directly to.

By restricting the number characters it also limits the type of scripts that an attacker can use.

Couterstring in a search field
Using a counterstring I can see that I can enter 524288 characters for doing a simple search which is the default length for a for html input tag. Performing the search will result in an error that input is too long which is a little bit curious.

2)   Test if there is a limitation of possible characters. Again, if you have an input field for entering a name you probably don’t need strange characters such as “<>/\;,!”. These characters are used when sending scripts into the input field and if they are blocked it will make it more difficult for an attacker.

Possible to enter special characters
These characters will probably not be used by an actual user but might be used to send in scripts.


3)   Check if XSS attack scripts are possible to execute. Check OWASP’s excellent collection of scripts: https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet. Simply copy the scripts into your input field and if they result in a popup, you have a problem. You don’t necessarily need to understand exactly how the scripts can be used to breach the system, only that a hacker can potentially use that hole. You can read more about XSS at https://www.owasp.org/index.php/Cross-site_Scripting_(XSS)

Popup when entering a script into the search field
This pop up came after entering a script into a search field thus showing the site is at least vulnerable to Reflective XSS attacks. In this particular case I am actually not sending in the text “XSS” but I’m sending in code that translates numbers to ASCII text. Several other types of scripts are blocked but obviously there are holes.

4)   Make sure the checks on input fields are not only located on the client side but also on the server side. If the checks are done locally on the attacker’s’ computer before sending the message to the server, the hacker can bypass those checks by sending a direct and unfiltered message to the server. An excellent tool to make this kind of test for web pages is Fiddler. Using it you can manipulate the HTTP message sent to the server and then check on the server if the message was filtered or rejected.

5)   Using Fiddler, I would also recommend making sure that encryption is always enabled. You don’t need to understand how a hacker might sniff traffic and use the data sent, you only need to understand that a determined hacker might use it for something evil.

6)   If the site uses any type of SQL-database you might want to read up on SQL-injection and how to prevent them: https://www.owasp.org/index.php/SQL_Injection_Prevention_Cheat_Sheet.

7) Check that if you are sending in data that the communication is encrypted. You don´t need to know exactly how an attacker can use personal information that is picked up in an unsecured network, you just need to know that it can be used against a person. An easy check to do is to simply look at the address bar, making sure the page is using HTTPS. Using HTTPS does not provide complete security of course but it is one piece of the puzzle making it more difficult for an attacker.

Https enabled
This particular page is encrypted using HTTPS thus making it harder for an attacker to listen in on.

8)   Make sure that the site administrator can be directly contacted so he or she can act swiftly if someone spots a security issue. If there is no way to contact anyone responsible for the site then a friendly person who spots something potentially serious might just give up and let it be, leaving the security hole open.


The steps above can be done by any tester, even with limited experience, and can prevent several basic attacks. But the steps will not guarantee good enough security and there is a lot more things to test. What I have described should be considered the bare minimum of what you should test and if your system is critical in any way you might need a security expert to analyse the system. For a more complete picture of security risks and how to mitigate them, I strongly recommend reading more at https://www.owasp.org.

To investigate and look for lack of limitations is of course not limited to input fields for websites. You have similar issues with desktop apps and generally everywhere where you can add something into a program. An attacker can exploit limitless inputs to create memory or storage problems to gain access to the system. Inputs in desktop or mobile apps might need slightly different testing but are not necessarily less important from a security perspective.

And again remember that you don’t necessarily need to understand exactly how an attacker can breach a system if you understand that lack of limitations on inputs can be a way in.

Update 2016-02-02 – I have updated with the correct naming of the Reflective XSS vulnerability I made an example of. Thanks for pointing it out Jari!


What on earth can just 2 days of testing do for a project?

, , , ,

The short answer: A lot.

In December 2015 I was called to the offices of a project, who went live with their website the month before. On the website, customers could search for and buy products that could be sent to other people via both physical and virtual shipping channels. The website had been in development for a year and then some, and they needed the website to be online for Christmas, but had a feeling that there were issues in some critical areas. The project did not have any formal test resources attached when I met them but relied on issues found by the developers and designers. And in short, they wanted me to test the “user experience” of the website on different platforms and devices.

Testing user experience

“How do you test user experience?”, I hear you ask. That’s a good question, and there’s really no easy answer to it. How you test user experience relies on your tester, your project, and what you want to get out of the test sessions. In this specific project, it was important that the visitors had the same experience with the website, no matter whether they visited it through different browsers or on different mobile devices. So obviously, I needed to look into if there were some unwanted differences across different platforms.

Furthermore, since I didn’t have a lot of time, I needed to use my own skills. Luckily I have quite a lot of experience with usability design, so I wanted to look into the usability of the site, as it in this project was closely related to the “user experience”. There are many ways to approach usability, but I like working from a simple recipe that involves the following 5 keywords:

  1. Learnability (How easy is it for users to accomplish basic tasks the first time they encounter the design?)
  2. Efficiency (Once users have learned the design, how quickly can they perform tasks?)
  3. Memorability (When users return to the design after a period of not using it, how easily can they reestablish proficiency?)
  4. Errors (How many errors do users make, how severe are these errors, and how easily can they recover from the errors?)
  5. Satisfaction (How pleasant is it to use the design?)

When you can answer these questions, you have a pretty good idea about how a visitor experience browsing the website.

The plan

In order to do as much testing as possible in only 2 days, I knew I had to follow a strict plan.

2015-12-01 10.43.41

A plan comes together. And lunch.

Test cases were out of the questions. I didn’t have time to write lengthy guidelines, I didn’t know the system, so how would I be able to write guidelines about looking in specific areas for specific issues. Furthermore, the company put their faith in my skills, so I didn’t have to prove what I did down to the last detail. They could read about what I did and didn’t do in the test plan and test report. So in short, test cases would be a waste of time. Instead, I went with testing any page and function I could find, trusting my skills and intuition. I traced my test steps in a mind map tool, that would later do double-duty as an easy way to go through the found issues with the project group.

After some thinking and planning, I ended up with a test plan. It’s quite possibly the shortest test plan I’ve ever seen. It’s actually just an image. And yeah, you can make test plans that the average mortal can read.


Do testing that fits the situation and you

“Can you do all these tests in 2 days?!” you might think to yourself. Yes and no, I say! With just 2 days, it’s obvious that you can’t get to the bottom of every aspect of the method. I did, however, go through with every method in the plan.

On day 1 I contacted a lot of people through my preferred social media and asked for their help in the First-Click-test. After setting up and starting the First-click test in Google Analytics, I left my home to ride around in Copenhagen for the in-the-wild testing on some of the mobile devices chosen for the test. During this trip, I visited two focus group test participants, who went through the website with me and pointed out issues that I never would have seen. After that, I returned home and tested on every browser that I could get my hands on. End of Day 1

On day 2 I did an expert review of the website, had lunch, and then went out to visit the last focus group test participants. After returning home I went through the remaining mobile devices, and after dinner, I visited a friend for some help with getting to the bottom of some technical issues with some mobile devices. End of day 2.

2015-12-02 11.22.45

Ready to ride the trains of Copenhagen for in-the-wild testing

I admit I had the best circumstances for doing these exact tests (And that’s why I chose them). I was not a part of the project team and did not have any prejudiced ideas about how the system I tested looked and functioned. That made it possible for me to do an expert review of the website. Had I been on the project from the beginning, I probably would have missed a lot of the issues I found in the expert review. Had I been a part of the project, I could not have used my friends and family as test participants, because I would’ve been invested in the system and could potentially have influenced the test participants while testing.  If I had to be situated in an office, I couldn’t have involved the focus group test participants, since most of them were busy students studying for their exams. If I hadn’t done a LOT of focus group interviews back in the days, I wouldn’t have been able to do them as quick as I did. And so on.

And that’s why I chose these methods because they suited the project and my test situation. I could’ve planned this in many ways, but ultimately chose what I knew would work.

The result and the moral

Testing is not about finding as many defects as possible. The amount of defects found in a system says nothing about the quality of the system, as the severity of the defects is wildly ambiguous to different people and different parts of the project. I will, however, reveal that in 2 days, I registered 67 unique issues. These ranged from very critical ones such as “Category menu does not show when the menu button is clicked“, to the maybe not-so-critical ones like “In the box where I choose the date, the symbol for the date looks like something that can be clicked, but it can’t“, and all the way to the more suggestive ones like “I would like a “cancel” button on the video upload“. But what’s more important than the number of issues found: The customer was very happy with getting another angle on the website and its current state. Not the angle of a developer or a designer, but from a professional tester and potential future customers. And as it turned out, there certainly was some critical issues in some areas that ought to work seamlessly.

So in short; Sometimes you don’t need a lot of planning and test documents, sometimes an image is all it takes to get an overview of the test process. Testing doesn’t have to be expensive, tedious, or repetitive. Sometimes you shouldn’t overthink testing, just jump into it and trust your tester. Professional testers will probably find issues that the project team does not notice in areas others would never even look in.

Sida 10 av 16« Första...89101112...Sista »