Let’s Test 2013 Sum-up

First off I want to thank everyone who attended Let’s Test and made it an awesome experience. This was my first test conference and all of you helped to make it a fantastic one.

There were so many great talks to attend that choosing between them was an impossible task, or at least a very hard one.

One thing all presenters had in common was that the information they were sharing was mixed with a great sense of humor. This made time fly and all the sessions felt way too short, even the full day with James Bach. As a colleague said when we attended a presentation James had in Malmö. “It is like a test related stand up”.

Below are my sum-up of the sessions I attended. Have a look at http://lets-test.com for the full program and the slides (coming soon). And I apologize if any quotes are wrong.

James Bach opened the conference with his keynote “How do I know I’m Context driven?”. He talked about the problem of shallow agreement where we might think we are talking about the same thing but really we are not. He also pointed out that context driven can mean different things such as.

  • A paradigm – The model by which I understand, experience value, explain and categorize good testing.
  • A community – People I influence and am influenced by who expose the principles of context driven testing.
  • An approach – A heuristic I apply to my projects.

In depth look at the art of Test Reporting – James Bach

I was one of the early registers so I got a seat at James full day tutorial. The day was composed of presentation and exercises where one exercise was to answer the unicorn question “‘Between 1-100, how completely did you test?”. Amazing how many answers you can get to that question. Here are some:

  • 1
  • 80
  • Silence
  • 100, I tested all 4 minutes
  • 0, I know 0 about how much I have covered.

He also talked about the 3 levels of test reporting:

  • 1. What is the status of the product? – Is the product any good?
  • 2. What did you do to get to that fact? – How do you know?
  • 3. Was that testing good? – Should I be pleased with your work?
  • 3+. What is the quality of the test report? – Is the report good enough?

These levels are good to keep in mind since it will help you create a thorough report fast.

There were a lot of topics covered such as tacit vs. explicit, the beer game, the low tech dashboard and safety language among other things. One topic was ways to increase your credibility as a tester by:

  • How you dress
  • Using correct language / words
  • Using the right tone
  • Growing a beard

Second day started off with Johanna Rothmans keynote “Kick-Ass Manager”.

I’ll give you some of my favorite quotes.

  • “Hire people who are smarter than you”
  • “Multitasking has never worked. It is the fastest way to get nothing done”
  • “Adapt / evolve or die”
  • “Bugs are not the interesting point to business, the effect of the bugs are.”

The leading edge of testing mobile apps – Julian Harty

I was one of the lucky few who got the “Mobile developers guide to the galaxy” book (which you can get for free here http://www.enough.de/products/mobile-developers-guide/).

One of the topics where that feedback is public and very important for the app. If you get bad feedback (and it can be enough with a couple of one star ratings among a bunch of five star) it is often easier to scrap and re-do. Another thing is that it takes roughly 2 weeks to get your app approved by Apple and around 6 weeks to get an update (fix) in.

Linguistics – On how to keep the dialogue Constructive – Leo Hepis

For this workshop we got two brave volunteers (Erik Brickarp and Martin Nilsson) to act out a phone screening for a job interview. Observers where to watch out for uncooperative answers e.g. “Will I get a build on Friday? – Our integrator is on vacation”. The workshop focused on the Meaning part of Satir interaction model (Intake, Meaning, Significance and Response). The workshop got me thinking on how I usually respond and will make me more careful in my responses from now on.

The evil testers Guide to Technical Web Testing – Allan Richardson

“Technical testing is a reminder to keep going deeper”. You can use MORIM (Model through Observation, Reflection, Interrogation and Manipulation) as a reminder to keep going deeper.

Allan uses a version of Chomsky’s Transformational Grammar to model his thinking in multiple surface structures to form one deep structure. Every new surface structure will either strengthen or re-form the deep structure.

And then a quote about how he work:

“Developers work in the backend so I work from the frontend and work myself back”.

Observation Ninjas & Description Superheroes – Ilari Henrik Aegerter

It’s amazing how easy it is to fool our brain. For this Ilari used pictures, mind-reader exercises and card tricks to show us. I especially liked the mind-reader one which I have amazed all my friends with (you can see it in the presentation http://vimeo.com/66929939 @29:34). Hopefully this will make me more aware of the things that might fool me and hence avoid being fooled.

The Art of Understanding – Carsten Feilberg

This was a workshop where we were divided in 3 teams and all the teams got a number of words that could have a lot of different meaning such as Project, Developer, Tester, Manager, Data, Transport, Delivery, Shop, Shop owner and so on. Our task was to arrange these words in relation to each other. The thing I found interesting here was that even though we didn’t have enough information to make any real meaning of this we still struggled to find meaning. Another interesting thing was that even though we didn’t know our solution was the right one and we only had worked with it for 10 minutes, most of us still defended our own solution when comparing to the other groups and wouldn’t change anything.

Bad Metrics and What You Can Do About It – Paul Holland

This is a really interesting topic and Paul attacked it with humor and passion. A lot of the bad metrics he mentioned I have unfortunately seen myself. Among others the salary based “you need to find 3 bugs a week”.

Goodhart’s Law which says “When a measure becomes a target, it ceases to be a good measure” is something we all should be aware of.

Paul also listed some elements of Bad Metrics:

  • Measure and / or compare elements that are inconsistent in size or composition
  • Create competition between individuals and / or teams.
  • Easy to “game” or circumvent the desired intention
  • Containing misleading information or gives a false sense of completeness.

A number of bugs found really don’t say anything. There is no way of knowing the status of a product by just a bug number. What kind of bugs is it, how easy can we fix them, are they critical and so on.

We know more than we can tell – Michael Bolton

There are a lot of things we know but we don’t know why and how. We learn all the time by imitations and trial and error.

Michael talked among other things about the 3 flavors of tacit knowledge:

  • Relational – (Head) “All possible chess games”.
  • Somatic – (Body) “Can we explain how to keep your balance when riding bike explicit?”
  • Collective – (Society) “Biking in Amsterdam, follow the “flow” of the traffic”

A good quote from Michael when talking about if a tester needs to be able to program:

“Either be a programmer or be charming”.


There were a lot of good sessions which have triggered a lot of thoughts.

But Let’s Test is not only about the sessions. The interaction during lunch, between the sessions and during the evenings are equally important if not more so. Meeting and talking to all of the participants is what really made an impression. Add good food and free beer to the equation and you have a recipe for success. So all in all this was a really good conference and I’m definitely coming back next year.

See you all next year at Let’s Test 2014, if not sooner.


A quick introduction to XSS

I recently had an interesting experience that gave me an introduction to cross site scripting (XSS). I have not dealt with this kind of security testing before but I have learned that there are some really fun things than can be done, and those things can be used to do really bad things.

It started when I was discussing with a colleague about a couple of text fields on the web-page I was testing. What I had found was that the text fields accepted any text given even though you were only supposed to enter a number from 0 to 100. Overhearing our conversation my desk neighbour and developer suggested we could try to enter a script into the text box and see what happens. On the first attempt nothing more happened than the script being accepted as text. But when trying the same script on the next field the script was suddenly executed and I got a popup with a friendly “Hello!”. The text field was supposed to accept a name entered as a text-string but took my small script and executed it. The script was not only executed once, it was executed every time any user entered the same area on the web page.
A friendly hello might not be so dangerous but if it instead said “Your Outlook password has expired, please enter your old password and enter a new.” then the users might be exposed to a severe risk. Even though my knowledge in security testing has been limited to this point, copy-pasting a small script was enough to uncover a potentially severe issue.
This experience also showed me once again how powerful the combined effort of testers and developers can be: I uncovered a potential place of an error and the developer showed how that error could be exploited.
Below is the small java script I used. If this one is entered somewhere in a website and executed then the site is potentially vulnerable to XSS and needs attention:

<script>alert(‘Hello XSS!’)</script>

For more learnings about security issues check out the CWE/SANS Top 25 Most Dangerous Software Errors. At place four for 2011 we have XSS and there is a short section about prevention and mitigation.
Another page I found very interesting to read is OWASP and their dictionary on different attacks.

Happy cross site scripting!


The Unteaming of Teams

Team spirit can be a good thing but when teams starts to compete internally despite having the same goal then it can be a problem.
During a startup meeting for a new project I suggested to the developers that they could bring code to me for testing in an earlier stage. The developers stood up as one entity and shouted “No! We don´t want you or anyone else to file bug reports on our unfinished code!”. The developers were used to having the development and test teams separate and in a kind of competition; the testers were handed the code after it was “done” and then pointed out the errors the developers had made. To calm the developers down and to stress that I was not looking for improving my statistics I told them: “I will not file any bugs before OR after you have delivered your code unless you first agree to it. I can provide help and value by being involved in an earlier stage supporting you.”. And then there was silence…
Once the confusion has settled the developers decided they were ok with the idea. Shortly after that meeting I started to get frequent visits from the developers with suggestions and requests for help with testing of different parts of the software. After four months I was leaving for another assignment and got the question who would be taking over as test lead after me and I replied: “I’m not test lead, that´s the guy over there.”. The answer was followed by the question “Who is that?”.
If you spot teams building walls against each other putting more effort into preparing for long sieges than working together an intervention might be in place. If the people in the teams are not working well together because they are in different teams then Unteaming the teams might help. By unteaming the development and test team and creating a bigger team the cooperation towards our common goal greatly improved. The actual test lead still had the mindset of his team being separate from the other teams and because of this he became less involved in the combined effort. Since he waited for communication via mail or management he missed out on the opportunity to work more proactive.
My conclusion from this experience is that team building can be good thing but if it is done at the expense of less collaboration between teams then unteaming can make a positive difference.

And I want to send a thank you to Michael Bolton for pushing me to write this post and who helped me with feedback!



How building rapport helps me in my work

After the PSL course I started to be more active in building rapport with my colleagues. On the first day of being assigned to a new project I took a different approach than I had done before: Instead of sitting down starting to play around with the development software I took a big cup coffee and started to visit the project members. I decided that the first two days I would not look at my tools or access the software: I would talk talk talk to people. I also decided to not push for information about the software, if anyone signaled they wanted to chat about something else I would do so (or leave if they did not want to chat).

What I experienced was that by not pushing, the developers often started to explain the system on their own initiative. By not saying how I wanted to test, the developers started to spawn ideas of were testing might be beneficial. That information might or might not end up being useful for my testing but it is was a type of information that I did not usually receive before. The developers started thinking on what I might need and such a small thing as someone remembering a meeting or demo that might be interesting for me was a great boost for my work. When the two days ended I had a better understanding than ever before about the new project.

Another specific thing happened to me the other day happened when I strolled over to a developer with whom I sometimes chat but he has no direct relation to the project in which I am working. When asked how things are going I mentioned that I had trouble finding a certain piece of information. Hearing about my small issue he turned to his desk neighbor and 30 seconds later I was making a copy of the information I had been looking for. Just by taking a coffee and chat around I saved a lot of time.

When development gets complex you don´t always know who might have what you need to solve a problem. Building rapport and talking with people can greatly improve your chances of stumbling on a key to a problem.


The importance of building rapport with your team mates – Learning from PSL

There were many lessons to be learned from the Problem Solving Leadership course but in this post I will focus on one thing that was a big lesson for me when it comes to problem solving: The rapport (the connection or relation) with the person or the people you work with might be much more important when solving a problem than I previously believed.

During the course we were one day given two tasks to solve. I had been talking a lot over dinner with one of my course mates and we teamed up to create a draft for the first task. We were able to create the draft quite quickly in comparison with the other groups within the team who had not had as much contact with each other before. And when our draft was presented and approved by the team the team got stuck and it took a long time before we were able to proceed and finalize the task. There were several strong wills that wanted to take charge and move the work in a certain direction giving the people who liked to wait for their turn to speak no opportunity to do so. Such a simple thing as refining our task draft, clear the room and order in food turned into a huge issue were the team had difficulties deciding on anything. After a while a couple of the strong wills decided themselves to take a step back and suddenly the workflow loosened and we were able to make progress and we finished in time.

The second task was solved with bravour despite us having much less time. It seemed that we had resolved our issues during the first task and gotten to know and understand each other. Despite the two tasks being given the same day the difference in the cooperation for solving the problem was dramatic.

In a comment from Jerry Weinberg during the course he approximated that about 70% of a developers time is spent on interactions with other people and that is close to an estimation I did for my own work a couple of years ago. Putting together that approximation with my experiences from the course it became much more visible for me how important the interaction is with my fellow colleagues when trying to solve a problem. In the future I can see myself putting a bigger effort on building rapport to tackle future problems in a better way.

Sida 3 av 41234