Why I signed the STOP ISO 29119 petition

First of all I must say that I’m sad that such an infected debate between various factions continue to rage in the testing community. It seems to entrench the various views and making it harder for to build bridges between different points of views.

I also think it’s important to point out that the contents of ISO 29119 is not all bad per se. I have read several comments in discussions where people find support in the material presented in ISO 29119 and rightly claim that we cannot all be thought leaders in the test community and come up with new ideas and thus, finding inspiration and guidelines in the work from other people is essential to become a better tester. the ISO 29119 does contain material and information that some testers will find valuable in some context, no doubt about it.

So far, so good, but I nonetheless signed the petition against ISO 29119. Why did I do that? I’m actually a latecomer to the petition and I had the opportunity to read the response to the petition, published here: http://www.softwaretestingstandard.org/29119petitionresponse.php before signing up. In there, I think Dr Stuart Reid does post some valid responses to some of the objections put forward in the petition and in conjunction with the petition; for example I can live with the material not being free and I’m certain that it has been applied to several organizations while the ISO 29119 was in draft state – probably also successfully.

However, then some details in the response that bugged me started to stand out. The first thing is that the response is posted on a site that does not seem to allow any comments or open discussion. Neither could I find any direct link to the response on discussion forums on e.g. LinkedIn. This does counter the argument that everyone is invited to discuss the material in an open and transparent way.

I’ll start a bit out of order, with the following passage from the response:

According to ISO, standards are “Guideline documentation that reflects agreements on products, practices, or operations by nationally or internationally recognized industrial, professional, trade associations or governmental bodies”.

They are guideline documents therefore they are not compulsory unless mandated by an individual or an organization…

Fair, but why not call them ISO guidelines, then? One of the advantages the Dr Reid mentions with the ISO 29119 is that is builds a common vocabulary and that it should not re-invent the wheel. I think standard English already has a perfectly functioning definition of “standard” – and it’s not the same as “guideline”. My fear, and I think many with me, is that many stakeholders in companies will not be able to make this very important distinction. They’ll just read “standard” and read their own interpretation into that.

Dr Reid also writes [emphasis added by me]:

There are some useful IEEE testing standards (e.g. IEEE 829, IEEE 1028) and national standards (e.g. BS 7925-1/-2) but there were large gaps in the standards relating to software testing, such as organizational-level testing, test management and non-functional testing, where no useful standards existed at all. This means that consumers of software testing services (and testers themselves) had no single source of information on good testing practice.

I believe that the single most important factor for the evolution/revolution in testing has been the fact that there has been a multitude of different sources of good information. Many of them conflicting with each other, but full of ideas that help me get different points of views which I can then bring back with me to my projects and apply as appropriate. Admittedly, some sources can be hard to find, and if there were a single portal with all testing related information on the web, then it would probably be helpful, but this is not what the ISO is aiming for. As far as I can tell ISO wants to create a single source of ISO controlled information, which again makes it looks much more like a standard than a guideline.

Moving on to the question about agile and exploratory testing. From Dr. Reid:

The scope of this initial work (ISO/IEC/IEEE 29119 parts 123 & 4) was largely defined by the existing IEEE and BSI standards (which they would replace), although it was clear from the onset that a completely new ‘Test Processes’ standard would be required, in particular to ensure that agile life cycles and exploratory testing were considered, as well as more traditional approaches to software projects and to testing.

Actually, the agile life cycles and exploratory testing are not my main concern; they are already out there and more or less widely accepted in the testing community. What worries me more is what we might miss out on in the future. If ISO 29119 requires “a completely new ‘Test Process'” to accommodate agile and ET, then that also means that agile and ET came about despite material from IEEE, BSI, ISO, ISTQB, etc, not because of it. How does ISO ensure that 29119 does not curtail the next advancement in software testing – something that would require the ISO 29119 test process to be completely rewritten?

The concern about future needs for the test process (or other parts of the ISO 29119) ties into the tailoring of the ISO 29119. It has been stated several places that an organization can basically tailor it any way they like, to fit any context, but if that is so, then it seems not so helpful. It’s like making scaffolding out of play-doh – I can form it into whatever I like, but i won’t really support me much. If, on the other hand, the scaffolding is more rigid, so that it supports me, then it will also be less possible to tailor to anything. Which one is the ISO 29119 – steel or play-doh?

Also, even if ET is covered in the ISO 29119, it might not bee on equal footing. I think it’s telling the Dr. Reid chose the following paragraph in his response to the petition:

“When deciding whether to use scripted testing, unscripted testing or a hybrid of both, the primary consideration is the risk profile of the test item. For example, a hybrid practice might use scripted testing to test high risk test items and unscripted testing to test low risk test items on the same project.”

And stepping back a bit to “standards” and “best practices”. Dr Reid writes [emphasis added by me]:

  • Definition of good practice in the testing industry – a guideline for testing professionals, a benchmark for those buying testing services and a basis for future improvements to testing practices. Note that we do not claim that these standards define ‘best practice’, which will change based on the specific context.
  • A baseline for the testing discipline – for instance, the standard on test techniques and measures provides an ideal baseline for comparison of the effectiveness of test design techniques (for instance, by academics performing research) and a means of ensuring consistency of test coverage measurement (which is useful for tool developers).

Isn’t “ideal” synonymous to “best”? Maybe splitting hairs a bit, but I find it interesting that the word “ideal” is used in the very next paragraph after stating that ISO 29119 does not define “best practices”.

As stated before, I’ve not really been involved in this debate before and I have not tried to get involved in the ISO work with 29119, but I can’t help having a few reflections on the following from Dr. Reid:

However, as a Working Group (WG) we can only gain consensus when those with substantial objections raise them via the ISO/IEC or IEEE processes. The petition talks of sustained opposition. A petition initiated a year after the publication of the first three standards (after over 6 years’ development) represents input to the standards after the fact and inputs can now only be included in future maintenance versions of the standards as they evolve

  1. The following quote from “The Hitchhiker’s Guide to the Galaxy” comes to mind:
    • “What do you mean you’ve never been to Alpha Centauri? Oh, for heaven’s sake, mankind! It’s only four light years away, you know! I’m sorry, but if you can’t be bothered to take an interest in local affairs, that’s your own lookout! Energize the demolition beam!”
  2. Does it really matter when feedback comes? It does worry me somewhat that ISO basically says “It doesn’t matter whether we have published something good or bad – since it has been published, then we must stick to our predefined change request process”.

After all of this, I must still say that I kind of wished that I could endorse ISO 29119; a lot of talented and experienced people have put a lot of effort and thoughts into this and it does contain some valuable information that I very well might end up using in some project some day.

I would not have signed the petition if I thought ISO really sees the 29119 as just another asset in the great library of thoughts and ideas out there in the testing community. However, the more I read the discussions around ISO 29119 and the responses from proponents like Dr. Reid, the more I feel that there is a disconnect between what e.g. Dr. Reid writes about standards, guidelines, best practices, tailoring, etc. and what he actually means. For something that has the potential impact like the ISO 29119, I believe the risk is just too great and it would need more testing before release – that’s why I have signed the petition to stop ISO 29119.

Leave a Reply

Your email address will not be published. Required fields are marked *