The Story of DACHS’ Informal TMMi Assessment

One of our highly motivated test managers attended a webinar about Test Maturity Model integration (TMMi), organized by the Hungarian Testing Board (HTB). She won a free informal audit opportunity for our company. This informal audit covered a two-day event during which testing, and quality assurance professionals evaluated the testing processes of two DACHS projects.

The examination was based on the TMMi Model, which has been confirmed to be the number one model for test process improvement in the world. This model looks at software testing at different maturity levels, with the starting assumption that all organizations start at TMMi level 1 of the maturity ladder. There are five levels for representing the different maturity levels of an organization.

During the evaluation, two experts of the HTB, Dr. Katalin Balla and Zsolt Hargitai examined how well our testing processes were documented, how well they were put into practice and how fruitful their application was.

So, every process was investigated regarding the:

• Approach we use,
• Deployment we do and
• Results we have.

All three factors are vital for success, but their level of professionalism can be different in different processes.

The objective of the Appraisal was to perform a gap analysis to:

• Understand how processes used by DACHS relate to TMMi processes,
• Understand strengths and improvement possibilities,
• Provide basis for a test process improvement program.

The experts examined for example our Test Policy and Test Strategy documents, and then based on these documents, they looked at how we plan and track project testing, how we write our test scripts, how we run them, and how we manage test environments. These are all tasks of the second level of the five-step model already mentioned.

The assessors also looked at what the testing organization currently looks like, what training we have, how reviews are conducted. They analyzed the integration of development and testing, and they investigated non-functional testing processes.

Our lessons learned and key takeaways

The preparation for the assessment alone, brought huge advantages. We had already learned a lot during the preparation: We were preparing a lot for the assessment, creating documentation that hadn’t been ready before, processes that hadn’t worked at the company level so far. Preparing for the assessment gave us a boost, for example to put together a formal test policy for the company.

We didn’t fly blindly; we knew in advance, based on the TMMi model what we needed to prepare for.

Thanks to our more than 20 years’ testing experience in regulated environment, the base for validated testing is very stable. There is a set of good practices in place and our employees are highly qualified in software testing. The management supports software testing, and it is a recognized and important task to be a tester.

It was very valuable feedback from the assessors, that we are moving in the right direction, we are on the right track. As one of the test-automation experts said: “Our work has paid off.”

Our progress was visible and now we know what else to focus on. Overall, the direction is good, we still need to document some further processes, review, and monitor more.

It laid the groundwork for us to see what we did, and how we did it. Through the three evaluation criteria (approach, deployment, and result) we found that although in some cases processes are not completely well planned and documented, but still they work well.

We found that monitoring and controlling activities could be expanded to our processes, bringing quality and progress reviews into prominence. One of the lessons learned is that TMMi gives us not just an extra documentation task, but a set of criteria that, if we take it into account and always try to work based on it, will bring us great results. Processes will become more monitorable, they can be better presented to the management or even to the client.

Another takeaway is, that we need to work out the KPIs better and put more effort on non-functional testing. At level 3, non-functional testing is a very important factor. Though we already use performance tests in non-functional testing, we need to put more effort in this area in the future. We received confirmation from the assessment that this is also an important area, which should also be monitored beside the functional tests. There are several aspects of non-functional testing we can focus on in the future.

We found that we must better analyze and define the real testing requirements of our projects. Different projects have different testing requirements, as well as the very same project can have different testing requirements in its different phases. It is not enough to acknowledge this and apply it in practice, but it must also be well documented, why we do it the way we do. Tailoring our testing processes to our projects must be well established based on sound analysis. The decisions made about e.g., which specific documents we choose to prepare for a specific testing process, must be based on real needs. Decisions made, and their causes must be well documented as well.

We got countenance from the assessors on our strongest processes:

• We have a company-wide-accepted Test Policy and Test Strategy.
• There are project specific Master Test Plans in place based on the Test Strategy.
• Test Planning is strong in the assessed projects with the estimation processes being exemplary well established.
• Test Design techniques are consciously chosen for each level and types of testing.
• Intake tests are available and used widely.
• External ISTQB trainings are complemented with internal application and testing related trainings to provide testing professionals diversified opportunities.

Of course, every coin has two sides, the assessment also highlighted some areas for improvement, such as:

• The level of institutionalization/usage of the standard documents is not complete,
• Tailoring of the high-level documents is not done in full detail yet. Projects can use different documents at different level of detail.

We received some recommended next steps, for example:

• Continue the process improvement (based on a set of best practices, e.g., TMMi) and
  optionally target an official TMMi level (which can be a motivating factor for the company).
• Complete parts that are missing from process descriptions
  g., Reviews (progress, milestone, and peer), KPIs, Non-functional testing…
• Further improve and institutionalize project level documents for the testing process (i.e., test strategy, test policy etc.).

About the assessment

It is very motivating for everyone dealing with testing that we have been evaluated in this assessment. It was altogether a very positive experience, and a highly motivating foundation for our future improvements.  The assessors’ competency was above question, and it was a great experience to receive feedback from them. Thanks to the TMMi model, now we see a clearly defined goal set, a path for development and the criteria how we can achieve our goals.

As it was an informal assessment, it was more like an internal audit. Documentation and processes weren’t checked by the assessors as profoundly as for a formal assessment. The assessment was based on the self-evaluation of the members of the test organization, so honesty and realistic self-evaluation are very important in such informal assessments.

Our plan is to use our own experiences together with the assessors’ suggestions to further improve our processes.

We believe that a formal TMMi Assessment, with a certification gained is an outstanding opportunity to show your customers the quality of your services. We hope that in the future we will have the opportunity and resources to prepare for a formal assessment as well.

Based on such an international evaluation, our existing and potential customers can get a comprehensive picture of the maturity of our testing processes.

More about TMMi

The TMMi Foundation is a non-profit organization, which supports organizations around the world to improve their software and system testing and achieve higher and sustainable levels of product quality for the systems they are developing and maintaining. With TMMi, these organizations can assess and improve their test processes in a cost effectively manner and, if required, become formally certified.

The TMMi Foundation provides:

• A freely available software quality and test maturity model named Test Maturity Model integration, (the TMMi® Model).  This model has been confirmed to be the number one model for test process improvement in the world.
• Knowledge certification called TMMi Professional and TMMi test process improver to demonstrate an understanding of the model and provide visibility for leading professionals in the field of test process improvement. Recognized training providers deliver TMMi Professional training world-wide.
• A path and training to become an accredited TMMi Assessor and/or Lead Assessor.
• A TMMi Assessment Method (TAM) under license, for use by accredited TMMi Assessors and Lead Assessors, to enable formal and informal assessments to be undertaken.
• A framework for organizations to build their own assessment method; the TMMi Assessment Method Application Requirements (TAMAR) describes all the details with which a TMMi assessment method must comply in order to be accredited.

Local representation of the TMMi Foundation is provided through so-called TMMi local chapters.

The Hungarian Testing Board is the TMMi Hungarian local chapter.

“HTB (Hungarian Testing Board) finds important to support Hungarian testers in their professional development. Extending HTB’s portfolio by becoming a TMMi Local Chapter in Hungary is a milestone in this regard, and we are looking forward to see companies around improving their testing practices. Since a growing number of testers hold expert (CTEL-ITP) certificates in Hungary, providing a further step in their development of individual professionals also a clear goal for HTB, – for them a logical next step is to put theory into practice, for which HTB is about setting up the scene.”- Zsolt Hargitai

• 

23/09/2021