Michel van Schie
Introducing a newly developed product to the market is entering uncharted territory. What hidden problems may arise when the product is deployed 'in the field' and how significant will their impact be?
The recent incident of the unfortunate disaster with the Philips sleep apnea device has demonstrated the potentially devastating effects of unforeseen issues. Another example is the accident with the Dutch Stint electric child transporting cart, all but bankrupting the manufacturer. Critical aspects of new products are often poorly understood until it's too late, with some failures only surfacing after hundreds or thousands of units have reached the market. There is always a large gap between prototype testing and real-world usage. A thorough approach to testing will narrow this gap.
While prototypes and pilot series are crucial for risk reduction before mass production and sales, we believe that their full potential during the development process is often underutilized. The scope of what can and should be tested is frequently underestimated. In hindsight, many product developers admit that they would have wanted to test more extensively and intelligently.
The cause of this underutilization is often attributed to the high pressure exerted by management or sales to meet release deadlines. However, we believe it also stems from a lack of drive and creativity in systematically eliminating as many risks as possible. “The designer wants something to succeed. When testing you should actually want things to fail.” Although it may not be feasible to eliminate all post-release failures, given their potentially significant consequences, we must reevaluate the value of seemingly redundant or unnecessary tests.
To elevate the professionalism of testing, several key factors are to be considered:
Start with an extensive risk analysis. Gather a team that can think of risks from different perspectives. Engineers from different fields, but also various types of users and indirect stakeholders. When generating potential risks, make sure to dive into actual use and mis-use scenarios. Use tools like roleplay or process trees to get to the details of the scenario. Of course, use proven methods to help organize, prioritize and mitigate risks, but it starts by being open and creative in ‘mining’ potential risks. You should start building the risk analysis already from the early concept stage.
Based on the long and often discouraging list of risks, you can start defining tests on your prototypes to find out if you can (re)produce the failure and its effect. This also includes designing what kind of prototype you need to test this. Sometimes to test an effect, you only need to build and test smaller, isolated parts of a system. Most tests do not require a fully functional and detailed prototype. This means you can fail fast and cheap, limiting also the risk of wasting development budget.
When you have done the planned tests, continue testing if you have the time. Quite often, when the defined test is done, the prototype is placed in a corner to collect dust. Why? This prototype can teach you so many more lessons. Who remembers the ‘Myth busters’. After they performed a nominal test with the outcome: “Myth plausible or myth busted”, Adam and Jamie always continued testing to the point of beakage. Keep performing tests on your prototype while waiting for other parts of the project to be finished. Test until your prototype is completely worn out and cannot teach you any more lessons. And always ask yourself the question: “what more can I test?”
Gather as much data on the tests as possible. Measure before and after. Look for micro-cracks, small deformations and wear marks. Perceive changes in sound. A worn running shoe can tell you what kind of runner you are. Be forensics and find that drop of blood.
When things finally break, don’t jump to conclusions. A smoking gun does not mean the victim was shot. The real root cause is often different or more complex than your original thought. A breakage of a structure is a rich source of information leading to an infinite set of possible solutions. To properly understand the mechanism will often require you to reproduce the failure, while only changing a single parameter each time.
A great deal of creativity is required in planning and performing your validation tests. Approaching strategic testing as a design process, enables us to determine what parameters to validate and how to validate them effectively, considering time and budget constraints. In the end, this philosophy will lead to longer-lasting products with longer-lasting customer satisfaction and thus an important factor in sustainable design.
