Ongoing reliability testing, ORT testing for short, is the last stage of reliability testing once you’ve already designed and developed your product, done all of the testing and validation for it including reliability work to make sure that it’s going to be reliable and durable once it’s manufactured, and you’ve launched the product it’s being sold. Even though the product is now on the market, we’re still doing ORT and we’re going to explain exactly what it is, the tests involved, the costs, how to analyze results, and the risks you face if you’re not doing it.
What is ORT?
ORT stands for Ongoing Reliability Testing and is a battery of reliability tests that is most beneficial while the product is in production and allows you to monitor the reliability of the product and ensure it doesn’t deviate from your reliability goals. It’s done once you have finished your development phase of the product, we’re talking about mostly electronic products here by the way, and you have done your testing and validation and all the reliability testing, and the product is ready to go to production.
Once you start production the first 30 days of shipping are critical for product reliability because a lot of the issues are found in production during that first month. In many cases following a product launch, as time passes, you become aware of these issues because you get all kinds of complaints from the customers and product returns and then you realize: “Oh no, we’ve got all kinds of issues that need to be fixed with the product!”
No one wants to be in this situation, of course, and if you put in a reliability process right from the first week of production which we call ongoing reliability testing then you will avoid a lot of headaches if you don’t do it.
What if you don’t do ORT testing?
The headaches you may suffer from if you don’t do ongoing reliability testing vary, including bad reviews, complaints, warranty claims, and product returns. Let’s address these.
There’s a chance that you may suffer from some product returns to the point where you need to get your engineering team together to figure out what is happening and fix the issue by maybe even redesigning some aspects of the product.
You could also have bad online reviews because the functionality or software is annoying your customers, or they’re just not happy with the product’s functionality and performance. Complaints might also be related to a whole bunch of other issues, for example, the product is dead on arrival or the software isn’t functioning properly.
There might be all kinds of unbelievable issues that could be related to transportation. For example, you know the product was fine when it went through final testing, however, during shipping the vibration, bumps and bangs, humidity, and the changes in the climate, affected the product and it arrived at the destination damaged.
When to start
As you can see, there could be all kinds of issues related to reliability that you need to figure out by doing ongoing reliability testing during the first couple of weeks of production, typically right after the first 5,000 units or so have been produced, as this is when variations in manufacturing are most likely to manifest themselves if they are occurring. Fewer units and the product is still new and the development team and the production team are usually on top of things and trying to iron out all the issues, but usually once you start going beyond 3 or 4,000 units that’s when everyone assumes that everything is all okay, and, therefore, the right time to do ORT to catch the hidden reliability issues that are not quite apparent to the production team.
How is it done?
The main purpose of ORT is to use several tests on full products to monitor product reliability, making sure that it catches design and manufacturing issues before too many products are made from a batch. You need to select some sample products to test and a statistically sound sampling of products to test will be anywhere from 2 to 5% of the daily production. As a minimum I would suggest at least two to three product samples should be put through each critical test in your ORT plan, for example, if you conduct a drop test maybe you test two to three samples and the same for temperature and humidity testing. Doing tests on fully-made products will allow you to confirm that the product consumers will receive is performing within its specifications. If not, it may mean that the product has some serious issues and will need to be put back through your full spectrum of reliability testing that you did during product development at the validation and verification stage.
To make sure that your ORT’s results are trustworthy, it’s critical to select your product samples very carefully from different batches made at different times. Let’s say you need 10 samples for your ORT testing, they should not all be taken from the same shift or same batch. Select them from three different shifts and if you’ve got two or three different batches, take one from each batch per shift. The idea is to make these samples as random as possible so you have the best possible chance to catch all the issues from every aspect of your production.
Once you’ve got your samples you will want to focus on critical reliability tests only because ORT typically won’t put products through the whole gamut of reliability tests. After all, it would take too long to get the results and by then you’ve possibly already shipped thousands of units to consumers which might be defective. A good way to proceed, for example, would be, in the first week, just before 5,000 units are completed, you need to pick the 10 samples and then do a one- or two-day test and get the results. Then you can determine from the results if the production can continue or needs to stop to fix the issues if critical issues are found. This means production isn’t on hold for weeks while you do the ongoing reliability testing.
Selecting tests for your ongoing reliability testing plan
Consider the reliability test plan you used during product testing and validation before production started and then determine which tests are critical, selecting just three to four of these tests to be in your ORT test plan. Depending on the type of the product you need to determine what the critical tests are that impact the performance as well as the reliability of the product. For example, if we were talking about an iPhone as an example product, the temperature and environmental humidity that the phone is exposed to during its lifetime go up and down, so those two will impact how your product will last and should be in your ORT plan. As with any phone, users are going to drop it so you’ll want to add the drop test in there, too, and then there are some software and battery-related tests that you might have to add to ensure that they perform correctly under hot and cold conditions, etc.
You need to find a balance between adding too few and too many tests into your ORT plan. It might be possible to perform a good number of tests if you’ve got your own in-house testing lab or one very local to your factory, as the tests can be done quickly. But if you don’t have your own test lab and you need to ship these units to another test lab, you have to keep into account the shipping, testing, and then return shipping time, because every day that you don’t have the results and you’re shipping a lot of units to customers which may be unreliable. Aside from the time, more testing costs more, so a very long ORT plan may not be cost-effective.
Conduct testing
The next step is to decide when these tests are going to be done. Are they going to be done weekly or monthly? This will depend on the product and how critical reliability is for you and your customers. It might make sense to do weekly ongoing reliability testing on a random selection of samples at the beginning of production, then, as time passes, switch to monthly if the results are good. Your testing schedule depends on the production volume, too. For high volumes, you should probably do ORT testing at least every two weeks during the first month of production because you’re going to be shipping a lot of products and you should minimize the risks of having a lot of returns of unreliable products sneak through. Then, assess the results and if you get any product returns following the ORT. If there are very few or no issues then you could relax the amount of times the testing is done, so going to a monthly, bi-monthly, and even quarterly schedule. Depending on the end of life of the product you can determine how often you want to test after that. Eventually, if it’s found that your manufacturing is running so effectively that there are no variations, you may not require ORT at all.
Analyze results and take action
Now you’re doing the critical reliability tests that are going to catch performance, reliability, and maybe even some quality and software-related issues and now you have results and some failures have been detected. Next, you need to quickly inform management there could be a problem and analyze the data to determine if this is a real failure and decide what action should be taken. You may need to stop shipping immediately if it’s a critical product failure. For example, if you were doing the reliability test and the battery exploded that calls for an immediate halt because we’re talking about a major liability if the products reach consumers.
If failures are found it’s also advisable to rerun the main reliability test that was approved and validated during development and, to go further, check if any of those critical failures were found during validation, and what corrective actions were done to fix them and why they haven’t been effective. New corrective actions will be devised to supersede the former ones that haven’t worked as well as expected.
In some cases, you must inform the customer immediately that you’ve caught an issue with ORT because if they’re expecting a certain amount of units to be delivered on a certain date and they don’t receive them they’re going to be impacted on sales and penalized by their own customers. It”s fairly likely that the penalty may be transferred to you, so there are a lot of money and investment issues here that need to be addressed so time is of the essence to fix the problem so you don’t let down customers.
Ongoing Reliability Testing costs
The cost of running ongoing reliability testing probably pales into insignificance compared to the potential costs that you’re going to incur as a business if you do have defective products going out there and getting returned by the customers, or if you’re missing shipments to your distributors or vendors. In comparison, the cost of ORT is minimal. Depending on the type of the product, you’re going to be running roughly anywhere from five to ten tests, each costing around US$200-500. So in total, your ORT might cost around a couple of thousand USD which is a small price to pay to confirm that your product reliability is on track with the manufacturing volumes you’re producing. Remember, as production is streamlined and any issues are found and fixed, the frequency of your ORT will decrease so it will cost loss over time.
Conclusion
Ongoing reliability testing fits into the category of ‘must-do reliability testing.’ It’s just somewhat unusual because it happens after the product has been launched which might seem a bit strange when you’re thinking about it because as an importer you’re caught up in your product idea, getting it developed, manufacturing, etc, and all of that excitement. But, as you’ve seen, even after the product is being manufactured, the job isn’t quite finished when it comes to the reliability testing.
It’s about adopting a reliability mindset for products that are already in production. Move forward and lock down product reliability by implementing ORT testing, taking corrective actions on issues found, making sure that the revised designs are going to be improved based on the old issues, and remembering to document everything and update your ‘Lessons Learned Database.’
