We previously discussed the importance of doing a pilot run before going into mass production, but in this post let’s look in more detail about who does what during pilot runs done for components and products and the risks of missing something.
Our head of New Product Development, Andrew Amirnovin, talks you through the topic of pilot runs.
In this post, I’m going to describe what a pilot run is, which teams do what and when, the risks of missing something or someone during the process, and requirements that are very critical to having a successful pilot run.
What is a pilot run?
Typically, the word pilot run really means that we want to test the manufacturing of this particular product or component that we’re trying to do the test on. So the pilot run is kind of like a test run. It is usually a manufacturing run, but it can be some other things that different teams might determine. But, at least in the electronics industry, pilot run means ‘test run.’ To do it, usually, you run tests on a small number of samples, either on the product itself in production.
What all this is supposed to reveal to you?
The pilot run is supposed to reveal compatibility with the design, compatibility with manufacturing (such as assuring that a key component is solderable), and that it can be tested with no problems.
Then you need to also perform engineering specifications and performance tests to make sure that the new component neither reduced the performance nor changed the specifications in any way of the overall product.
Is a pilot run only done just before production starts?
Pilot runs can take place throughout the product development process. There are 3 key scenarios where a pilot run may be used:
- Pre-production pilot run of the initial prototype for production.
- Design validation pilot runs.
- Supplier / Critical To Quality CTQ component pilot run for qualifying suppliers.
Pre-production pilot run
While it’s true that a pilot run is sometimes called a pre-production run, they aren’t exclusively before mass production. You can also do the same on certain samples of, for example, electronic products that are using different vendors and suppliers for different components, so that they can keep track of which PCBs are performing better or are more reliable, for example. Doing pilot runs using components produced by different suppliers, allows one to monitor which are higher quality or better performing based on features and specifications.
Design validation pilot run
Another thing that can be considered to be a pilot run is where engineers and the design team run samples of their design to see which design performs best and which one comes close to the actual design requirements, too. You can then start to understand why a lot of product samples need to be produced for testing like this during the NPI process.
Supplier / Critical To Quality CTQ component pilot run
On the other hand, let’s say the design is completed. And now you have a supplier for Critical To Quality (CTQ) components, like display, battery, and maybe the frame that covers some of the components that are custom parts. These are critical components because not having them can derail production completely, so, in fact, it’s usually better to have more than one supplier for these CTQ components (unless you have a very reliable single supplier).
Now, if you’re trying to approve a second supplier for backup (providing second source components), you also have to run a pilot run meaning you’ll build a limited number of products with that particular supplier’s component instead of those from your primary supplier. Then you try to evaluate those products and determine whether they also meet the same requirement/s as the original product or original prototype. So that’s another use case for the pilot run: evaluating and qualifying second source components (which are often called ‘replacement parts’ where form, fit, and function are exactly the same. You might want to write ‘insert drop-in replacement part’ in your product design documentation, etc).
So, when the pilot run takes place depends on at what level and what the objectives are. So for example, if you’re trying to do CTQ, right at the beginning of design, you’re just trying to figure out which supplier to use for the primary part and then you could be doing at PVT second source suppliers.
Earlier product development and pilots
There might be an assumption that a pilot run comes only during PVT and that anything at an earlier stage isn’t really a pilot run, but you’re not doing a pilot run of the final production processes at this point, rather you’re piloting the components and making sure everything is going to be suitable (as we saw in the prior example). In the industry usually, EVT, DVT, and PVT are the prototyping and actually building the product.
On the other hand, different companies with different materials and products, and different types of products might have different reasoning for testing something. So they might do any kind of test run at any point during product development.
An example of the pilot run process
What is really critical for the pilot run is when to do it and what the objective is. A good way to illustrate how a pilot run process benefits us is with this example of design team experimentation to narrow down the correct component/s:
Let’s say you need a new part for your design and you don’t really know how to use a part from supplier A or supplier B and they’re not drop-in replacements. This kind of situation is where the design team wants to do some experimentation and in doing so, they will probably buy sample parts from many suppliers, maybe even five different suppliers, in small quantities of around 10 to 20 pieces/samples.
They will then try and run a very small pilot run on those components, just to see which one seems to be closer to the required performance of the PRD (product requirement document). Some parts might be better, more expensive, or have more features. Some might have a higher wattage or, in the case of displays, they might be brighter or sharper.
So in order to make a decision they have to go through the pilot run process on a component by component basis. The customer might say: “I really would like to have a 4k display, but honestly, I don’t know how expensive it is, maybe we can just have a regular TFT.”
If that’s the case, they need to source TFT displays and test if they work.
As they do that, they might make the decision that this display is cheap and isn’t bright enough. If so, eliminate that supplier. On the other hand, the next display might be amazing, but they’re $50 each and our entire BOM isn’t meant to exceed about that cost. If so, eliminate that supplier, too, due to the cost. And then you go along the same line until you find a display that is optimal in terms of cost and performance.
If a company doesn’t know which supplier to choose, they might suggest doing a pilot run like this on several suppliers’ components.
The timing of the pilot run
Let’s say you’re trying to do a second source component, pilot run. If you do it at PVT and discover that there is a serious issue with the design, would that be a good thing?
For instance, if you started with a primary source component, such as a processor, and built your design around that component. Then, all of a sudden, you tried to do a second source in PVT, having total confidence that your design is good and you discovered that there’s a better part with much more capabilities. For example, a Pentium chip, for example, versus some other. What will you do now? It’s too late to make wholesale changes without big delays and cost overruns in all likelihood.
So this is the reason why you don’t always do a pilot run at PVT. Sometimes you do it early so you have time to revert the design, change the design, update it and make it manufacturable. By waiting until PVT to test second source components, even if you find one that is actually better (higher quality, higher performance, etc), the product has been designed with, say, a cheaper lower-end primary source component in mind and it’s probably too late to replace it with a higher-end second source later because the high-end one is not going to be compatible with the low-end one, and the low-end one doesn’t have the capabilities to handle the high-end one.
However, if the product was designed with a higher-end CTQ component, but later you find that a lower-end component does work ok it will be a different story because the design will be able to accommodate it more easily.
So this is an excellent example, for situations where the pilot run was done wrong in terms of timing, or in terms of choosing the right component.
What do the different teams do during a pilot?
So, in a typical manufacturer’s R&D team you’ve got mechanical, electrical, and component engineering teams, and then also manufacturing engineering, too. Each has different types of objectives and goals, but in the end, they must all work together to make a better product. Unfortunately, if these teams don’t communicate clearly, all kinds of mayhem can happen.
Here’s an example of where communication issues can lead to product safety, quality, or reliability issues:
An electrical engineer had designed a battery that needs to be able to be removed for repair or replacement.
The mechanical engineer assumed that because it needs to be able to be taken out, that is should be made easy to take it out.
They used adhesive on the back of the battery to stick it to the housing and the battery compartment wasn’t too small so the battery could easily be removed.
Having ‘fulfilled the brief’ the mechanical engineers sent the prototype for testing.
This seems logical so far.
The problem was when the product was finished and the reliability team actually performed a drop test, some of the batteries almost exploded.
The root cause was found to be because the mechanical engineering team didn’t apply or choose a stronger adhesive for the battery and, as a result, the battery got loose inside the compartment during the drop test. Because it got loose, the little sharp points aimed at the battery to keep it in its place acted not as stabilizing elements but like knives that went through the battery every time the product was dropped on either side, piercing it leading to smoke, heat, and near explosions.
Why did this occur?
This happened because the electrical engineers never even had a chance to review the design. The mechanical engineers were so busy that they didn’t get a chance to send the initial prototype to them to get their feedback on whether it would be OK. But, most importantly, reliability engineers were not even involved in the design process. Most reliability engineers will just test the product when they receive it if they’re not asked to check designs, etc.
If the teams had simply communicated they would have found that the product had a serious problem, and, unfortunately, it cost plenty of time and money to redesign the product and plastic tooling, and it also illustrates why in a good product design and development team, you always need a reliability engineer.
What would a better strategy be to avoid problems like this?
The electronics and mechanical engineers should have had better communication before reaching that point and especially before ordering the tooling which can cost tens or even hundreds of thousands of dollars.
They should have had reviewed the design between themselves to provide feedback on its pros and cons and then brought in the reliability engineers to give their opinions on if there might be risks with the product design.
At this point only drawings are being discussed, then it may be that a few sample prototypes might be made in small batches in the lab to be scrutinized by the reliability engineering team and other teams.
The key lesson here is before you order the plastic tooling the risks have to be identified. If not, you may need to reorder the tooling at a huge cost and lose the six weeks or so it takes to reproduce it. Worst case, you end up with a terrible disaster like the Samsung Note 7 smartphone that hit the market and put users in harm’s way by exploding (for very similar reasons as given above in the example).
The need for clear objectives and good management
The most important thing that designers and engineering teams, in general, need to keep in mind is that when doing the pilot run the objectives need to be clear and there should be a clear evaluation process afterwards. This evaluation process is how we evaluate and determine if the pilot run was successful or not.
The next thing they need early in the design is to decide and communicate clearly with each other about what to do in the pilot run and when to run it. This is to give them enough time to redesign the product if something goes wrong in the pilot. We already discussed the risks of doing pilots only during PVT. That’s really the last resort before mass production starts, however, and could lead to a fully ready production line standing idle, so program managers will be thinking about when to run pilots in order to give themselves that time buffer that won’t affect the delivery time too badly if a redesign is required.
The program manager plays a very important role here because they lead the process of organizing and monitoring the design, review, and pilot runs processes. The product should only go into production when they say it’s ready. Beforehand, they’ll be asking who is doing the reliability test, if they have gotten the samples yet, when it is scheduled for, if the design and mechanical engineers have made the samples or testing yet, and feeding back on if more are required. All that on top of collecting all the product information and tracking the product development process, schedule, and cost, shows that they’re the fulcrum that the design, validation, and pilot run process revolves around.
Let us know your thoughts on pilot runs. Are they an integral part of your NPI process? Does your Chinese manufacturer embrace or avoid them?
If you have any questions or need help with prototyping, feel free to contact us and we’ll be happy to answer your questions.
