Many AI pilots in operations never end. They run for a year, show promising results on some days, and bind the same few experts who are needed elsewhere. Nobody decides, because nobody agreed what success would look like.
A pilot is a question with an end date.
Write the question
One sentence: “Can a camera station replace the visual check at the end of Line 3?” Not “explore AI for quality”. The question defines what is in scope: two product families, three shifts, rejected parts going to a manual check station.
Targets and stop criteria before the start
Agree four to five measures with the sponsor, each with a target and a stop level at the end of the pilot:
| Measure | Target at week 8 | Stop if |
|---|---|---|
| Detection of defective parts | 98 % or more | below 95 % |
| False rejects of good parts | 2 % or less | above 5 % |
| Time per part | 1.2 s or less | above 3 s |
| Operators rating it useful | 4 of 6 or more | 2 of 6 or fewer |
The stop criteria are written before the first camera is mounted. Once a team has invested weeks, it will always find a reason to continue.
Run in parallel
During the pilot, the human check continues. Every part is judged by the camera and by the inspector. Every disagreement is reviewed by the quality engineer.
This does three things: it protects the customer, it shows the real detection rate on the real line, and it produces new labeled images exactly where the model is weak.
Review every week
Thirty minutes every Friday: the numbers against the targets, what was learned, what changes next week. In the example, detection rose from 91.0 to 98.6 percent over eight weeks. False rejects fell from 6.8 to 1.7 percent, with the largest drop in week 3, after the station was shielded against sunlight from the hall windows.
Weekly reviews also catch the problems no test set shows: reflections, new variants, a different supplier, an operator who places parts slightly differently.
Decide on the set date
At the end there are three options:
- Stop: a stop criterion is hit. Record why, and keep the idea on the portfolio.
- Extend: results are close and the reason for the gap is known. Extend once, with a new date.
- Scale: all targets met and stable for several weeks.
In the example all four targets were met and stable for three weeks. The decision: Line 3 in production from April, Lines 1 and 2 in the third quarter.
Put the decision date into the sponsor’s calendar on day one. A pilot with a date gets decided. A pilot without one becomes part of the furniture.