Skip to content
AI in Operations | Part 6 of 10

Every AI Pilot Needs Stop Criteria

Targets, a fixed duration, written stop criteria and a parallel run turn an AI pilot into a decision instead of a permanent experiment.

Many AI pilots in operations never end. They run for a year, show promising results on some days, and bind the same few experts who are needed elsewhere. Nobody decides, because nobody agreed what success would look like.

A pilot is a question with an end date.

Write the question

One sentence: “Can a camera station replace the visual check at the end of Line 3?” Not “explore AI for quality”. The question defines what is in scope: two product families, three shifts, rejected parts going to a manual check station.

Targets and stop criteria before the start

Agree four to five measures with the sponsor, each with a target and a stop level at the end of the pilot:

Measure Target at week 8 Stop if
Detection of defective parts 98 % or more below 95 %
False rejects of good parts 2 % or less above 5 %
Time per part 1.2 s or less above 3 s
Operators rating it useful 4 of 6 or more 2 of 6 or fewer

The stop criteria are written before the first camera is mounted. Once a team has invested weeks, it will always find a reason to continue.

Run in parallel

During the pilot, the human check continues. Every part is judged by the camera and by the inspector. Every disagreement is reviewed by the quality engineer.

This does three things: it protects the customer, it shows the real detection rate on the real line, and it produces new labeled images exactly where the model is weak.

Review every week

Thirty minutes every Friday: the numbers against the targets, what was learned, what changes next week. In the example, detection rose from 91.0 to 98.6 percent over eight weeks. False rejects fell from 6.8 to 1.7 percent, with the largest drop in week 3, after the station was shielded against sunlight from the hall windows.

Weekly reviews also catch the problems no test set shows: reflections, new variants, a different supplier, an operator who places parts slightly differently.

Decide on the set date

At the end there are three options:

  • Stop: a stop criterion is hit. Record why, and keep the idea on the portfolio.
  • Extend: results are close and the reason for the gap is known. Extend once, with a new date.
  • Scale: all targets met and stable for several weeks.

In the example all four targets were met and stable for three weeks. The decision: Line 3 in production from April, Lines 1 and 2 in the third quarter.

Put the decision date into the sponsor’s calendar on day one. A pilot with a date gets decided. A pilot without one becomes part of the furniture.

Newsletter

One Lean idea.
Every week.