The buzz surrounding Artificial Intelligence is deafening. From the manufacturing shop floor to the high-stakes world of banking, AI is being hailed as the ultimate solution for reducing defects and reclaiming labour hours.
However, there is a growing gap between “installing” AI and “validating” AI. As a consultant working across these diverse environments, I see a recurring theme: businesses are abandoning the rigours of Lean Six Sigma exactly when they need them most.
AI is often Just a Form of Measurement
Whether it is a machine learning model or a tailored LLM, AI is making decisions. In the language of quality, it is a measurement tool. Before we trust it, we must put it through the standard rigors of Measurement Systems Analysis (MSA).
1. Stability: The LLM Consistency Problem
In manufacturing, if we use a colour-sorting AI, we check its stability using control charts. Does it give the same result over a two-month period? In banking, this is even more critical. If you give an AI the same script and data on different days, does it provide the same outcome? With LLMs, the “background” info can shift, potentially causing a drift in decision-making that could violate governance and compliance standards.
2. Attribute Agreement Analysis
We often don’t know how many defects our manual inspectors are making, or how much rework is hidden in back-office banking processes. By using Attribute Agreement Analysis, we can compare:
3. Statistical Analysis of Old vs. New
To move beyond anecdotal evidence, we must use Hypothesis Testing and Design of Experiments (DOE). We need a formal Statistical Analysis of the old system versus the new AI system.
Back to Basics
The principles of MSA and Process Mapping have worked for generations, whether we were checking a vernier caliper or a quality inspector’s gut feeling. They work just as effectively for AI.
As we explore new technology, let’s not ignore the tried-and-tested basics. If you are implementing AI, ensure you are measuring its success with the same rigour you would any other mission-critical tool.
Statistics for Non-Statisticians | Practical Data Skills For Businesses
Lean Six Sigma Training & Business Improvement | Counter Gravity Consulting
Process Mapping for AI Optimisation | Counter Gravity Consulting
