Human-in-the-loop is a control design in which people review, correct or authorize selected outputs from an automated or AI-supported process. It is most useful when consequences are material, data is uncertain or judgment cannot be fully encoded.
Where can human review occur?
- Before an automated decision takes effect
- When confidence falls below a threshold
- For unusual or high-value transactions
- After a user disputes an outcome
- During quality sampling and model monitoring
- When new products or document formats appear
How should the workflow be designed?
- Define which decisions require review.
- Present the source evidence and system output.
- Give the reviewer controlled actions and reason codes.
- Prevent material changes after approval without re-review.
- Record the decision and corrected data.
- Use confirmed outcomes to improve the process.
Human-in-the-loop vs. human-on-the-loop
Human-in-the-loop requires intervention within selected decisions. Human-on-the-loop supervises an automated process and can intervene when needed. The appropriate model depends on risk, volume and reversibility.
What makes review effective?
Reviewers need expertise, sufficient time, independent evidence and authority to reject the recommendation. A mandatory click without meaningful information creates the appearance of control rather than real oversight.
What should be measured?
Track referral rate, reviewer agreement, overturn rate, error types, resolution time and downstream outcomes. A very low overturn rate can mean accurate automation or passive reviewers, so sample audits remain important.
