Definition

Human-in-the-loop (HITL)

Human-in-the-loop (HITL) describes AI systems that incorporate human judgment at some stage of their pipeline: labeling training data, evaluating outputs, resolving low-confidence cases at runtime, or auditing automated decisions. The term covers everything from annotation workflows to production systems that escalate uncertain predictions to people.

Where the human sits

Three placements dominate. Training-time: humans produce the labels and preferences models learn from (RLHF is human-in-the-loop training). Evaluation-time: humans judge model outputs to measure quality, catch regressions, and calibrate automated judges. Runtime: the system routes low-confidence or high-stakes cases to a person, using the model for the easy majority.

The engineering challenge is latency and scale. Human judgment has historically been the slowest, most operationally heavy part of any pipeline. API-based human feedback platforms compress this: judgment requests go out programmatically and answers return in seconds to minutes, making evaluation-time and even near-runtime human input practical for agentic systems.