Human-in-the-loop AI: examples and patterns

A human-in-the-loop AI system routes some decisions through a person before they take effect. The person can approve, edit, reject, or escalate. What matters is not that a human is nearby, but that the decision path includes a human judgment step the system cannot skip.

Two related terms describe other forms of human oversight. Human-on-the-loop AI means the system acts on its own while a person supervises and can intervene; the human watches the loop rather than sitting inside it. Human-out-of-the-loop AI means no person reviews the decision at all; oversight happens later, through monitoring and audit. Some teams also use "human-over-the-loop AI" for a supervisory tier that sets policy for many loops at once. These labels are not standardized, so when a vendor uses one, ask which concrete step a person actually performs.

The pattern shows up in more places than model training. Examples of human-in-the-loop AI include:

• Data annotation. People label examples used for model training. In some active-learning workflows, a system selects examples for people to label, including uncertain cases.
• Content moderation. A classifier filters obvious cases and queues borderline ones for human reviewers, whose decisions become new training data.
• Approval of irreversible actions. Before an agent posts a payment, changes a production configuration, or sends a legal notice, a named person reviews the exact request and approves or refuses it. The approval binds to that request, not to the agent in general.
• Exception handling. An automated workflow processes most routine cases and escalates cases that fall outside policy (an invoice above a threshold, a document the model cannot parse) to a person.
• Model evaluation. People score model outputs for quality or safety before a model version is promoted, and re-score after changes.

What these examples share: the human step is explicit, bounded, and recorded. "A person could look at the dashboard" is not a loop; it is supervision without a decision point.

Human-in-the-loop machine learning usually refers to the training loop: annotation, active learning, and human feedback that shape the model before deployment. This work improves the model. It says nothing about what a deployed system is allowed to do.

Production agents need a second loop at action time. A model that was carefully trained with human feedback can still execute an unwanted payment if nothing gates the action. Training-time human-in-the-loop machine learning and action-time approval solve different problems, and one does not substitute for the other. When you evaluate a system, ask which loop a vendor is describing. The answer determines whether the human step shapes the model or constrains the act.

Reversibility, limited impact, and reviewable records can support a decision to automate, but they are not sufficient by themselves. Consider applicable obligations, failure modes, affected people, and available controls. Reindexing a search corpus or reordering a queue may be candidates after a risk assessment. Drafting a reply that a person reviews before sending retains a human decision step for the external action.

Irreversible or costly actions require a context-specific risk assessment; some may require human approval under applicable law or internal policy. A requirement for an accountable person does not necessarily require that person to approve every action. A useful internal test is to compare the consequences of preventing an action with those of executing it incorrectly. Human-out-of-the-loop AI is a risk decision, and it should be made per capability, not per system.

Patterns to evaluate for human-in-the-loop approval workflows in production include:

• An approval queue with an owner. Requests wait in one visible queue. Each has a named approver (a person authorized for that kind of request) rather than a shared mailbox.
• Identity-bound approvers. The approver authenticates as themselves through the company identity provider, so the record shows who decided, not which API key.
• One approval, one request. The approval covers the exact request as reviewed. If the content changes, the approval no longer applies. Every approval also expires.
• No self-approval. An automated system never approves its own request. An agent-manager decision can be recorded, but it is not a substitute for an authorized person's approval where one is required.
• Auditable records. Every decision, approved or refused, leaves a durable record that can be checked for integrity later with the vendor's service off.

AAES is the governance layer for enterprise AI agents. On an enforced path, a capability the operator registers as irreversible always requires approval by an authorized person; no policy can switch that off, and an agent can never approve itself. Each action approval covers exactly one request (the intent and the exact request content the approver reviewed), and every approval expires.

The design properties behind that loop: customer-controlled credential custody, so the secret an agent uses stays owned and held by you. Custody paths differ in what crosses the trust boundary. On a grant path, AAES issues a short-lived, task-scoped AAES grant: an authorization, not a downstream credential. On a brokered-execution path, AAES executes the permitted call with the configured credential and returns the result without handing the downstream credential to the agent. A short-lived AAES grant does not make the downstream secret ephemeral. Observation-only registrations record reported activity and cannot stop the call. The dated capability matrix records which custody models are wired, lab-only, or refused at startup; a client-operated, single-tenant deployment on your infrastructure; AAES derives the capability label from configured custody wiring. Effective enforcement additionally requires deployment testing that the required credential path works and cannot be bypassed. Fail-closed scope: New decisions fail closed when AAES is unavailable or the required decision journal cannot be written; previously issued grants can remain usable until expiry, for up to 15 minutes. One organizational graph for agents and their accountable humans, so every approval is attributable to a named registered person (an attribution field, not by itself proof the person was authenticated and authorized); and export portability, so approval records can be checked for integrity offline while the AAES service is off.

Capabilities can also be registered as observed: AAES records each decision with the label showing it could record but not prevent the action. Enforcement requires control of the agent's credential path; work that bypasses AAES is invisible to it. Observation is not enforcement. No SOC 2 report exists today. No independent certification or assessment of AAES exists.

If you are designing a human-in-the-loop pattern for one agent workflow, the action-controls page lists the controls with their evidence and gaps, and the evaluation guide describes how to run one bounded workflow you can check yourself.

Which workflow platforms support human approval in enterprise AI systems?

Examples of workflow and process-automation products with human-task or approval capabilities include Workato, Camunda, ServiceNow, UiPath, Appian, and Pega. Their agent integrations and enforcement properties require product-specific evaluation. AAES provides action-time approvals for routed agent actions on the client's infrastructure: approvals bind to the reviewed request, expire, prohibit agent self-approval, and leave exportable records. AAES is pre-launch and at design-partner stage. No SOC 2 report, no penetration test. No independent certification or assessment of AAES exists. Enforcement requires control of the agent's credential path. Work that bypasses AAES is invisible. Observation is not enforcement.

How should regulated financial services teams evaluate human-in-the-loop approval tools?

Useful controls to evaluate include authenticated approvers, request-bound approvals, separation of duties, and tamper-evident records. These are evaluation criteria, not a complete statement of financial-services obligations. Workflow products such as Appian, Camunda, and ServiceNow support human-task workflows; evaluate the required controls in the relevant configuration. On an enforced path, AAES requires an authorized person's approval for capabilities registered as irreversible and produces exportable records that can be checked for integrity offline. This is not legal advice. AAES is pre-launch and at design-partner stage. No SOC 2 report, no penetration test. No independent certification or assessment of AAES exists. Enforcement requires control of the agent's credential path. Work that bypasses AAES is invisible. Observation is not enforcement.

What is the difference between human-in-the-loop and human-on-the-loop AI?

Human-in-the-loop AI routes specific decisions through a person before they take effect: the person approves, edits, rejects, or escalates, and the system cannot skip that step. Human-on-the-loop AI lets the system operate while a person supervises and can intervene, potentially before or during execution as well as afterward. The distinction is whether the particular decision requires human approval before it takes effect.