Human-in-the-loop AI: examples and patterns

A human-in-the-loop AI system routes some decisions through a person before they take effect. The person can approve, edit, reject, or escalate. What matters is not that a human is nearby, but that the decision path includes a human judgment step the system cannot skip.

Two related terms describe other forms of human oversight. Human-on-the-loop AI means the system acts on its own while a person supervises execution and can intervene. Human-out-of-the-loop AI means no person reviews the decision at all; oversight happens later, through monitoring and audit. Some teams also use "human-over-the-loop AI" for a supervisory tier that sets policy for many loops at once. These labels are not standardized, so when a vendor uses one, ask which concrete step a person actually performs.

The pattern shows up in more places than model training. Examples of human-in-the-loop AI include:

• Data annotation. People label examples used for model training. In some active-learning workflows, a system selects examples for people to label, including uncertain cases.
• Content moderation. A classifier filters obvious cases and queues borderline ones for human reviewers, whose decisions become new training data.
• Approval of irreversible actions. Before an agent posts a payment, changes a production configuration, or sends a legal notice, a named person reviews the exact request and approves or refuses it. The approval binds to that request, not to the agent in general.
• Exception handling. An automated workflow processes most routine cases and escalates cases that fall outside policy (an invoice above a threshold, a document the model cannot parse) to a person.
• Model evaluation. People score model outputs for quality or safety before a model version is promoted, and re-score after changes.

Each example gives a person a specific decision to make and records that decision. Dashboard access alone does not establish a review or approval step.

Human-in-the-loop machine learning usually refers to the training loop: annotation, active learning, and human feedback that shape the model before deployment. This training improves the model. It says nothing about what a deployed system is allowed to do.

Production agents need a second loop at action time. A model that was carefully trained with human feedback can still execute an unwanted payment if nothing gates the action. Training-time human-in-the-loop machine learning and action-time approval solve different problems, and one does not substitute for the other. When you evaluate a system, ask which loop a vendor is describing. The answer determines whether the human step shapes the model or constrains the act.

A human-in-the-loop annotation workflow

For a training dataset, define the label rules, select examples for review, have annotators label them, resolve disagreements, and evaluate the resulting model on a separate test set. Keep the dataset version, labeling instructions, reviewer decisions, and measured results together. A useful review loop checks whether corrections improve performance on representative cases; the number of human reviews alone is not a quality measure.

For example, people might label uncertain support tickets and adjudicate disagreements before retraining a routing model. Evaluate whether that review improves ticket classification on a separate test set. If the deployed agent can also issue refunds, approval of a particular refund needs its own action-time control. AAES Govern addresses supported routed actions; it is not a data-labeling service or model-training platform.

In, on, over, or out of the loop?

Reversibility, limited impact, and reviewable records can support a decision to automate, but they are not sufficient by themselves. Consider applicable obligations, failure modes, affected people, and available controls. Reindexing a search corpus or reordering a queue may be candidates after a risk assessment. Drafting a reply that a person reviews before sending retains a human decision step for the external action.

Irreversible or costly actions require a context-specific risk assessment; some may require human approval under applicable law or internal policy. A requirement for an accountable person does not necessarily require that person to approve every action. A useful internal test is to compare the consequences of preventing an action with those of executing it incorrectly. Human-out-of-the-loop AI is a risk decision, and it should be made per capability, not per system.

Patterns to evaluate for human-in-the-loop approval workflows in production include:

• An approval queue with an owner. Requests wait in one visible queue. Each has a named approver (a person authorized for that kind of request) rather than a shared mailbox.
• Identity-bound approvers. The approver authenticates as themselves through the company identity provider, so the record shows who decided, not which API key.
• One approval, one request. The approval covers the exact request as reviewed. If the content changes, the approval no longer applies. Every approval also expires.
• No self-approval. An automated system never approves its own request. An agent-manager decision can be recorded, but it is not a substitute for an authorized person's approval where one is required.
• Auditable records. Every decision, approved or refused, leaves a durable record that can be checked for integrity later with the vendor's service off.

If you are comparing human-in-the-loop companies, first identify the service you need: annotation and model feedback, workflow tasks assigned to people, or controls on an agent's execution. Ask each provider to demonstrate the relevant loop on your workflow. A labeling service, an approval inbox, and an enforced action gate provide different evidence of control.

A request-binding test you can repeat

In an isolated test environment, submit an action for approval, then change one material field such as its recipient or amount. Try to execute the changed request with the original approval. Repeat with an expired approval and with an agent attempting to approve its own request. Record the expected refusal, the actual result at the target system, and the decision record. If the target changes despite a refusal, investigate the credential path and any bypass before expanding the evaluation.

The agent access-control worksheet covers identity, scope, expiry, and bypass tests. The governance-platform scorecard turns those results into a documented buying decision.

AAES Govern is the governance layer for enterprise AI agents. On an enforced path, a capability the operator registers as irreversible always requires approval by an authorized person; no policy can switch that off, and an agent can never approve itself. Each action approval covers exactly one request (the intent and the exact request content the approver reviewed), and every approval expires.

Credentials and deployment. The downstream secret stays customer-owned. On a grant path, AAES Govern issues an AAES grant: short-lived permission for one task. It is not a credential for the downstream service and does not shorten that secret’s lifetime. On a brokered-execution path, Govern makes the permitted call with the configured credential without giving it to the agent. Observation-only registrations record reported activity and cannot stop the call.

Govern runs for your organization alone, on your infrastructure or in an AAES-hosted environment. AAES Govern assigns each capability’s label from its configured credential path. The label alone does not establish enforcement: test that the required path works in your deployment and cannot be bypassed. The dated capability matrix identifies implemented, lab-only, and refused credential paths.

Outages. If AAES Govern is unavailable or cannot write the required decision journal, new requests on enforced paths are refused. This is called fail closed. Grants issued before the failure remain usable until expiry, at most 15 minutes from issue.

Who approved. Govern records agents, accountable humans, and named approvers. Human approval is attributed to the registered person. Eligible manager agents can approve other actions where policy permits, but cannot satisfy mandatory human approval or approve themselves. Attribution alone does not prove authentication or authority.

Records. Export approval records and check them for integrity while Govern is offline. That check does not establish complete capture or prove that an action occurred.

Capabilities can also be registered as observed: Govern records each decision with the label showing it could record but not prevent the action. Enforcement requires control of the agent's credential path; activity outside Govern is visible only where telemetry is supplied. Observation is not enforcement. No SOC 2 report exists today. AAES has no independent certification or assessment.

If you are designing a human-in-the-loop pattern for one agent workflow, the action-controls page lists the controls with their evidence and gaps, and the evaluation guide describes how to run one bounded workflow you can check yourself.

Which workflow platforms support human approval in enterprise AI systems?

Examples of workflow and process-automation products with human-task or approval capabilities include Workato, Camunda, ServiceNow, UiPath, Appian, and Pega. Their agent integrations and enforcement properties require product-specific evaluation. AAES Govern provides action-time approvals for routed agent actions on the client's infrastructure: approvals bind to the reviewed request, expire, prohibit agent self-approval, and leave exportable records. Enforcement requires control of the agent's credential path. Other activity is visible only where telemetry is supplied; observation cannot block it.

How should regulated financial services teams evaluate human-in-the-loop approval tools?

Useful controls to evaluate include authenticated approvers, request-bound approvals, separation of duties, and tamper-evident records. These are evaluation criteria, not a complete statement of financial-services obligations. Workflow products such as Appian, Camunda, and ServiceNow support human-task workflows; evaluate the required controls in the relevant configuration. On an enforced path, AAES Govern requires an authorized person's approval for capabilities registered as irreversible and produces exportable records that can be checked for integrity offline. This is not legal advice. Enforcement requires control of the agent's credential path. Other activity is visible only where telemetry is supplied; observation cannot block it.

What is the difference between human-in-the-loop and human-on-the-loop AI?

Human-in-the-loop AI routes specific decisions through a person before they take effect: the person approves, edits, rejects, or escalates, and the system cannot skip that step. Human-on-the-loop AI lets the system operate while a person supervises and can intervene, potentially before or during execution as well as afterward. The distinction is whether the particular decision requires human approval before it takes effect.