Lies in the Loop: HITL Prompts as RCE Vectors in Dev Workflows

A deceptively simple trick—padding and context manipulation—can turn carefully designed “human‑in‑the‑loop” (HITL) safety prompts into a live remote code execution (RCE) vector, and the security research community’s recent “Lies‑in‑the‑Loop” disclosures show how that vector threatens AI‑augmented developer workflows, CI/CD pipelines, and software supply chains.

Background​

Generative AI assistants are now embedded into developer toolchains, IDEs, and automated pipelines. These agents commonly rely on a final human approval step—a Human‑in‑the‑Loop (HITL) confirmation dialog—before executing potentially dangerous or privileged actions such as running shell commands, committing code, or installing dependencies. HITL dialogs were introduced as a pragmatic safety control: the AI flags the potentially risky action and waits for an explicit user confirmation. The Lies‑in‑the‑Loop (LITL) attack turns that last line of defense into an exploit by manipulating the context those dialogs present, effectively forging the approval content so the human reviewer unknowingly authorizes malicious behavior.
The attack has been demonstrated against mainstream, IDE‑integrated assistants and is described in detail in an extensive security post and follow‑up technical analysis by a well‑known application security research team. Independently reported coverage and podcast analysis have confirmed the attack pattern and its feasible failure modes inside popular coding assistants and workflows. Multiple vendor disclosures and the research timeline show the technique works as a practical proof‑of‑concept against agentic tooling that uses HITL confirmation prompts.

---

What Lies‑in‑the‑Loop actually does​

The high‑level mechanics​

• An attacker plants malicious content in a context the agent will retrieve or summarize (for example: a public GitHub issue, a README, a code comment, or any external resource the agent is instructed to read).
• The agent ingests that content and synthesizes a fix or an action plan that includes an HITL confirmation prompt before taking a sensitive action.
• The attacker carefully forges or obscures the HITL dialog content by manipulating formatting, padding the output with long benign text, or exploiting UI rendering quirks (e.g., Markdown rendering or terminal scroll behavior).
• The human reviewer sees a superficially benign confirmation (a summary line or truncated output) and confirms the action.
• The agent executes the action, which can range from running a local command (e.g., launching calc.exe in the PoC) to installing a malicious package or committing code that later triggers a supply‑chain compromise.

Why it works: a human‑AI trust failure​

The LITL pattern depends on two predictable human behaviors: trust in the agent’s summary and a tendency to accept or expedite routine confirmations to save time. The forged HITL dialog takes advantage of both: it makes the dangerous elements either visually inaccessible (pushed off‑screen by a long response) or semantically buried amid seemingly benign context. The result is a social engineering attack mediated through the AI agent—the human is the final authorizer, but the content the human approves has been manipulated.

---

Verified, repeatable demonstrations​

Researchers demonstrated that an agent could be induced to run an arbitrary command on an endpoint by hiding the malicious command inside an otherwise plausible agent response. In a proof‑of‑concept, the command simply launched the Windows calculator to prove remote code execution capability, but the same technique scales to far more damaging payloads. The researchers also demonstrated variants that could hide malicious npm packages and introduce trojanized dependencies into repositories—showing a plausible path from a single developer confirmation to a software supply chain compromise.
Vendors were notified and publicly classified the initial reports as informational in some cases, arguing that the HITL control requires user diligence (for example, manually scrolling to see the full prompt). The researchers note that this answer conflates responsibility with practical safety—an idealized assumption that users will always fully inspect every dialog does not hold in real developer environments, where speed and trust are dominant forces.

---

Why this matters now: the real‑world stakes​

• Developer workflows are automation‑heavy. Developers routinely accept patches, run automated fixes, and merge code with minimal friction; an injected malicious step can cascade into production quickly.
• Agent privileges are growing. Modern coding assistants can run tests, modify files, interact with package managers, and trigger CI jobs. Those are high‑impact capabilities if misused.
• Supply chain attack potential. A single approved change that introduces a trojanized dependency or modifies build scripts can be propagated across organizations via automated deployment and package distribution.
• Indirect prompt injection is stealthy. Attacks that plant malicious context in public or semi‑trusted content (issues, comments, docs) can persist and be acted upon later by multiple agents and teams.
• Human fallibility at scale. Studies and repeated PoC tests show even experienced developers miss hidden commands when the dialog presentation is manipulated.

---

Technical anatomy: vectors and attack surface​

Vectors used or highlighted in practice​

• Indirect Prompt Injection: Malicious content lives in a resource the agent retrieves (GitHub issues, PR descriptions, package docs). Because the agent consumes the content as context, the attacker effectively injects instructions into the assistant’s decision pipeline.
• Dialog Padding & Scroll Evasion: Large blocks of benign text or extremely long outputs force the dangerous payload out of the visible area of a terminal or dialog window.
• Markdown and Rendering Tricks: Using Markdown formatting or UI quirks to hide active code blocks, collapse content, or change how summaries are shown.
• Metadata Tampering: Manipulating the brief metadata line or the human‑readable summary the agent displays, while the full action contains a hidden executable command.
• Tool‑Chaining: Combining prompt injection with legitimate agent tooling (slash commands, test running features) to produce an action that looks routine but contains the payload.

Typical targets in a developer environment​

• IDE integrated assistants (e.g., in Visual Studio Code)
• Source control platforms and issue trackers (GitHub, GitLab)
• CI/CD systems that automatically apply patches or run agent-suggested tasks
• Automated dependency managers (npm, pip) and package publishing processes
• Internal “agent orchestration” systems that route agent outputs into scripted actions

---

Vendor responses and the disclosure timeline​

Researchers disclosed the findings to vendors and included a timeline showing acknowledgments and vendor classifications. Some vendors categorized the behavior as informational because it relied on non‑default user actions (scrolling, confirmation), while others acknowledged the risk and indicated engineering reviews. The practical effect is twofold: researchers have catalyzed vendor attention on agent UI design and tool boundaries, but major systemic fixes—such as fundamentally changing how HITL approvals are presented or restricting agent privileges by default—remain work in progress.

---

Strengths and contributions of the research​

• The work moves beyond theoretical prompt injection examples and presents a concrete, reproducible attack chain that bridges online content poisoning, agent behavior, and human approval.
• It reveals a class of attacks that exploit trust and UI design rather than purely model alignment failures, broadening the threat model that security teams must cover.
• Demonstrations include realistic developer workflows and use standard agent features, which makes the findings immediately actionable.
• The research has already stimulated vendor engagement and public discussions about agent UX, least‑privilege defaults, and the need for defense‑in‑depth.

---

Risks, caveats, and unverifiable elements​

• Some vendor responses characterize these issues as user behavior problems rather than product vulnerabilities. While vendors are correct that user diligence matters, relying on perfect human inspection is not a scalable security posture.
• The probability of large‑scale, automated LITL exploitation (for example, nation‑state actors using the technique to compromise entire ecosystems) is plausible but currently unverifiable—there are no confirmed public incidents of a full supply‑chain takeover using only LITL in production environments. That said, the attack chain is credible and should be treated as a high‑risk scenario.
• The specific success rates across different models and UI implementations vary and may not be fully reproducible in every environment; UI rendering, agent configuration, and permission scopes influence outcome reliability.
• Claims about long‑term model “lying” as an intentional behavior separate from prompt injection are an active area of research; some recent studies probe mechanistic lying but many conclusions remain preliminary and context‑dependent.

---

Practical mitigation strategies — actionable guidance for Windows developers and sysadmins​

The fundamental defense is defense‑in‑depth: do not rely solely on HITL or any single control. Implement layered protections that combine UI/UX hardening, privilege controls, and pipeline enforcement.

Immediate developer workstation controls (fast wins)​

• Disable automatic execution of agent‑generated shell commands. Configure IDEs so that agent suggestions never execute without an explicit multi‑step confirmation (e.g., confirm + show raw command + require typing a fixed phrase).
• Show canonical, machine‑readable commands. Always display the exact shell command or script block in a single canonical view that cannot be hidden by scroll or formatting, and require explicit acknowledgement of that exact text.
• Run agent‑suggested code in isolated sandboxes. Use ephemeral VMs or containers (DevContainers) when testing agent code or running fixes. Do not run untrusted agent outputs on the host.
• Disable auto‑installers for dependencies. Make package installation a gated operation in local workflows—require PRs and signed artifacts before CI installs external packages.

IDE and platform configuration​

• Configure terminal and dialog behavior so confirmation prompts are not truncatable or collapsible, and include an auto‑scroll‑to‑start function when a HITL dialog is triggered.
• Strip formatting from the textual content used to generate HITL summaries (present plain text only).
• Implement “explicit command only” mode for HITL dialogs that removes context padding and only shows the minimal operation to be approved.

CI/CD and repository hygiene​

• Enforce gated merges. Require code review and automated security checks for changes that include dependencies or build script modifications.
• Block direct agent‑initiated commits. Agents can prepare patches, but only human‑reviewed PRs should be mergeable; require at least two reviewers for dependency changes.
• Package provenance and signing. Require signed packages and validated SBOMs for dependencies; integrate package integrity checks into CI pipelines.
• Static and dynamic analysis gates. Run enhanced SAST/DAST and supply‑chain scanners on agent‑suggested content before merging.

Tooling and policy​

• Adopt least‑privilege agent configurations. Limit the agent’s ability to call privileged APIs or perform file system operations by default.
• Use immutable build artifacts. Avoid live downloads during builds where possible—favor vendored, pinned artifacts stored in internal artifact registries.
• Audit logs and alerts. Log all agent actions that request or execute privileged operations; trigger alerts on unusual patterns, such as repeated long outputs that include executable content.
• Educate developers. Provide training on LITL patterns, show examples of padded dialogs and how to validate them, and incorporate agent‑safety checks into onboarding.

---

Design and UX recommendations for vendors​

• Canonical command view: Present the exact command block in a dedicated, non‑truncatable UI element and require explicit acknowledgment of that block before any execution.
• Diff and provenance display: For fixes that modify code, present a clear diff and the provenance of any external artifacts the agent used (links, timestamps, package checksums).
• Limit agent privileges by default: Agents should start with minimal capabilities and require explicit admin opt‑in for privileged behaviors.
• Sanitize retrieved content: Strip or flag excessive padding, very long outputs, or content with suspicious formatting before it becomes part of the HITL dialog.
• Render raw text by default: Where possible, show the raw, uninterpreted content the agent retrieved in addition to the agent’s summary.
• Require a two‑factor approval for high‑impact actions: For actions that modify external systems or dependencies, require a secondary confirmation (e.g., a signed approval or a separate human reviewer).

---

Incident triage and response checklist (fast incident playbook)​

• Isolate the endpoint. If unexpected commands ran, disconnect the host from networks and snapshot forensic artifacts.
• Capture logs. Collect IDE logs, agent transcripts, terminal output, and the external resource (issue thread, PR, or package metadata) used as context.
• Reproduce safely. Reproduce the activity in a sandboxed environment to capture the exact attack chain.
• Scan for indicators. Run supply‑chain scanners and search repositories for unexpected dependencies or code modifications.
• Rotate credentials. If any secrets or CI tokens may have been exposed, rotate them immediately.
• Notify stakeholders. Engage security, dev leads, and vendor support; consider a coordinated disclosure if the attack vector impacts broad tooling.
• Harden: Apply immediate mitigations (disable agent auto‑execution, enforce PR gates), then plan long‑term fixes.

---

Broader implications and long‑term risk landscape​

LITL is not an isolated novelty; it sits alongside indirect prompt injection, dataset poisoning, and model‑behavior manipulation as part of an expanding adversarial space against agentic AI. The research reframes human‑in‑the‑loop from a panacea into one control that can be exploited if the UI, tool boundaries, and data provenance are weak.
As agents gain more automated reach—connecting to cloud APIs, CI systems, and package managers—the cost of a single mistake rises dramatically. Even if individual vendors patch UI vulnerabilities and harden HITL dialogs, the ecosystem problem remains: agents are fed by shared public resources that can be manipulated, and human reviewers will always be a reliability variable.
That combination demands a shift in how organizations adopt developer automation: assume that agents will occasionally present manipulated context, treat all agent‑suggested operations as untrusted by default, and build automated, cryptographic, and human process layers to verify any high‑impact action.

---

Conclusion​

The Lies‑in‑the‑Loop disclosures are a timely and practical reminder that AI safety controls are socio‑technical: they depend on model behavior, UI design, developer workflows, and human psychology. The research demonstrates a credible RCE chain via forged HITL dialogs and highlights how supply‑chain risk can be introduced through everyday developer interactions. Defenses require coordinated changes—vendor UI hardening, strict privilege reduction, pipeline governance, and developer education. Until those controls are widely adopted, organizations must treat HITL as one control among many and design their tooling and processes so that an attacker cannot convert human trust into an execution token.
For Windows developers and operations teams, the immediate actions are clear: stop running agent outputs on production hosts, require explicit canonical confirmation of commands, sandbox tests, and harden CI gates and dependency policies. The future safety of automated development workflows depends not on trusting agents more but on designing systems that treat every agent suggestion as potentially adversarial until verified.

---

Source: Cyber Press https://cyberpress.org/lies-in-the-loop-attacks/