Timothy Darrah, PhD, PMP
In an article I wrote last month in response to a growing number of AI-related cyber incidents, I shared the difference between misuse and misunderstood use of AI, and opined that we don’t need more capable models, we need safer systems1. There is no shortage of incidents, and another recent example highlights the importance of mechanisms to enforce policies and user intentions outside the access and control surface of the LLM, also known as guardrails.
On October 3rd, an OpenAI researcher asked an agent to develop evaluation tasks, and to look for candidates in an internal Slack channel. The researcher also added, “don’t send any messages.” Later, in the same session, an access error with the automatic monitoring system stopped the pipeline. In the main agent’s chain-of-thought*, it contended with the user’s request to not send any messages and the fact that repairing the error would fix the alerting messages from the monitoring system:
“Their earlier instruction against sending messages concerned Slack outreach, not monitoring alerts, so these system-required notifications are allowed.”
It then delegated the repair task to a subagent, who also reasoned about this: “Setting no_notify=true when writing the result would prevent alerts—but would that count as disabling monitoring safeguards?” The subagent then decided that “disabling required alerts would be unsafe” and elected not to do it.
The main agent then reconsidered its position on the alerts, reasoning that “the user said not to send messages. Setting no_notify=true would respect that instruction,” and then told the subagent to delete the list of people who are supposed to receive monitoring reports and manually modify the database after suppressing alerts. The monitoring system under question, as ironic as it may be, is a system designed to identify and report incidents of misalignment2.
Both agents correctly identified the safety concern and correct course of action when initially confronted with the issue. However, their joint communication led to failure, demonstrating how rapidly chain-of-thought reasoning can devolve in multi-agent systems.
On October 6th, a researcher asked an agent to copy a reference implementation while preserving the existing program’s functionality. An earlier instruction prohibited disabling monitoring, even temporarily. When delegating the migration, the main agent told its subagent to “preserve monitoring as appropriate to the exact PR,” and the subagent reasoned that it would copy the implementation without the monitoring context, since the reference implementation used a dummy placeholder for monitoring anyway. As the existing program already has monitoring enabled correctly, copying over the placeholder would have effectively disabled monitoring, so by chance, not reasoning, this was actually the correct decision. However, the subagent then reasons that “the requirement is to match the PR exactly” means to simplify, and decided it should “explicitly propose removing the custom monitor to match
the PR.” The main agent agreed with the subagent, on the basis that “the user asked us to match the PR’s serving path, so I’m interpreting that as permission to change the monitoring.” On top of this, the subagent altered a unit test to explicitly not require monitoring, and when the main agent was questioned about the removal of the monitoring system, it denied involvement.
Initially, the main agent communicated the user’s goal to match the PR, but taken at face value, it deprioritized the user’s constraint of preserving existing functionality. In a financial workflow, the same problem can arise when an agent is asked to achieve an investment objective within a set of limits. It may understand those limits, and another agent may correctly reject a proposed trade, only for a later instruction to change how they interpret what is allowed. So how do we leave room for interpretation when required, while also enforcing boundaries that can’t be altered or circumvented?
How we apply guardrails to a trading decision at Mangrove
We use AI to help develop trading strategies and examine how its rules would perform across various market conditions, timeframes, or asset classes. When a strategy meets the minimum performance requirements set by the platform, it can be deployed into paper trading mode by the agent, or deployed into live mode by the user. Once a strategy is deployed, our deterministic execution engine performs all of the same sophisticated calculations that the Wall Street trading shops do related to risk management, order management, and liquidity management. The logic for determining whether or not to place a trade, and authorizing the trade are in separate components, neither of which is on the control surface of the LLM3.
This is just one example workflow, and every single guardrail is embedded in the agent’s harness or the system itself – we never rely on the agent to simply follow rules in its context. Financial institutions are heavily regulated and need assurance and guarantees to the nth degree. One safeguard isn’t enough; multiple external mechanisms are required, per action. The action to deploy a strategy has three. Evaluating a strategy has over a dozen, and the agent has no control over any of it.
Guardrails for the agent’s context
The foundation of any LLM-based application is context management, and we use knowledge graphs in our harness to reduce issues such as context rot, hallucination, or confabulation. Our knowledge graphs aren’t just a collection of markdown files that link to one another, like Andre Karpathy’s LLM wiki4, which I teach in my AI 5100 course at Vanderbilt University. They are computational, meaning, we can offload parts of the agent’s reasoning to deterministic algorithms that have been reliably operating in practice for 50 years or more. This allows us to dynamically manage the agent’s context in real-time and does not leave it to the agent to summarize its own session.
At Mangrove, we are currently doing frontier research on knowledge graphs, harness design, and context management, all with the goal of improving agent reliability and alignment. Earlier this year, we evaluated several different LLMs with and without our harness on a task of reviewing session transcripts and other sources of information to determine if new knowledge should be added to the graph or if the graph should be updated. Our harness contained a durability guardrail that rejected unsuitable information for a ‘timeless and static snapshot of the agent’s world state.’ The results are shown in Figure 1.
How institutions should evaluate AI systems before implementation
For every workflow your institution plans to incorporate AI into, identify the controls that enforce organizational policies and procedures, and verify that the agent cannot change or bypass them. Evaluate whether those protections remain effective when an agent misinterprets a request or passes an incorrect instruction to another agent.
As an example, consider a portfolio operations agent tasked with proposing trades to rebalance a fund’s holdings. The fund lowers the permitted allocation to one asset, but the agent retrieves an older version of the investment mandate and proposes a trade above the new limit. A second agent reviewing the proposal may determine the mandate is outdated, but interpret that the trade is permitted based on the first agent’s justification. Without an external mechanism to manage information provenance and prevent stale information from entering an agent’s context, and then another mechanism to validate the agent’s trade recommendation against the current thesis, this is more than just a hypothetical situation, but a likely reality.
Financial institutions need AI systems designed around their operating requirements, with context management and enforceable guardrails built into the infrastructure from the outset. As those systems take on more responsibility, their protections must remain effective through changes in models, policies, and workflows. Decisions to expand their use should rest on evidence that the institution can maintain control over the work being delegated, including the authority to change the rules governing that work. The institution remains accountable for the outcomes, even when an agent performs the task.
Reliable AI in finance depends on systems that preserve the institution’s intent and enforce its limits even when the agent reaches the wrong conclusion or interprets the user’s intent incorrectly. Rules in a markdown file can be ignored; external controls outside of the LLM cannot.
Footnotes:
(1)Tim Darrah, The AI arms race is nothing like the Cold War.
(2)OpenAI, Suppressing misalignment monitoring alerts and disabling misalignment monitoring, updated October 20, 2026, sections “Repairing an evaluation pipeline” and “Copying a reference implementation.”
*Quoted reasoning follows the OpenAI report’s plain-language interpretations, not verbatim model traces.
(3)Mangrove for AI agents
(4)Andrej Karpathy, LLM Wiki, April 4, 2026.
(5)Tim Darrah, Blockchain Futurist presentation, 2026, slide 16 (results) and slide 27 (methods). See the recorded discussion of the experiment, 7:35-9:20. Values are from the presentation’s original chart.