The Good Regulator: What an AI Agent Must Know Before It Can Act

Before an AI agent can make a good decision, it needs accurate information about the current situation. That information comes from its task, documents, memory, tools, and observed system state. If any of it is stale, incomplete, or contradictory, even a capable model can take the wrong action and explain it convincingly.

This appears in ordinary failures. An agent quotes an archived policy because retrieval ranked it highly. It edits the correct repository with an outdated acceptance criterion. It sees that a command exited successfully and concludes that the customer outcome succeeded. In each case, the action follows the internal model it was given.

Roger Conant and W. Ross Ashby formalized this relationship in Every Good Regulator of a System Must Be a Model of That System. A regulator that works well must embody a model of what it regulates. Applied to AI operations, the theorem makes context architecture, policy versions, and observability part of control rather than prompt decoration.

What does the good regulator theorem mean for AI agents?

The good regulator theorem says effective regulation depends on an internal model that captures the distinctions relevant to the desired outcome.

The model need not reproduce the whole company. It needs the parts that can change the decision. A release agent may need the repository state, target environment, approved scope, deployment policy, rollback procedure, and current authority. It does not need every sales call or design draft in the workspace.

A useful internal model answers five questions:

Question	Example
What state is the operation in?	The release candidate passed tests but awaits client approval
What condition should be maintained?	Only approved, verified builds reach production
Which policy applies now?	Production changes require reviewer approval and a rollback path
Which actions are available and authorized?	Inspect, test, propose, deploy after approval
What evidence will establish the result?	Deployment record, health check, and reviewer sign-off

Missing any one of these can make the agent's reasoning locally coherent and operationally wrong.

Why is more context not a better model?

A larger context window can hold more material. It cannot decide which material is current, authorized, or relevant.

Dumping documents into a prompt creates several problems:

• old and current policies appear with similar authority
• sensitive information crosses task boundaries
• irrelevant detail competes with the state that changes the decision
• the prompt records document fragments without recording why they were selected
• long documents are reduced to arbitrary prefixes that may omit the relevant passage

A safer pattern separates discovery from deliberate retrieval. The system can identify permission-filtered Library paths that may be relevant, then let the agent fetch an exact document only when the task requires it. The prompt carries guidance, not document bodies. Internal metadata retains the selected version, path, score, and source so the context decision remains auditable.

This does not make retrieval correct. It narrows accidental disclosure and forces the agent to choose which source to inspect.

Why must policy and structure be versioned?

A mutable instruction answers “what is true now” but not “what governed this execution.” Reliable review needs both.

Suppose a workflow starts under a policy requiring a finance approval for credits above EUR 500. The policy changes while the run is waiting on another step. Re-resolving the current policy midway can produce an execution that never corresponded to one approved configuration.

Immutable versions let the run snapshot its relevant configuration before action. Stable identities still connect relationships across the organization, while exact version identifiers explain the execution later.

Without versions	With immutable versions
A policy changes in place	A new policy version is published
A run re-reads current configuration	A run uses its snapshotted version
History shows the latest instructions	History identifies the instructions that governed the run
Rollback erases the intervening state	Rollback publishes another version and preserves evidence

Versioning does not guarantee a good policy. It makes the policy inspectable and replaceable.

How should memory enter the model?

Memory should carry durable observations that are relevant to future work. It should not become an unbounded transcript archive.

Facts, decisions, preferences, and experience have different lifetimes. A customer preference may remain useful for months. A temporary incident status may expire in hours. A rejected proposal is evidence about a decision, not a standing instruction.

A bounded memory system should preserve provenance, scope, and supersession. The agent needs to know where a memory came from, which work it applies to, and whether a newer decision replaced it. Without those boundaries, remembering more can reduce control.

Why is a successful process exit insufficient evidence?

A process exit status measures one narrow condition: whether the process reported success. It does not establish that the intended operational result occurred.

A command can exit zero after generating the wrong file. A deployment command can finish while the service remains unhealthy. A document can be selected by semantic retrieval while lacking the answer the agent needs. A model can declare a task complete while missing the acceptance criteria.

Evidence should match the controlled variable:

Claim	Evidence that fits	Evidence that does not establish it
Code meets stated behavior	Tests plus review against acceptance criteria	Agent says implementation is complete
Release serves healthy traffic	Deployment record and health observation	Deployment command exited zero
Client report is acceptable	Source checks and attributable client or reviewer judgment	Document exists
Policy was followed	Snapshotted policy version and action history	Current policy page

Observability therefore belongs in the regulator's model. The system needs to see what action was attempted, which versions and permissions governed it, and what happened afterward.

How should uncertainty change action?

An internal model is always incomplete. Good regulation requires recognizing when the missing part matters.

An agent should be able to distinguish:

• known state backed by a current source
• inferred state with stated evidence
• conflicting sources
• unavailable state
• a decision beyond its authority

These distinctions should alter behavior. Missing nonessential context may allow work to continue. Conflicting policy should stop an irreversible action. Unavailable evidence should prevent a success claim. The right response may be a question, proposal, or escalation rather than another generation attempt.

No model can prove that it contains every relevant fact. Human review does not remove that uncertainty either. It adds a different model, different experience, and legitimate authority at selected boundaries.

Why must the model remain replaceable?

Providers, models, prompts, tools, and retrieval methods change. An operation tied to one model's undocumented behavior cannot be reviewed or improved independently.

Keep the organizational model outside the model provider where possible:

• persist policy and configuration as versioned records
• expose tools through explicit contracts
• record selected context and execution evidence
• define acceptance criteria independently of the generated answer
• evaluate replacements against the same bounded operation

Model replacement will still change behavior. The goal is to make that change observable and testable rather than pretending models are interchangeable.

How does Task Machine provide an operational model?

Task Machine assembles an agent's work from scoped tasks, current and snapshotted resource versions, authorized Library paths, skills, connectors, permissions, budgets, and workflow state. Worker and workflow runs retain configuration and evidence so later review can reconstruct which model of the operation governed the action.

That representation remains partial. A team still chooses what enters the Library, which policies matter, and which evidence is sufficient. Task Machine supplies boundaries and history, not omniscience.

The complete cybernetics for AI operations series

Continue with the viable AI-native company to place individual regulators inside an organization that can coordinate, audit, adapt, and preserve policy.