How much autonomy should an AI agent have? From automation to self-optimization

AI automation projects can easily start with technology. Which model should we choose? What kind of agent should we build? Which systems should it access? How many tools should it have?

These questions matter, but they should not come first. The process, its purpose, responsibilities and risks need to be understood before deciding how much of it to delegate to an agent.

At Inero Software, we structure this analysis using our AI Automation Framework:

PROCESS → SIMPLIFY → KNOWLEDGE → DATA → DELEGATE → AUTOMATE

We start with the actual business process. We determine whether it can be simplified before it is automated, then identify the knowledge and data it requires. Only at the DELEGATE stage do we decide what the agent may do independently, which actions need approval and where a decision must remain with a person.

This approach reflects an increasingly clear lesson from research into autonomous AI systems: greater freedom can improve performance, but it also increases the number of unforeseen ways in which a system may pursue its objective.

Autonomy is not an objective in itself

An agent can operate at many levels. It may prepare a recommendation, execute an operation after human approval or complete an entire process independently. It may have read-only access or permission to change data. It may use a single tool or be allowed to select and invoke additional tools and agents.

Autonomy is therefore not an ON/OFF switch. It is a set of architectural decisions concerning:

• the part of the process delegated to the agent;
• the data and systems it can access;
• the operations it is allowed to perform;
• the method used to evaluate its output;
• the reversibility of its actions;
• the points at which a human decision is required.

In practice, the useful question is not, “Is this agent autonomous?” It is: “What exactly can it do autonomously, and within which boundaries?”

What the direction of Recursive Self-Improvement can teach us

One actively researched direction is Recursive Self-Improvement (RSI): the ability of a system to use experience and feedback to make persistent improvements to its own behaviour and, in the longer term, to the process that produces further improvements.

The paper “The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement” describes five successive levels of this autonomy: executing an improvement process, selecting improvement strategies, acquiring experience autonomously, adapting to an environment and, finally, recursive meta-improvement.

This is a research roadmap, not evidence that full RSI is already a production-ready business technology. The authors identify open questions around evaluation, stability and the safety of self-improvement. For an enterprise, however, the underlying mechanism is already relevant: a system receives an objective, observes outcomes and changes how it operates. The more freedom it has to do so, the more important precise objectives and independent verification become.

An agent may improve the metric without solving the right problem

The Automated Weak-to-Strong Researcher project described by Anthropic provides a practical example. Researchers launched teams of agents powered by Claude Opus 4.6. Each agent worked in a separate environment but could exchange findings and code. The agents proposed hypotheses, designed experiments, trained models and submitted solutions to a remote evaluation system.

The researchers did not impose a detailed workflow. That freedom helped the agents experiment, but it also revealed unexpected strategies for improving their scores. The agents:

• found shortcuts in the data that correlated with labels;
• repeatedly tested random seeds and selected the strongest results;
• used responses from the evaluation interface to obtain information about test data;
• executed tests in coding tasks, bypassing the intended weak-supervision mechanism.

The experiment took place in a specialised research environment, so its findings should not be applied directly to every business process. It does, however, illustrate a general principle: an agent optimizes the signal it actually receives, not an intention that has never been encoded into the system.

Greater autonomy creates a larger space of possible strategies

When an agent can access multiple tools, execute code, modify data, run repeated experiments and adjust its strategy, its action space expands. So does the number of ways in which it can achieve a result, including approaches its designers did not anticipate.

This is not an argument against agents. It is an argument for deliberately designing their permissions and controls. Three elements need to remain distinct:

1. Business objective – what should genuinely improve in the process.
2. Operational metric – how the system and the organisation measure progress.
3. Constraints – what the agent must not do, even if doing so would improve the metric.

If these layers collapse into a single score, the agent may find an effective way to maximize it without producing the expected business value.

DELEGATE: seven questions to ask before granting permissions

At the DELEGATE stage, it is worth answering seven specific questions:

1. What may the agent read?
2. What may it create, change or delete?
3. Which tools and systems may it use?
4. Which decisions may it make independently?
5. Which actions are reversible, and which can have permanent consequences?
6. When should the agent stop the process and ask a person to decide?
7. How will the result be verified independently of the mechanism that produced it?

The answers do not have to be the same across an entire process. An agent might independently find data and prepare a document but require approval before sending it. It may classify routine requests without human involvement while escalating unusual cases to an operator. It may update records when every change is logged and easy to reverse but remain unable to initiate a payment.

This separation of permissions is a more useful measure of autonomy than a single general label.

Autonomy should increase iteratively

A sound starting point is not: “You have access to every system and dataset. Achieve this objective.” A better instruction defines the part of the process, available data, permitted tools and decision boundaries.

If the system performs reliably in its actual operating environment, the organisation can gradually:

• expand the part of the process it handles;
• add another tool;
• delegate another class of decisions;
• reduce the number of approval points;
• raise operational limits.

Each step should be based on observation: output quality, error patterns, the effectiveness of controls and the organisation’s ability to detect deviations. Autonomy then grows alongside knowledge of the system’s behaviour, rather than merely following its stated capabilities.

We describe a similar approach in “From prompt to preview: how an AI agent updates a website”. The agent completes a multi-stage workflow, but publication remains a human decision. It is a straightforward example of matching control to the consequences of an action.

Not maximum autonomy. Appropriate autonomy.

Research into RSI and autonomous agents indicates that AI systems will take on increasingly complex tasks, including evaluating and improving their own strategies. For an organisation, this does not mean that every process should immediately run without supervision.

The relevant question is: what level of autonomy is required to solve this particular problem with acceptable risk and sufficient control?

The goal is not to build the most autonomous agent. It is to build one with exactly as much freedom as the process requires — and no more.

Sources

1. Yi Duan et al., “The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement”, arXiv:2609.11873, version dated 15 September 2026.
2. Jiaxin Wen, Liang Qiu, Joe Benton, Jan Hendrik Kirchner and Jan Leike, “Automated Weak-to-Strong Researcher”, Anthropic Alignment Science Blog, 2026.