How to evaluate code generated by AI agents: from uncertainty to a method

An agent can modify a dozen files, add tests and prepare a pull request in a single task. The challenge is no longer the speed of producing code, but the ability to assess the change reliably before it reaches production. At Inero, we address this by reviewing the whole branch and matching the controls to the consequences of a potential defect.

Speed alone has little value, however, if it produces AI slop: a large volume of plausible code that works in a simple scenario but fails to account for system context, business rules and the consequences of defects.

At Inero, we address this neither by rejecting AI nor by burdening every change with the same heavy process. We combine AI agent capabilities with risk-based engineering quality controls. AI shortens delivery time, while specialist attention is directed to the areas where it matters most.

AI productivity needs a different quality model

Human-written code also contains defects. We therefore do not treat code origin as the criterion for a production release. We establish whether the change has been verified in proportion to what can happen if it is wrong.

This distinction lets us use the advantage of AI without creating an artificial bottleneck. A copy, styling or filter change can move quickly when the build and tests pass, the diff is readable and the deployment is easy to reverse. A data migration, authorisation rule, pricing calculation or automated communication needs a different level of control. We discuss the same separation between rapid prototyping and production readiness in Your AI demo works. Now what?.

Our reference point is not the number of generated lines or even the number of system users. It is the consequence of failure and whether that consequence can be reversed.

AI reduces the cost of a fix, not the full cost of a bug

An agent can often repair a defect as quickly as it prepared the first version. Implementing the correction is only one part of the cost.

Somebody must first notice the problem and identify its cause. In the meantime, it may affect users, data or external systems. After the repair, the team may still need to restore data, correct documents or explain the situation to a client. AI shortens implementation and helps with diagnosis, but it does not undo damage that has already occurred.

We therefore use several practical questions to determine the required controls:

• can the effect of a defect be reversed completely;
• does the system write, delete or transmit data;
• does the change affect money, permissions or commitments;
• how quickly will the team detect an anomaly;
• do we have a working rollback and a tested restore procedure.

A dozen people making operational decisions based on a system may require more care than thousands of users of a simple informational application. Traffic volume matters, but it does not replace an assessment of actual impact.

We treat the agent as a capable author without complete context

Code from a language model is often tidy, well named and supported by confident comments. This helps developers, but it can also lower a reviewer’s guard. The most important problems rarely look like obvious syntax errors.

An agent may follow a request too literally and miss a business rule that was never written down. It may add a silent fallback that hides an error, or produce tests that repeat the same flawed assumption as the implementation. It may expand the task unnecessarily or refer to a configuration option or package that does not exist.

We therefore evaluate more than whether the code looks correct. We check whether it solves the right problem, preserves existing rules, fits the architecture and exposes failures instead of masking them. The agent accelerates execution, but accountability for the result stays with the team.

Review covers the whole branch

We assess an agent’s work as one change set, from the common base to the tip of the branch. Looking only at the latest commit may hide an earlier decision that was later half-reverted, a weakened test or new configuration without a safe default.

Our process includes:

1. defining the complete scope of the change from the branch’s common base;
2. reading the commit history and the intent of the task;
3. examining every changed file in its wider context, not only the diff hunks;
4. checking correctness, edge cases, security and error handling;
5. identifying missing tests, migrations, defaults or monitoring;
6. classifying each area by risk and recording concrete manual checks.

A small, readable scope is part of quality assurance. When a branch is too large to understand properly, we split the change instead of accepting a review that exists only as a formality.

A repository, separate branch, pull request and preview before release also provide the foundation for agents that perform tasks across several tools. From prompt to preview: how an AI agent updates a website shows this workflow in practice.

Four control levels instead of one heavy procedure

We classify a change by the consequence of failure, not by the file extension or technology involved. Supporting code that feeds a high-risk area inherits that risk. When one file touches several levels, we apply the strictest one.

This creates a fast lane for reversible changes and a more cautious one for operations whose effects may remain after a fix. We retain the productivity of AI for routine work without extending the same freedom to data, access or money.

Automated gates are the foundation, not a substitute for review

Every change goes through the available project checks: build, type checking, linting, tests, security scanning and dependency checks. These controls are fast, repeatable and unaffected by the confident style of generated code.

For important areas, we add scenarios that originate outside the reasoning of the agent that wrote the implementation. The same agent can build code and tests around one mistaken assumption. We verify expected behaviour against the business process, representative data and known exceptions.

We also use agents as an additional review layer. A separate pass can inspect the whole branch quickly, identify contradictions and prepare a list of items for manual verification. AI review does not replace domain expertise, but it increases coverage and allows specialists to focus on decisions that cannot be reduced to an automated rule.

The review result has to support a decision

A general statement that the code “looks good” is not sufficient. We separate the review output into two parts.

The first is findings: the code location, an explanation of the defect and a concrete scenario in which it fails. This helps the team distinguish an actual issue from a reviewer’s stylistic preference.

The second is items for human verification. Every high-risk area belongs here, even when an automated reviewer has not found a defect. Each item identifies:

• the file, section or commit;
• the reason for the risk and the possible irreversible effect;
• one concrete check for a person to perform.

That may mean running a migration against a copy of production data and comparing row counts, recalculating an invoice independently or confirming that a user from another tenant cannot access the record. The review ends with a clear decision: ready, ready after the listed checks or requiring changes.

A person remains accountable for the result

The person approving a change does not transfer responsibility to the model or tool provider. High-risk areas are assessed by a reviewer who understands the domain, and an agent does not have an independent path to production.

This follows the principle of controlled autonomy. An agent may prepare a change, run checks and describe the risk, while the decision to perform an irreversible action remains with a person. We discuss this model in more detail in How much autonomy should an AI agent have?.

More code is not our objective

At Inero, the purpose of AI is not to maximise the number of commits or replace engineering judgement. It is to move faster from a well-defined problem to a working, maintainable solution.

We therefore assess the value of agents on more than code generation time. Readability, alignment with the business process, testability, release safety and the ongoing cost of maintenance also matter.

Our rule is simple: we review according to the consequences of failure, not the author of the code. This makes AI part of an efficient engineering process rather than a source of rapidly growing debt hidden behind plausible code.