---
title: "AI agents in a team: evaluate quality, costs and responsibility"
source: "https://www.matteovincenti.com/en/blog/ai-agents-team-quality-costs-responsibility"
language: en
author: "Matteo Vincenti"
published: 2026-10-20
updated: 2026-10-23
category: "AI"
tags: ["AI agents", "Team management", "Software quality"]
---

# AI agents in a team: evaluate quality, costs and responsibility

Evaluate AI agents at work: quality, complete-task cost, ownership and criteria for adopting, limiting or stopping a pilot.

![A hand checks a green component with a gauge, separating unsuitable pieces](https://cdn.sanity.io/images/xrqkwry7/production/685b94fcdf5f785865ce72daed59365baa1c4f8a-1728x910.png?w=1400&fit=max&auto=format)

Category: AI · Tags: AI agents, Team management, Software quality

An agent that prepares a change in eight minutes can leave a team with more work than one that takes fifteen. Someone still has to read the code, check its behaviour, fix errors and decide whether to merge it.

To evaluate AI agents in a team, I would start with that cost: what does it take to deliver a change we can accept? And how much checking can we sustain while agents keep producing work?

Here, a development agent can read a project, edit files and use tools such as tests and a terminal. The workflow also includes instructions, automated checks and people's work. That whole process belongs in the comparison with the way the team works today.

## Decide what to accept before starting the clock

Take a bounded task: add a status filter to an API that returns orders. The API already exists and an interface calls it. We want to search for paid orders while preserving the existing behaviour when no filter is supplied.

For this example, the contract accepts `paid`, `pending` and `cancelled`; any other status produces HTTP 400. Orders are returned by descending ID, with pagination applied to the orders selected by the filter.

A small dataset makes the error visible: order 42 is paid, 41 is pending, 40 is paid and 39 is cancelled.

_Decide what to accept before starting the clock_

| Requirement | Request or check | Expected result |
| --- | --- | --- |
| Preserve existing behaviour | No filter, first page of two items | Orders 42 and 41 |
| Filter before paginating | Status `paid`, first page of two items | Orders 42 and 40 |
| Handle invalid input | Status `unknown` | HTTP 400, without silently ignoring the filter |
| Keep the caller compatible | Open the list in the existing interface and use the new filter | The initial list works and the filtered list shows the expected results |

If the implementation takes the two most recent orders before filtering, it returns only 42. The filter appears to work, but the page is incomplete: order 40 was excluded too early. Checking this requires a case that exercises filtering and pagination together. A test of whether the API recognises `paid` would leave the problem untouched.

I would ask for a pull request, the proposed code change, accompanied by the checks run on the submitted revision and a check of its caller. The person accepting the work needs to reconstruct the table's results. An agent's summary saying “everything works” still leaves that work to the reviewer.

There is a concrete precedent for this distinction. In a 2025 METR evaluation, four of fifteen sampled pull requests passed reference tests, yet manual review found none of the fifteen mergeable as submitted. These were tasks in two mature repositories, tackled by an agent using Claude 3.7: a bounded result that helps explain why passing tests does not settle acceptance. [METR, evaluating changes through mergeability](https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/).

## Eight minutes can become thirty-four

Compare two workflows. The case uses constructed data to show how to make the decision; both workflows reach the same acceptable result.

_Eight minutes can become thirty-four_

| Workflow | Preparing the first submission | Subsequent checking and corrections | Time through acceptance |
| --- | --- | --- | --- |
| A | 8 minutes | 26 minutes | 34 minutes |
| B | 15 minutes | 10 minutes | 25 minutes |

The phases run consecutively, so the totals represent elapsed time. B takes longer to submit its first version and reaches acceptance sooner. But one input is still missing from the decision: how many of those minutes required a person?

Ten minutes of automated checks can leave the reviewer free to do other work; ten minutes spent fixing a query require that person's attention throughout. If two people each work for ten minutes during the same interval, that is twenty minutes of human effort and ten minutes elapsed. Mixing the measures makes the benefit of parallel work hard to interpret.

The wait before review belongs in the record, too: a ready pull request can sit idle because nobody has time to read it. If that queue grows, adding agents shifts the bottleneck towards the people who accept their work.

## What productivity studies actually tell us

In METR's July 2025 randomised study, sixteen experienced developers worked on 246 tasks in their own open-source projects. Using early-2025 AI tools took them 19% longer, despite their perception of a saving. That is a good reason to measure the complete task rather than rely on a feeling of speed. [METR, study of experienced developers](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/).

Applying that 19% to any team today would be just as hasty. In its February 2026 update, METR explains why newer data are difficult to interpret: participation changes, developers select different tasks, and parallel agents complicate time accounting. A later survey reports perceived benefits while distinguishing speed from the value of work produced; these are self-reports, not a new causal estimate from the same experiment. [METR update](https://metr.org/blog/2026-02-24-uplift-update/), [METR survey on AI's value](https://metr.org/blog/2026-05-11-ai-usage-survey/).

The useful pilot question is: for which tasks, with which checks and how much human intervention do we gain something? The comparison needs a recognisable task family, such as compatible changes to existing APIs. Averaging those tasks together with wording changes and migrations hides substantial differences in the work.

## Use an agent to help with checking, too

Restricting agents to writing code narrows the comparison considerably. They can also help find regressions, prepare tests or inspect a change before a person receives it.

Salvatore Sanfilippo, antirez, describes a QA approach that starts from commits since the previous release and looks for integration problems and regressions. What interests me is the choice of material: the changes introduced in a release give checking a concrete scope. [antirez on agents and software testing](https://antirez.com/news/168).

For the order filter, I would give an agent a specific question: “Is pagination applied to the filtered set? Show the request's path and a case that distinguishes the two implementations.” The second agent would receive the contract and revision to inspect, without starting from the first agent's reassurance.

Its report also needs assessment. Finding the missing order 40 helps; producing five unfounded warnings gives someone five more things to inspect. Uber addresses this problem in uReview through filtering and deduplication, developer feedback and evaluations against known defects. Counting comments rated useful answers a different question from counting missed defects. [Uber Engineering, uReview](https://www.uber.com/us/en/blog/ureview/).

Before adding that step to the workflow, I would try a change containing the pagination defect and a correct change. The case designer keeps an answer key with each expected outcome and the reference fix for the faulty change. The review agent receives the contract and candidate code without that answer key. For each case, I would then compare decisions made by people reviewing with and without the agent's report. Each person sees only one condition and has not seen the reference answer. The person who prepared the cases can check defects found, false alarms and human decision time. This initial trial helps decide whether to investigate the check further.

## More agents need a problem worth dividing

My first choice for the filter is one implementation agent and the checks already available. A second assignment makes sense when it covers a specific risk, such as the interaction between the query and pagination. Opening several sessions to rewrite the same endpoint leaves the coordinator to select and combine their solutions.

Anthropic's guidance distinguishes workflows, whose steps are defined in code, from agents that choose dynamically how to proceed, and recommends starting with the simplest suitable approach. Before delegating a stable check to another model, I would consider adding it to the test suite. [Anthropic, Building effective agents](https://www.anthropic.com/research/building-effective-agents).

Coordination belongs in the cost: preparing each agent's context, reading duplicate reports, resolving disagreements and checking the combined change are workflow activities. Even a useful division of work needs someone to accept the overall result.

## Who may change, approve and release

A promise in a prompt does not technically restrict the operations available.

If an agent's job is to propose code, I would give it the access that job requires. Review, merge and release need identifiable owners and appropriate permissions. One person can hold several roles; the decision they are taking should remain clear.

On GitHub, for example, branch protection can require reviews and checks before merging; bypass options also need consideration. In Actions workflows, token permissions can be restricted at workflow or job level. These are constraints to configure, while the prompt gives instructions for working within them. [Branch protection](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches), [GITHUB\_TOKEN permissions](https://docs.github.com/en/actions/tutorials/authenticate-with-github_token).

For our filter, the reviewer checks the contract and callers; the person authorising release also considers the environment receiving the change. A review agent can give an assessment without production credentials.

## A completed record and a decision

The pilot starts with the current process as its baseline. Agree on the task family, essential requirements, consumption limits, human-time budget, desired benefit and decision date before it begins. Assign comparable tasks to workflows before seeing their results, too: repeating a problem after reading its solution favours the second attempt.

For the compatible filter changes in our example, set this objective: reduce time through acceptance while staying within 15 minutes of human work and EUR 1.20 in tool spending per accepted change. All four requirements in the opening table are mandatory. A faulty page does not become acceptable because it costs less. The time and spending thresholds govern adoption, after accounting for the complete work.

Return to A, the current process, and B, the candidate, and fill in the missing data behind the 34 and 25 minutes. The record follows one change per workflow through acceptance, including every submission, correction and coordination step needed. Record the pilot's initial setup separately.

_A completed record and a decision_

| Measure | A — current process | B — candidate |
| --- | --- | --- |
| Final requirements for the filter and its caller | All met | All met |
| Accepted changes | 1 | 1 |
| Time through acceptance | 34 minutes | 25 minutes |
| Wait before review, already included in elapsed time | 0 minutes | 0 minutes |
| Preparation: human work | 4 minutes | 6 minutes |
| Coordination: human work | 2 minutes | 4 minutes |
| Review: human work | 5 minutes | 6 minutes |
| Corrections: human work | 3 minutes | 2 minutes |
| Total human work | 14 minutes | 18 minutes |
| Tool spending, including every submission | EUR 0.40 | EUR 1.10 |
| Outcome against the agreed limits | Within both budgets | Exceeds the human-time budget by 3 minutes |

I would keep A for this task family. B arrives nine minutes sooner but requires four more human minutes; the limit was fifteen. The next trial should focus on B's preparation and coordination: those two steps account for the additional work. Averaging speed and cost into a score would hide the reason for the decision.

The choice changes when the problem changes. With a hard deadline of 30 minutes, a human-time budget of 20 and the same EUR 1.20 spending limit, B meets the constraints and A arrives too late. I would choose B for that delivery. Extending it to the task family needs more comparable tasks, including less favourable cases.

Keep the record in the system the team already uses to track work. In an actual trial, link the numbers to the requirement, initial and final code revisions, model and tool versions, instructions, checks performed and owners of acceptance and release. Another person can then see which work the comparison describes.

These records give two separate measures: human minutes per accepted change and tool spending per accepted change. For each measure, include all attempts in the chosen task family and period, including rejected and stopped work, and divide by the number of accepted changes. To calculate a total cost in euros, first value human work at a stated hourly rate using the same rule for both workflows, then add tool spending. Keep setup separate and make any allocation rule explicit. With no acceptances, the ratios cannot be calculated; incurred costs and outcomes remain visible. Fixing the task family and unit of work beforehand prevents artificial splitting from improving the average.

If the candidate works for compatible changes but needs too much checking when a public contract changes, I would restrict it to the former. The pilot also needs execution limits that stop an attempt. An out-of-scope action or reaching one of those limits stops the work; an essential open check puts acceptance on hold. Keep these outcomes in the record, too.

Keep some cases out of the work used to refine the instructions, so they show what happens on problems not already used to correct them. For detailed token, cache and attempt accounting, see the [Claude and Codex consumption experiments](https://www.matteovincenti.com/en/blog/reduce-claude-codex-tokens-experiments).

If output grows while review piles up, I would invest in freeing that capacity first.