How to evaluate an AI agent skill: two procedures put to the test

A skill is a package of instructions and resources an AI agent can load for a task. Before adding one, it helps to identify the error it should prevent. An agent may already produce a good answer without the package; a useful procedure may also appear ineffective when assessed at the wrong stage.

This comparison examines two public procedures: grilling, used by Matt Pocock’s grill-me to clarify decisions, and part of Supabase Postgres Best Practices for reasoning about queries. The cases reveal a specific difference in an SQL validation plan, several equivalent outcomes, and a limitation in the criteria used to assess the interview. They provide a basis for choosing what to test in your own work before adopting a skill.

What the package adds

In the Agent Skills format, SKILL.md contains metadata and instructions. Its folder can also contain references, scripts and templates. A procedure can tell an agent what information to gather, how to order decisions or which checks to perform. The tools and access it needs depend on the task.

Instructions for reading an SQL plan, for example, do not automatically provide a database connection. Asking an agent to follow an interview procedure also does not establish that its client can discover and invoke the plugin distributing it. The guide to building and activating skills covers that step; this article examines answers produced after the instructions have been read.

The version of grilling examined here orders decisions by their dependencies. Each round asks what can be decided without assuming answers that have not yet been given, then waits before proceeding. The grill-me wrapper invokes this procedure.

For Supabase, the selection comprises the main file and three references: missing indexes, composite indexes and diagnosis with EXPLAIN ANALYZE. This is a subset of the collection, chosen for the cases being examined. Both repositories use the MIT licence; the links identify the revisions read, keeping later updates distinguishable.

The comparison and its scope

The method starts from the comparison with and without additional instructions described in the Agent Skills evaluation guide. Twelve initial responses were collected on 26 October 2026: six synthetic cases, each assigned to two separate agents. One condition received the case and a bounded task; the other also required the agent to read the procedure and selected references.

The sessions used the same nominal model, gpt-6-astra, with ultra reasoning, and started without the conversations from earlier trials. They still shared environment instructions and the skills catalogue. The exact model build was not exposed. The protocol, input files and criteria were fixed before the responses. The traces allow file reads and outputs to be checked, although the full text of the dispatched task messages is not inspectable in the audit.

Instructions were loaded manually, without installing the plugins. No database was queried: schemas, plans and SQL timings were invented inputs for reasoning exercises. For each case, one response was collected per condition, restricted to the first turn. The comparison describes those texts; it does not measure reliability, token savings, task duration or the quality of a complete interview.

A clear request may already be enough

The case with settled requirements described a weekly order report. Authorised recipients, aggregate-only data, the reporting period, document access, retention, retries and ownership had all been decided. The question was whether any product decision was still needed before technical design could begin.

Both responses recognised that design could proceed, without inventing a blocker or reopening approved decisions. The response using grilling explained that no decisions remained open; the response without the procedure reached the same outcome.

This case was chosen to see whether the interview would stop when it should. It did, and so did the baseline response. For a request this well defined, the result provides no reason to add the procedure. The remaining question is what happens when the problem is less fully specified.

Another case presented incompatible requirements: send named customer data as an attachment readable without an account, or show only aggregates inside an authenticated area. Both responses requested an authorised decision without inventing agreement. With the procedure, the first question concerned only who had authority to resolve the conflict. Without it, the response also asked which requirements would be confirmed. The difference is in the sequence. Counting questions does not establish which interview will reach a valid decision sooner.

SQL: a useful difference and two counterexamples

In the selective-query case, a query returned 100 rows out of a million, filtering by customer and status. The only available index was on the primary key. The synthetic plan showed a table scan. The scenario also included roughly 200 writes per second, making the maintenance cost of a new index relevant.

Both responses proposed a plausible candidate. The baseline chose an index on the customer column, restricted to orders with pending status; the response with the procedure chose a composite index on customer and status. These are alternatives to evaluate against the workload, rather than a right answer and a wrong one.

The more useful difference was in the validation plan. The response with the procedure explicitly called for a before-and-after comparison under representative conditions. The baseline asked for the plan, timings and write impact to be checked with the new index, but did not explicitly call for measuring performance before adding it. The evaluator therefore could not confirm that the before-and-after requirement had been met. This is a specific omission in the answer, not evidence that the skill consistently improves diagnosis.

The other two situations tested whether the agent would prescribe indexes prematurely:

The first counterexample matters because the Supabase reference examined contains the shortcut “Seq Scan on large tables = missing index”. That wording is too absolute: reading almost an entire table can make a sequential scan appropriate. The PostgreSQL EXPLAIN documentation shows how selectivity affects the plan and distinguishes estimated costs from actual timings.

The response using the procedure corrected the shortcut in light of the case. That response is not enough to assess its effect on other inputs: before reusing the skill, it is worth correcting wording that turns a clue into a diagnosis.

For an actual investigation, EXPLAIN ANALYZE executes the query, so access and execution conditions must suit the workload.

When the test judges the wrong turn

The incomplete-requirements case asked the agent to clarify an automated report without making decisions on the product manager’s behalf. The criteria prepared before the trials required questions about recipients, content and delivery in the first turn.

The baseline response included them. The response using grilling first asked about purpose, recipients and operational ownership, explicitly deferring content and delivery until later answers. Under the criteria, two requirements were missing.

That result should be retained, but reading the procedure changes its interpretation: deferring dependent decisions is exactly what grilling asks the agent to do. The test measures coverage in the first turn. It does not establish whether the interview would have forgotten those topics. Evaluating that would require continuing both conversations with consistent answers, then comparing unresolved decisions, introduced assumptions and premature questions.

What I would keep in the catalogue

The approach I propose is to connect each skill to a recurring error and a check that can be maintained. If an agent omits before-and-after comparisons, a procedure that makes them explicit deserves a trial on your own workload. If it already recognises insufficient evidence, adding instructions needs another concrete justification.

Before adoption, I would choose a real task, a reasonable baseline request and a verifiable outcome. I would add a case where the procedure should stop or take no action. The length of the trial should match the work being promised: an initial response may be enough to examine an initial diagnosis, while an interview also depends on what emerges in later turns.

I would then retain the skill revision, inputs and observed errors, with someone responsible for updating both instructions and cases. For procedures that include scripts or external services, tools, credentials and permissions need to be examined alongside the text. An outcome equivalent to the baseline request does not, by itself, justify another item to maintain. These twelve responses help identify the next check; adoption requires that check to address a problem in your own work.