Greyquill

Product engineering

Definition of done for an AI feature

Everyone says AI features ship with evaluation, guardrails and an audit trail. Here is what those three words have to mean before a customer sees it.

One page, the whole gate list.

Why the usual definition breaks

A normal definition of done assumes a feature that passes today will pass tomorrow. Three things break that assumption once a model is involved.

Three ways a model breaks the usual definition of done First, the same input can produce different output across runs. Second, behaviour can change without any code change when the model is updated. Third, failure looks like a plausible answer rather than an error, so nothing alerts. Same input, different output input right wrong right one green run proves less than it used to Changes without your code your code unchanged the model updated a provider-side update is a new feature, not maintenance Failure looks fine wrong answer no error no alert an exception gets fixed a plausible number gets shipped

The gates

An evaluation set existsReal inputs with known-correct outputs, written before the build finished, so it describes the requirement rather than the behaviour of what got built. Include the awkward cases your domain expert can name, and the cases where the right answer is to refuse.
Blocks
Graded on outputs, and the grader is calibratedIt asserts on what the feature produced, not that it produced something. And the grader has been proven to fail a wrong answer.
Blocks
Run more than once, spread recordedPass rate and consistency. A feature that alternates on the same input is not ready even when the average looks fine, and the average is what hides it.
Blocks
Every guardrail has a test that attempts the thingSee the distinction below. An untested guardrail is a hope.
Blocks
The failure path is built and exercisedWhat the user sees, whether work falls back to a person, whether partial output is worse than none. Test it by making the dependency fail.
Blocks
It leaves a recordInput by reference, model and version, the decision, any confidence, whether a person was involved. This cannot be added retrospectively for work already done.
Blocks
The ordinary bar still appliesTests on every commit, release gates that block regressions, accessibility, observability. Our own GST Co-Pilot ships with 162 tests and 88 percent coverage, and the AI parts did not get an exemption.
Blocks
Cost and latency measured at peakAt the volume it will actually see once live. Rarely stops a launch, often changes a plan.
Advisory
Refusal and escalation cases coveredBlocking wherever a wrong answer has consequences.
Advisory

Control or preference

A control

Enforced outside the model, in code around it. There is a test that attempts the thing and shows it refused.

A preference

Expressed only as an instruction to the model. It is a request, and it holds until it does not.

Know which of your guardrails are which, and never describe a preference to a customer as a control.

After launch

The loop after launch The evaluation set feeds the gates, the gates lead to release, the release is monitored in production, production failures are promoted back into the evaluation set, and the cycle repeats. A separate trigger, a model version change, re-enters the gates directly. Evaluation set the baseline The gates Release In production retries, overrides, abandonment move before the metrics do every production failure becomes a case, so the same fault cannot regress twice model version change

Re-run the set on a schedule, and always before a model version changes. Keep the baseline: a case that used to pass and now fails is a regression whatever the average says.

Related

The verification machinery behind this, including calibrating a grader and reading a distribution, is in Harness and verification.

Greyquill
greyquill.io/dc/services/ai-feature-done.html
Greyquill Software

Definition of done, AI feature

What has to be true before a customer sees it.

An evaluation set existsWritten before the build finished. Includes the awkward cases and the ones it should decline.
Blocks
Graded on outputs, grader calibratedProven to fail a wrong answer. An untested grader produces a number people trust.
Blocks
Run more than once, spread recordedPass rate and consistency. The average hides a feature that alternates.
Blocks
Every guardrail has a test that attempts the thingEnforced in code is a control. Stated in a prompt is a preference.
Blocks
Failure path built and exercisedIt gets used first during an incident.
Blocks
It leaves a recordInput by reference, model and version, decision, confidence, human involvement. Cannot be added later.
Blocks
The ordinary bar still appliesTests on every commit, release gates, accessibility, observability. No exemption for the AI parts.
Blocks
Cost and latency at peakRarely stops a launch, often changes a plan.
Advisory
Refusal and escalation coveredBlocking wherever a wrong answer has consequences.
Advisory

Why the usual definition breaks

  • Same input, different output across runs.
  • Behaviour changes without your code changing.
  • Failure looks like a plausible answer, so nothing alerts.

After launch

  • Re-run the set on a schedule, always before a model version changes.
  • Keep the baseline. A case that used to pass and now fails is a regression.
  • Watch retries, overrides and abandonment. They move first.
  • Promote every production failure into the set.

Greyquill Software builds products with AI in them and the discipline around it, including our own. greyquill.io