ds-eval

The problem

Case component-button-001 asks for a toolbar: Save, Cancel, Delete. The answer that knows the system uses the component and its variants:

import { Button } from "@ds";

export function Task() {
  return (
    <div className="ds-row">
      <Button>Save</Button>
      <Button variant="secondary">Cancel</Button>
      <Button variant="danger">Delete</Button>
    </div>
  );
}

The naive answer keeps the same three labels and draws the controls itself:

export function Task() {
  return (
    <div>
      <button style={{ background: "#3B82F6", color: "#fff", padding: "12px 16px" }}>Save</button>
      <button style={{ background: "#6B7280", color: "#fff", padding: "12px 16px" }}>Cancel</button>
      <button style={{ background: "#EF4444", color: "#fff", padding: "12px 16px" }}>Delete</button>
    </div>
  );
}

Both files are buttons. On a screenshot the difference is easy to miss. For the system they are different interfaces. adversarial-color-001 asks for a button in #8B5CF6 and forbids a raw color: here plausibility is the violation.

ds-eval is for catching that kind of violation and comparing two answers when the task, the system, and the measured property stay the same. That is not a claim that a team started bypassing the system less often. That observation does not exist.

What the published run shows

The smoke suite is 10 cases and costs $0. The recorded models are fixture and fixture-naive: prepared files, not a call to Claude or Codex. The tasks and the demo system stay fixed. The answer pack changes.

The gap in the aggregate comes almost entirely from the design-system check. The published figures were produced by the old checker: it did not recognize the Button import that is in the file. So 87.8 and 59.3 are the archived record of that specific pre-fix run, not a final score of the two interfaces.

The code axis barely moves, 98 against 99. The naive buttons pass it. The check looks for any, file length, and export function Task, not for whose button it is.

The import check deducts for a line the file already has

• The fileimport { Button } from "@ds", and three Button variants.
• The run’s regular expressionIt expects from "@ds" import. An ordinary TypeScript import does not match. The note says “no @ds imports.”
• The archived scoreMinus 25. Static score 75. The Button tag is found separately, and compliance scores 100. The DS axis for the case is the average, 87.5. Across the suite it is 87.

The old checker does find differences between the files: the naive one has no Button, uses native button elements, and the colors #3B82F6, #6B7280, and #EF4444. Still, the archived gap of 87 against 5.7 cannot be treated as a clean design-system score because the existing import in the good file was missed.

In the current runner the import check has moved to Babel (grade_ast) and is covered in tests/test_graders.py. The published JSON record in runs/fixture-good is deliberately kept as-is from the pre-fix run.

Why this is not one judge

Deterministic checks cover concrete violations in the source. The browser and a model judge cover properties that one reading of the code cannot establish. If a check did not run, its result is not a measurement.

I do not hand an import, a hex value, or a native tag to a model. A rule answers the same way on the same file, including when it is wrong. In the published run the model judges did not run, skipped: true. Their reproducibility was not evaluated on this run.

In the current code a judge turns on only as a pair, an Anthropic or OpenAI provider and a screenshot. The visual judge scores layout, hierarchy, and rhythm from 0 to 5. The UX judge asks whether the task is done. The kind field separates deterministic from llm. A zero marked skipped is not a verdict on the screen. Their 30% of the weight moves onto the other axes, so the aggregate does not collapse, and it still contains no judgment of how the screen looks.

What the results can support

On these ten files, the source check separates a Button tag from a native button with a hardcoded color. That comparison holds because the task and the system are the same, and the property is one the check actually read. The filled-in 70s cannot be compared: there is no measured property there.

The gap cannot be carried over to a live model: there is no such run. It cannot support a claim that a team started bypassing the system less often. The agent-ui page draws the same line: how much the rules raise correct use has not been measured with ds-eval.

The tools next to it

ds-context builds a short pack and does not call a model. ui-repair writes a patch from system rules. prompt-regress checks whether a new prompt broke a case that used to pass. They do not call ds-eval. ds-eval scores a file that has already been written, and only with the checks that actually ran.

Run it

cd ds-eval
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

ds-eval run --model fixture --suite smoke
ds-eval run --model fixture-naive --suite smoke
ds-eval compare RUN_FIXTURE RUN_NAIVE

Live models read ANTHROPIC_API_KEY and OPENAI_API_KEY from the environment. --runtime turns on the browser: the build, Playwright, and axe. It does not start the judges; those still need a live model and a screenshot. Without the flag the check does not start, and the current runner records a functional score of 0 marked runtime: skipped. That is a missing run, not a result. If the browser starts and the build or the page fails, the record keeps the execution error, and the functional zero no longer carries that mark. The filled-in 70 in the published run is a third state: the browser never started, and the old checker wrote a constant.