Ori Eval

Ori Eval shows you which model is the best model for your project. It gives you scores, not opinions. Ask the question in usual words. Your coding agent then finds test material in your project, writes the eval, runs the models that you want to compare, and gives you the results. An eval is a *.eval.ts file. You can write the file yourself when you want more control, but you do not need to start there.

Start with your coding agent

The most simple method needs no commands from you. Give this instruction to your coding agent:

go to https://openrouter.ai/skills/spawn-ori-eval to get started

The spawn-ori-eval skill gives the agent all of the steps. The agent installs Ori, makes sure that you have a login, then starts ori code -p with your request. In the run, the create-eval skill of Ori asks the questions that shape the eval. It asks which part of your program to test, what that part must do well, if you have real data, and the maximum cost. Ori then writes the eval, runs the models, and gives you a recommendation.

The spawn-ori-eval skill does its work in a temporary directory. It does not put eval files in your project. Use the manual steps below if you want to keep the eval files in your project.

Why you run evals with Ori

The most difficult part of an eval is a stable test bench. Ori is a harness, and not only a skill. Ori resolves one harness and one model for a run, then holds them for all of the tests in that run. A prompt cannot change them. Two runs of the same eval files therefore use the same configuration. Ori sends its requests through OpenRouter. One comparison can thus include models from all of the large providers. A harness that is specific to one vendor can only test the models of that vendor.

Run the steps manually

The steps below do the same work by hand. Use them when you want more control, or when you want to know what the agent made.

Before you start

Install the CLI:

curl -fsSL https://openrouter.ai/labs/ori/install.sh | bash

ori login

An eval sends requests to a real model. Ori must therefore be signed in. The ori login command signs you in with your browser, then keeps the credential for later runs. Ori runs your eval files with Bun. If Bun is not installed, ori eval asks for your approval to install it. On a terminal that is not interactive, or when CI is true, ori eval does not ask. It stops and shows you how to install Bun.

Your project does not need to be a TypeScript project, and you do not need to install TypeScript. The eval file tests the agent, and not your code.

Start with a question

Go to your project directory, then ask Ori to make the eval:

cd my-project
ori code -p "What is the best model for my support agent?"
# For a long request:
ori code --prompt-file /tmp/ori-task.txt

The ori code -p command does not need a terminal. It writes its output to stdout, and it stops when the prompt is complete. Use --prompt-file for a long request. You cannot give -p and --prompt-file together, and Ori rejects a prompt that has no flag. You do not need to know eval methods or model names. The agent does these steps:

It looks for test material in your project. It looks for prompts, tool definitions, data files (.jsonl, .csv, chat logs) and known correct answers. Tests in other languages can also give test cases, but the eval file is always TypeScript.
It asks what is important. It asks a question, for example “accuracy first, or speed and cost?”, only when the answer changes the eval.
It writes the eval, then runs it. It makes evals/<feature>/<name>.eval.ts. It gets candidate models from the live catalog of OpenRouter, below your maximum price, then runs them.
It recommends one model. It gives you the scores, the times and the costs that support the recommendation.

The eval files stay in your project as usual code. Run them again when a provider releases a new model, make the criteria more strict, or run them in CI.

The eval file

Read this section when you want to write an eval yourself, or when you want to know what the agent made. An eval file looks like a usual bun test file:

evals/support/recommends.eval.ts

import { test } from 'bun:test';
import { setupAgent } from 'ori/eval';

const agent = setupAgent();

test('recommends restaurants using the search tool', async () => {
  const run = await agent.run('Where should I eat dinner in Lisbon?');

  run.tool('search').toBeCalled();
  run.tool('delete_file').toNotBeCalled();
  run.toComplete();
  run.toCostAtMost(0.01);
  run.toFinishWithin(30_000);
});

The setupAgent() function with no arguments gives you the harness and the model that your workspace resolves. This is the agent that you already run. Then use one of these commands:

ori eval
ori eval --report eval-report.md
ori eval --baseline best

Ori finds all of the *.eval.ts files below the current directory. It does not look in node_modules, .ori or .git. Ori then starts a temporary runtime and gives the files to bun test. The exit code of ori eval is the exit code of bun test. A failed eval therefore fails your CI job. The create-eval skill puts new eval files in top-level evals/<feature>/ directories. Ori can also find eval files in other directories. Use --baseline last|best|model:<slug> to compare a run with an earlier run. Ori keeps the earlier runs in .ori/eval/history.jsonl. A comparison is possible only between runs that included exactly the same eval files. Use --report <path> to write a Markdown report that you can give to other persons. When a model in a comparison stops before it answers, the report shows that model as unmeasured. The report does not remove the model, and it does not show a cost of zero.

Compare models

Get the candidate models from the live catalog of OpenRouter instead of a list of slugs that you write yourself. Set a maximum price, then run the same eval with each model:

evals/support/model.eval.ts

import { test } from 'bun:test';
import { candidateModels, setupAgent } from 'ori/eval';

const candidates = await candidateModels({
  limit: 5,
  maxPromptPrice: 0.000005,
});

for (const model of candidates) {
  test(`handles a refund request on ${model}`, async () => {
    const run = await setupAgent({ model }).run(
      'A customer wants a refund for order #1234. What do you do?',
    );
    run.tool('lookup_order').toBeCalled();
    run.toComplete();
  });
}

The candidateModels function can also select on maxCompletionPrice, minContextLength, the quality indexes, the necessary parameters, the input modalities, and excludeExpiring. An unknown value in the catalog therefore cannot pass a limit. Use assertModelIsLive(slug) when you name one specific model in a file. The eval then fails with a clear message if that model leaves the live catalog.

Give a score to an open answer

Some questions do not have one correct answer. Use an LLM judge to give a score to the answer. The setupJudge() function makes a separate agent on its own grading model. The score is therefore independent of the model that you test. You can also give your own agent to setupJudge() when you want a different judge.

evals/support/quality.eval.ts

import { test } from 'bun:test';
import { setupAgent, setupJudge } from 'ori/eval';

const agent = setupAgent();
const judge = setupJudge({ minScore: 0.8 });

test('gives an accurate, grounded answer', async () => {
  const run = await agent.run('What is our refund policy for digital goods?');
  await judge.autoEvals({
    criteria: 'Cites the 14-day window and does not invent exceptions.',
    run,
  });
});

The startingCriteria object has rubrics that you can edit for common dimensions: accuracy, completeness, instructionFollowing, safety, structuredOutput and toneAndVoice. Give one of them to judge.autoEvals as the criteria.

Use your own data

Real data from your users is better than prompts that you invent. Use test.each for chat logs, or for pairs of a question and an answer:

import supportPairs from './support-pairs.json';

test.each(supportPairs)(
  'answers: $question',
  async ({ question, mustMention }) => {
    const run = await agent.run(question);
    run.toMention(mustMention);
    run.toComplete();
  },
);

Run an eval in CI

An eval spends money on model requests. Put your evals in a separate job that a person starts, and not in your usual unit-test job. Give that job the credential from ori login, and keep it in a secret. A failed eval fails the build. An agent that becomes worse therefore stops the release, and does not reach your users. The ori eval --list --allow-no-key command needs no credential. Use it for a cheap test that your eval files are present.

Overview

Models & Routing

Features

Ori

Privacy

Best Practices

Community

Start with your coding agent

Why you run evals with Ori

Run the steps manually

Before you start

Start with a question

The eval file

Compare models

Give a score to an open answer

Use your own data

Run an eval in CI

​Start with your coding agent

​Why you run evals with Ori

​Run the steps manually

​Before you start

​Start with a question

​The eval file

​Compare models

​Give a score to an open answer

​Use your own data

​Run an eval in CI

Start with your coding agent

Why you run evals with Ori

Run the steps manually

Before you start

Start with a question

The eval file

Compare models

Give a score to an open answer

Use your own data

Run an eval in CI