OY1 evaluation.
Evidence and scope.

01 / PUBLISHED API RUN

100% across 25 public games.

Our completed 9 September 2026 run used GPT-6 Astra with high reasoning and Standard service. It scored 100% across 25 public games and 183 levels for $415.37 in successful-run inference costs.

Model

GPT-6 Astra · high reasoning

Total tokens

153,058,391, including compaction

Environment actions

6,732

API operations

1,773, including 45 compactions

Source

Commit 0c7848d · Apache-2.0

View the API-run scorecard ↗ · Published run totals ↗

The public games were used during development. Earlier attempts and development costs are excluded from $415.37 and disclosed in the source repository. This is a published public-set result, not evidence of unseen-game performance, general intelligence or ARC Prize verification. Community leaderboard submission and disclosures ↗

02 / TOKEN COMPARISON

83% fewer reported total tokens.

We compared OY1’s published total with the Provider Adapter’s published High public-game replays, using the same 25 game versions and GPT-6 Astra High in both runs.

Provider Adapter

901,970,049 total tokens

OY1 API harness

153,058,391 total tokens

Reduction

83.03%, rounded to 83% on the homepage

Formula: 1 − 153,058,391 / 901,970,049. Provider totals sum input plus output usage in 7,078 replay events. Cached inputs are included within input tokens; reasoning tokens are included within output tokens, so neither is added twice. OY1’s aggregate includes compaction.

This is a comparison of published runs, not a controlled rerun or a component ablation. Repeated and cached context contributes to the totals: 83% fewer tokens does not mean 83% less reasoning, compute, or cost. It is not a comparison with the separate semi-private evaluation. Different prompts, implementations and development histories can affect the result.

Provider Adapter results and public replays ↗ · OY1 per-game results ↗

03 / METHOD

Keep observations. Check predictions.

The API harness retains visual history and per-game memory. It asks for predictions before action batches, compares them with observed outcomes, and stops a batch when a prediction fails or the game changes state. Compaction keeps the conversation manageable while retained observations remain available to inspect.

The open reproduction guide identifies the evaluated code, dependencies and run procedure. It also provides a synthetic fixture that makes no paid model calls. Re-running the benchmark requires API access and incurs inference costs.

04 / THE HOSTED SERVICE

What MCP validation covers.

The hosted OY1 MCP adapter has its own task controller and versioning. It uses the original memory and empirical planner, while the connected client controls reasoning, model calls and external actions. It does not execute the native ARC runner or its visual perception pipeline.

Local integration checks exercise the real HTTP endpoint, authentication, SQLite persistence and original Python memory/planner. They cover:

• One pending action at a time, version conflicts and exact write retries.
• Revision after a prediction mismatch and explicit resolution of uncertain outcomes.
• Memory backed by recorded observations and refusal of nonexistent evidence references.
• Completion only after recorded criteria pass; invalidation when later observations contradict earlier checks.
• Account ownership, key permissions and restoration of saved tasks after reopening the store.
• Planning over recorded transitions without inventing unseen routes.

Run the checked-task example to inspect one complete workflow. It deliberately corrects a prediction of 50 to a calculated total of 60, checks a budget of 65 and records 5 remaining. The example makes zero model calls.

These checks establish controller and protocol behavior. A controlled study of general-task accuracy, completion rate, latency and cost against the same model without OY1 has not yet been published. The hosted adapter does not inherit the ARC score.

05 / REPRODUCTION

What a comparison must preserve.

1. Identify the system.

   Record the exact source version, dependencies, model, provider and reasoning settings. Distinguish the native runner from the MCP adapter.

2. Fix the task set and budget.

   Use the same game versions or task definitions, tool access and limits. Declare any change before interpreting the outcome.

3. Keep the evidence.

   Preserve predictions, observations, action counts, failure reasons and scorecard snapshots. An uncertain external action should be checked before it is repeated.

4. Report successes and failures.

   Separate completion rate from action efficiency, latency and model cost. Include a matched baseline and repeat runs before claiming a general improvement.

Our public ARC-AGI-3 API harness repository includes source code, results and reproduction instructions under the Apache-2.0 license. It documents the API run featured on the homepage, separately from the earlier native baseline.

The published ARC run can be inspected through its scorecard and replays. The downloadable MCP example can be run against your own OY Labs account. An independent native-harness reproduction and a general-task comparison remain future evaluations.