Skip to content

RESEARCH / EVALUATION

OY1 evaluation.
Evidence and scope.

The ARC-AGI-3 scorecard, the original harness and the hosted MCP adapter answer different questions. Here is what was measured, how to inspect it, and what remains untested.

By OY Labs · Published · Updated

01 / ARC-AGI-3

A result with a defined scope.

The ARC Prize scorecard published on 6 September 2026 reports 100.00%, with 183 of 183 levels completed across 25 of 25 environments and 6,659 actions. It includes per-environment results and links to replays.

Score
100.00% on the linked scorecard
Completed
183 levels across 25 environments
Actions
6,659, as reported by the scorecard
Baseline
OY1 harness v0.2.7, according to OY Labs’ release records

These figures describe that recorded run. They do not establish performance on every ARC-AGI-3 environment, an independent replication, competition placement or general intelligence. A published scorecard is not a peer-reviewed technical report.

02 / THE ORIGINAL HARNESS

Keep the method identifiable.

The original native harness used GPT-6 Astra High through Codex App Server: high reasoning effort, with fast mode off. OY Labs’ release records identify these as the baseline’s settings. The native runner controls model calls and interaction with the ARC environment. It includes visual perception, memory and planning.

The preserved follow-up candidate is version 0.2.7+oy1.1. Its repeat protocol specifies the selected 25 games, a maximum of 1,500 actions and 1,000 model calls per game, a 7,200-second time limit, and batches of up to eight actions. These are protocol ceilings, not measurements of consumption in the published baseline run.

Source identities

The baseline source digest recorded by OY Labs is 8b2119cb7094e0e08a2c2f3c35319cb31ea6995ee4e33d83afbee663eea5b7a4. The preserved candidate digest is a88242122f4ebbd9ba37d6e0a8a49bea08a3e66cc5b6c2c190c18ecc6f35747d.

The digest procedure sorts the Python modules by basename and hashes each basename’s UTF-8 bytes followed by its file bytes. The candidate repeat remains pending. These digests identify the earlier native baseline and candidate. The separate ARC-AGI-3 API harness is now open source on GitHub, with its own run evidence and reproduction guide.

The scorecard supplies the public result and replays. The version and configuration details above come from OY Labs’ preserved release records; they have not been independently reproduced for this page.

03 / THE HOSTED SERVICE

What MCP validation covers.

The hosted OY1 MCP adapter has its own task controller and versioning. It uses the original memory and empirical planner, while the connected client controls reasoning, model calls and external actions. It does not execute the native ARC runner or its visual perception pipeline.

Local integration checks exercise the real HTTP endpoint, authentication, SQLite persistence and original Python memory/planner. They cover:

  • One pending action at a time, version conflicts and exact write retries.
  • Revision after a prediction mismatch and explicit resolution of uncertain outcomes.
  • Memory backed by recorded observations and refusal of nonexistent evidence references.
  • Completion only after recorded criteria pass; invalidation when later observations contradict earlier checks.
  • Account ownership, key permissions and restoration of saved tasks after reopening the store.
  • Planning over recorded transitions without inventing unseen routes.

Run the checked-task example to inspect one complete workflow. It deliberately corrects a prediction of 50 to a calculated total of 60, checks a budget of 65 and records 5 remaining. The example makes zero model calls.

These checks establish controller and protocol behavior. A controlled study of general-task accuracy, completion rate, latency and cost against the same model without OY1 has not yet been published. The hosted adapter does not inherit the ARC score.

04 / REPRODUCTION

What a comparison must preserve.

  1. Identify the system.

    Record the exact source version, dependencies, model, provider and reasoning settings. Distinguish the native runner from the MCP adapter.

  2. Fix the task set and budget.

    Use the same game versions or task definitions, tool access and limits. Declare any change before interpreting the outcome.

  3. Keep the evidence.

    Preserve predictions, observations, action counts, failure reasons and scorecard snapshots. An uncertain external action should be checked before it is repeated.

  4. Report successes and failures.

    Separate completion rate from action efficiency, latency and model cost. Include a matched baseline and repeat runs before claiming a general improvement.

Our public ARC-AGI-3 API harness repository includes source code, results and reproduction instructions under the Apache-2.0 license. It documents a separate API run from the earlier native baseline above.

The published ARC run can be inspected through its scorecard and replays. The downloadable MCP example can be run against your own OY Labs account. An independent native-harness reproduction and a general-task comparison remain future evaluations.

Explore the worked example →Read about OY1 →