LLMs do fine on ARC-AGI-3 if they are allowed to search over game logs

I was reading the comments to this post and the overall opinion seemed to be that harness makes little/no difference for ARC-AGI-3. Turns out, it makes a huge difference: Hill-climbing ARC-AGI-3 TLDR: if you save game logs - taken actions, board states and scores - and let LLMs search over them with tools, LLMs are only moderately less efficient than humans in terms of the number of actions taken to beat ARC-AGI-3 games. Frontier LLMs struggle out of the box on this benchmark. In our preliminary

More from Singularity

Figure's First Full HQ Tour: From the Lab to the Factory Floor

Singularity·6

Figure AI’s CEO shows the Figure 03 humanoid robot’s feet, designed to allow the robot to charge wirelessly

Singularity·7

Scaling the physical Singularity: Autonomous 18-wheelers are no longer a 'pilot project'

Singularity·7

ARC-AGI-3 Update (GPT-5.5 High and Opus4.7)

Singularity·3

Anthropic Won't Let You Use Their Best Model. Prediction Markets Are Trying Anyway.

Singularity·3

UPDATE: The method from the proof generated by GPT-5.4 Pro for Erdos Problem #1196 was successfully applied to other problems including another 60 year old Erdos conjecture.

Singularity·4