Daniel Franzen & Jan Disselhoff - ARC Prize 2024 winners

Solution Design

Daniel Franzen and Jan Disselhoff, winners of the ARC Prize 2024, shared their innovative approach to solving ARC tasks using large language models (LLMs). They began with a 12 billion parameter LLM, tokenizing tasks into text and feeding them directly into the model without pre-processing. This approach, combined with test-time fine-tuning, significantly improved their model's performance. Daniel explained, "We do another training process on the examples of the validation set, which we get during inference without the final challenge, which we then try to predict with the model, and that gives a big improvement to the score." 1 2.

   

Refinement

The iterative refinement of their approach was crucial, involving constant testing and adaptation. Jan noted that many ideas initially seemed promising but didn't pan out, leading them to focus on strategies that reliably improved performance. The depth-first search algorithm and scoring process were pivotal, enhancing their model's capability to generate and select solutions effectively. Jan remarked, "The DFS was a very natural extension of the scoring, I think after we had a good working scoring and selection algorithm, the challenge became to generate good candidates." 3 4.

   

Augmentation Techniques

Data augmentation played a critical role in enhancing model accuracy and generalization. They employed symmetry augmentations and test-time training to maximize data utility, given the limited examples per challenge. Jan explained, "We heavily use augmentation in basically all our inference steps. Also in training. We also use it for the pre training in rehab, actually. But it helps a lot to get enough data during the retraining or the test time training." 5 6.