Daniel Franzen & Jan Disselhoff - ARC Prize 2024 winners

Model Adaptations

In the ARC Prize 2024, Daniel Franzen and Jan Disselhoff explored model adaptations to enhance performance. They discovered that larger models, like a 32 billion parameter model, didn't necessarily outperform smaller ones due to the increased complexity and computational demands 1. Jan noted that models often struggled with tasks like counting or size estimation, but excelled in structured problems, indicating a nuanced understanding of problem-solving 2.

Transformers are just very, very good at learning facts. I think it's frankly incredible what they can store.

--- Jan Disselhoff

This adaptability suggests that continuous fine-tuning and learning from user interactions can significantly improve model efficiency over time 1.

   

Tokenization & Sampling

The team employed innovative tokenization and sampling methods, notably using Depth First Search (DFS) to enhance solution accuracy. Jan Disselhoff explained that DFS helped mitigate errors by generating solutions above a certain probability threshold, thus ensuring the most likely outcomes were considered 3. Daniel Franzen highlighted the challenge of predicting computational budgets for DFS, noting that iterative deepening could be a viable strategy to manage resources effectively 4.

The space of solutions the LLM generates is much, much smaller than the theoretical large space because a lot of the pixels are trivial.

--- Jan Disselhoff

This approach underscores the importance of balancing computational efficiency with solution accuracy, especially in complex problem-solving scenarios 5.