Program Aided Language Models

PAL Model

The PAL model, or Program-Aided Language Model, introduces a novel approach to solving arithmetic problems by leveraging code generation. Aman Madaan explains that instead of forcing language models to perform calculations, PAL generates a Python program to solve the problem, thus offloading the computation to a Python runtime 1. This method reduces errors by eliminating the need for arithmetic calculations within the language model itself. Shuyan Zhou adds that Python was chosen for its strong performance and ease of execution, making it a natural fit for this approach 2.

Not having to do arithmetic calculations is the biggest source of gains.

--- Aman Madaan

By focusing on generating a plan rather than executing calculations, PAL enhances the accuracy of language models on arithmetic tasks 3.

   

Performance

PAL's effectiveness is evident in its performance on various benchmarks, particularly in arithmetic tasks. Aman Madaan notes that PAL achieves a 78-79% accuracy rate on GSM 8K, surpassing text-only reasoning methods 4. This improvement is attributed to PAL's ability to handle complex calculations by generating code rather than relying solely on text-based reasoning. Despite its strengths, language models still face challenges with basic arithmetic, such as multiplying large numbers, highlighting the limitations of current models 5.

There's a very surprising failure mode in multiplication, but also in adding very large numbers.

--- Aman Madaan

These insights underscore the need for continued refinement and innovation in language model capabilities.