Which Programming Language is ChatGPT Best At

Methodology

Alessio Buscemi's research investigates the performance of ChatGPT 3.5 in generating code across ten programming languages. He selected languages like C, Go, JavaScript, and Python, among others, and tasked ChatGPT with 40 challenges in domains such as basic algorithms, games, data science, and security. Alessio aimed to determine whether certain languages are inherently more suitable for code generation by large language models (LLMs).

I conducted this study to see whether indeed there were maybe some not so popular languages that could perform as good as the popular ones.

--- Alessio Buscemi

The study revealed unexpected results, sparking curiosity about the characteristics that make a language more apt for LLM code generation 1.

   

Unexpected Findings

The research uncovered intriguing and unexpected findings, particularly in how ChatGPT interprets tasks. Alessio intentionally kept task descriptions vague to assess the model's creativity and adaptability across different languages. For example, tasks ranged from checking if an integer is a palindrome to implementing a simple version of Pong, with varying success depending on the language used.

My analysis of the output was just made on whether, first of all, if the model would output any code, because in many instances it just refused to output any code.

--- Alessio Buscemi

These results highlight the complexities and limitations of LLMs in code generation, emphasizing the need for further exploration into their capabilities 2 1.