SDS 523: Open-Source Analytical Computing (pandas, Apache Arrow) — with Wes McKinney

Origins

Wes McKinney shares the origins of the pandas library, highlighting its roots in his work at AQR Capital Management during the 2007 financial crisis. He explains how pandas was inspired by R's data frames, aiming to bring similar functionality to Python. The library has since evolved significantly, with a dedicated team of developers enhancing its features and performance.

I got a job in quantitative finance at AQR, Capital Management... right as the financial crisis was beginning.

--- Wes McKinney

Wes emphasizes the importance of community contributions in the ongoing development of pandas, noting that he has not been the sole maintainer since 2013 1 2.

   

Community

The open-source community has played a crucial role in the evolution of pandas. Wes McKinney highlights the contributions of core developers and the importance of community-driven events like documentation hackathons. These efforts have made pandas more accessible and robust, fostering a vibrant ecosystem of contributors.

Pandas has become this essential glue between different types of systems.

--- Wes McKinney

Jon Krohn adds that pandas is now used by over half a million projects on GitHub, underscoring its widespread adoption and impact 3 4.

   

Challenges

Scaling pandas to handle large datasets presents significant challenges. Wes McKinney discusses issues like memory use and performance bottlenecks, particularly when dealing with data sizes beyond a gigabyte. He mentions projects like Dask and IBIs that aim to address these limitations by enabling distributed computing and scale-out processing.

It's hard to make everybody happy in a tool like pandas.

--- Wes McKinney

Despite these challenges, the ecosystem continues to evolve, with ongoing efforts to improve pandas' scalability and efficiency 5 3.