Scaling Pandas Challenges
Wes discusses the limitations of pandas when dealing with large datasets, highlighting the performance and memory challenges that arise as data scales. He emphasizes the tension between the desire for pandas' user-friendly API and the realities of big data processing. Additionally, he introduces innovative projects like IBIs that aim to bridge this gap by offering a different approach to data frame frameworks, while still incorporating some of pandas' core ideas. The conversation reflects a collective aspiration for Python to remain a leading data programming language.In this clip
From this podcast
Super Data Science: ML & AI Podcast with Jon Krohn
SDS 523: Open-Source Analytical Computing (pandas, Apache Arrow) — with Wes McKinney
Related Questions
What challenges arise in data sets as discussed in the episode SDS 523: Open-Source Analytical Computing (pandas, Apache Arrow) — with Wes McKinney and the clip Open Source Insights regarding Scaling Pandas Challenges?
What challenges arise in data sets as discussed in the episode SDS 523: Open-Source Analytical Computing (pandas, Apache Arrow) — with Wes McKinney and the clip Open Source Insights regarding Scaling Pandas Challenges?
What challenges arise in data sets as discussed in the episode SDS 523: Open-Source Analytical Computing (pandas, Apache Arrow) — with Wes McKinney and the clip Scaling Pandas Challenges?