Ian Cook

Next Tuesday, get ready to meet the mind behind #Pandas & #ApacheArrow!

@wesmckinney.com shares his origin story (Part 1) on #TheTestSet. From speedruns to shaping the data stack, this is one you won't want to miss.

Mark your calendar for Tuesday & subscribe at thetestset.co!

#DataScience #Python

@duckdb.org 🤝 @arrow.apache.org

🚀 Introducing **Bauplan**

A serverless, code-native platform for building data and AI pipelines — directly on your object store. No clusters. No notebooks. No GUI based workflows.

Just Python + SQL + S3.

👉 www.bauplanlabs.com/blog/hello-b...

How the Apache Arrow Format Accelerates Query Result Transfer Arrow speeds up query result transfer by slashing (de)serialization overheads. We outline five key attributes of the Arrow format that enable this.

2025 is shaping up to be a breakout year for fast query result transfer with Apache Arrow. But what exactly makes it so fast? David Li, Matt Topol, and I break it down in this new blog post: arrow.apache.org/blog/2025/01...

[🎥 Watch] Apache Arrow is a columnar format and multi-language toolbox for fast data interchange and in-memory analytics.

Matt Topol, Arrow PMC member, talks about Arrow subprojects and how you can get involved with the project. https://buff.ly/40qNPAL

#opensource

There are some great books in this vein by Joseph Stiglitz and David Graeber

YouTube video by Saturday Night Live Jiffy Express - Saturday Night Live

The best illustration of how users and buyers of a product can have opposite priorities

Would love to see more discussion of this. To Andy's point we have this wonderful Arrow eco system that's underdeveloped. In a world of columnar data we're still stuck using ODBC/JDBC for connectivity to most data systems. Hard to get some vendors to even talk about ADBC.

A grocery store receipt showing an undecipherable series of characters followed by ",bin.base64"

Base64 decoding error on my Food Lion receipt today

You might like to try Ibis: ibis-project.org. It was created by Wes McKinney (also the author of pandas) as a solution to some problems pandas couldn't solve well. It's had a lot of recent growth and improvement. There's a tutorial for users coming from dplyr: ibis-project.org/tutorials/ib...

Yes—this came up for me because I’m playing with the Swift implementation of Apache Arrow, which is maintained in a directory of the Arrow monorepo, but Swift’s package manager assumes a 1:1 relationship of repo:package

Pretty funny that the easiest way to do it was with an SVN command, until GitHub dropped support for that earlier this year.

It's 2024 and there's still no way to download just one subdirectory of a Git repo without using a sketchy third-party website or tool? Wild.

There’s some great news coming soon on this. Unfortunately I can’t share it publicly now, but DM me if you’re curious

There’s also ADBC which aims to replace ODBC and JDBC in analytics applications with a much faster Arrow-based alternative.

DM me if you’re interested to discuss

Arrow IPC stream format works great for sending over HTTP APIs. There’s also Arrow Flight which is a framework for sending Arrow IPC data through RPC APIs.

Arrow is much more efficient than JSON for data transport in OLAP applications because you avoid transposing columns to and from rows.

The Arrow IPC stream format has become common in recent years as a data exchange format.

Basically: If you design an optimal columnar format for on disk, you end up with something like Parquet. If you design an optimal column format for in memory and over the wire, you end up with Arrow IPC format

arrow-experiments/http at main · apache/arrow-experiments Apache Arrow Development Experiments. Contribute to apache/arrow-experiments development by creating an account on GitHub.

If you’re sending Arrow IPC over HTTP APIs, there are some great examples showing the basics at github.com/apache/arrow...

Simply using the Arrow IPC stream format as a transport format (instead of JSON) usually gives most of the performance of Arrow Flight with lower implementation complexity