I wonder if anyone has done a cost analysis of Python code running in the wild compared to a language that actually is performant. I suspect companies are needlessly spending millions because of lazy developers.

Not me, a different Bruce RItchie

F3: The Open-Source Data File Format for the Future SIGMOD 2025

Our SIGMOD paper with our friends at Tsinghua + @wesmckinney.com + @pateljm.bsky.social on creating a next generation open-source data file format is out. F3 is a future-proof file format avoids the mistakes of Parquet.
📄 Paper: db.cs.cmu.edu/papers/2025/...
📁 Code: github.com/future-file-...

GitHub - future-file-format/F3 Contribute to future-file-format/F3 development by creating an account on GitHub.

github.com/future-file-...

Optimizing ClickHouse for Intel's ultra-high core count processors Intel's latest processor generations are pushing the number of cores in a server to unprecedented levels. For analytical databases like ClickHouse, ultra-high core counts represent a huge opportunity ...

Interesting read on what it takes to optimize a database for high core count machines - clickhouse.com/blog/optimiz...

Vortex | An extensible, SOTA columnar file format Vortex is an extensible, state-of-the-art columnar file format, with associated tools for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.

I'm tempted to try out the vortex file format (vortex.dev) in my project to see if it has an appreciable impact on performance.

ashtom.github.io/developers-r... ... so much absurdity in this it's crazy. Never trust a damn thing from someone whose job depends on selling you something.

@apachedatafusion.bsky.social 49.0.0 released. Async UDF's, Parquet modular encryption, WITHIN GROUP support, Dynamic Filters and TopK pushdown and much more ... datafusion.apache.org/blog/2025/07...

Medium has turned into a wasteland of AI generated or AI augmented posts. I'd say less than 25% of the daily digest highlights are actual 'real' articles. Sad.

A 200 Ok response from S3 ... isn't always ok. Way to go AWS for making your service horrendous to support. repost.aws/knowledge-ce...

I am unsure whether Google Summer of Code is a benefit or a hindrance to an open source project. Time will tell I suppose by the PR's submitted.

It's been well over a year since I started the process of rewriting a large and very long running job from Apache Spark/Scala to Apache DataFusion/Rust. We're now well into doing poc's to rewrite a few other expensive jobs the same way. It's a very nice feeling.

This one was going around the office today and made me chuckle :)

Not by me, I'm not in Florida, nor in the US.

coworker in chat: "... cluster is rebalancing and I'm trying to get the jello to stop shaking". Best explanation of rebalancing I've heard in a long time 🤣

Thank you Doug Ford for the $200 vote bribe. I'll use it to contribute to another party and vote to get your kind out of office.

Had a good chuckle this morning. Gemini was enabled on company corporate accounts and lasted all of 2 days before it was disabled.

The latest paper from the #1 CMU-DB PhD student @samarchdb.bsky.social is wild compilation magic! He automatically makes UDFs run 300x faster on SQL Server and 1.3x faster on DuckDB.
Code: github.com/SamArch27/PR...
Paper: www.vldb.org/pvldb/vol18/...

Working in Rust for the last year has really made me aware of just how useful some features in other languages really are.

- variadic functions
- Default values for arguments
- Named arguments
- Enum variants as types

Rust is getting if let chains in the 2024 edition though so that is something.

Lately there are two things I've been wishing that #Rust had: variadic functions and enum variants as types. Using a builder or macro to work around the first is just that, a workaround. Having the second would make some things much nicer

64GB of ram is not enough any more.

Datafusion v43 has seen a lot of performance work especially around reading parquet and the numbers are very nice! From the clickbench benchmark on the same hardware type: