Julien Hurault

Indeed it s not simple unfortunately..just the a way to get started quickly atm .

❤️❤️

For iceberg catalog hard to find a simpler setup..

Nice! Can you orchestrate lambda or ecs tasks that way?

Just use Pyiceberg with AWS Glue, probably the fastest way to get started.

In term of volume of data exchange over the marketplace? No idea

SF sales rep told me that markeplace was THE feature that helped a lot to convert

Saw it lot in finance!

Multi pipelines will probably get tricky no?

Dlt + duck + evidence + @blef.fr's baby?

GCP & Iceberg Ju Data Engineering Weekly - Ep 77

For those lost in GCP terminology:
I wrote a summary of Iceberg integration in GCP a couple of weeks ago:
juhache.substack.com/p/gcp-and-ic...

0$ Data Distribution Ju Data Engineering Weekly - Ep 78

New blog post: Building a 0$ Data Distribution System.

juhache.substack.com/p/0-data-dis...

30 million rows is just one month of data, right?

check catalog.boringdata.io/dashboard/in...

Do you support swiss banks by any chance?

" bash / make knowledge, a single instance SQL processing engine (DuckDB, CHDB or a few python scripts), a distributed file system, git and a developer workflow (CI/CD)" what s your best option to orchestate sql models in such setup?

Some learnings after helping +50 companies in high performance data engineering projects

javisantana.com/2024/11/30/l...

Super good thx!
"immutable workflow + atomic operation" 100%!

ATTACH url_to_your_dabase.duckdb;

Oh yeah

wiki.postgresql.org/wiki/Don%27t...

S3 to Snow in the same aws region is free no?

Niiiice, your view is doing a read_parquet(*) on their bucket? Or do you copy the data?

YouTube video by Aaron Francis DHH discusses SQLite (and Stoicism)

1 Docker container embedding app code + SQLite DB → live chat app with 10k simultaneous users.

youtu.be/0rlATWBNvMw

Where is the data stored, then? In DuckDB itself?
So, if you have a 1GB dataset, does that mean you’ll share a single .duckdb file containing the entire dataset? Or either a view pointing to parquet files: CREATE VIEW... as read_parquet(*.parquet) ?

Do you see DuckDB as a format?

For me:
• Parquet = Standard storage format
• Iceberg = Standard metadata format
• DuckDB = One possible distribution vector

Yup, there are almost fifteen million SQLite databases on Bluesky’s PDS servers. It’s wildly efficient and simple but not without trade offs of course.

Makes sense for this use case in large part because each users atproto repository is self contained, with links to other repos, like a website.

just do both -> catalog.boringdata.io/dashboard/in...

A screenshot of .zshrc code with the following content: # Function to auto-activate virtual environment function auto_activate_virtualenv() { if [ -d ".venv" ]; then source .venv/bin/activate fi } # Hook the function to the 'chpwd' event (triggered when you change directories) autoload -U add-zsh-hook add-zsh-hook chpwd auto_activate_virtualenv # Also call the function for the initial directory when the terminal starts auto_activate_virtualenv

Here's what I put in my ~/.zshrc file to make sure my virtualenv autoactivates when I move to a directory with a .venv file. Works well for me so far. Do the rest of you do something like this?

#Python #DataBS

Prediction: poeple will monetize custom feeds
github.com/bluesky-soci...

Something interesting is brewing in Iceberg-on-S3 land. 👀

lists.apache.org/thread/v7x65...

cc @eatonphil.bsky.social