@ananthdurai.bsky.social
Editor Data Engineering Weekly; subscribe www.dataengineeringweekly.com. In Prgress, LakeByte
Real-time contextual data is no longer optional — it’s foundational.
LLMs demand live, relevant signals.
We’re not building for dashboards anymore. We’re building for agents.
From batch to streaming.
From static to dynamic.
From reports to APIs.
#LLM #AI #DataOps
The 230th edition of Data Engineering Weekly is out!! Read more
www.dataengineeringw...
Catch up on all the latest happenings in the data industry this week's DEW edition
www.dataengineeringw...
The upstream pipeline design and data modeling techniques, leveraging a vector store like S3 Vectors, and the Lakehouse architecture will drive the next-gen evolution of data engineering.
16.07.2025 16:51 — 👍 1 🔁 0 💬 0 📌 0I truly wish to switch to Claude Code, but the reliability sucks. In a second attempt, I got the error. Dropping it for now.
API Error (529 {"type":"error","error":{"type":"overloaded_error","message":"Overloaded"}}) · Retrying in 1 seconds… (attempt 1/10
@AnthropicAI
I keep meaning to read all the Atlassian Work Life blog posts, but ironically, they’re stuck in my backlog—and every time I open Jira, my work-life balance dies a little more. 🤷♂️
What do you think of DuckLake?
02.06.2025 22:01 — 👍 5 🔁 0 💬 0 📌 0
The 222nd edition of Data Engineering Weekly is out!!
www.dataengineeringweekly.com/p/data-engin...
🚨 Beware of quick fixes!
“Applying a half-baked theory is like using duct tape on a parachute—technically a fix, until gravity files a formal complaint.”
Scaling flawed ideas often scales their consequences too. Always write down your assumptions before shipping.
The DuckLake theme seems to speed up the metadata lookup. A couple of things I'm not sure
1. Does this still work with the Iceberg format? Do people need a migration from Iceberg?
2. How long will this design hold with AWS Express One slashing costs?
duckdb.org/2025/05/2...
The 221st edition of Data Engineering Weekly is out with a fresh set of articles.
www.dataengineeringw...
Systems like Gluten/ Comet replaces executors with the native implementation, but keep the planner, cluster manager and task scheduler as it is of Spark.
21.05.2025 22:28 — 👍 3 🔁 0 💬 0 📌 0
The rise of drop-in replacements for Apache Spark is hard to ignore. Is Spark finally showing its age? The one from Onehouse is pretty interesting.
www.onehouse.ai/blog...
Photon and Comet are attempting the same.
datafusion.apache.or...
www.databricks.com/p...
The 219th edition of Data Engineering Weekly is out. Find the latest trends in AI & data by clicking here 👇🏼
www.dataengineeringw...
The 218th edition of Data Engineering Weekly is out, featuring the latest AI & Data stories.
www.dataengineeringw...
Chaos testing is a feature.
25.04.2025 21:34 — 👍 2 🔁 0 💬 0 📌 0
This expert panel could have been replaced with ChatGPT, Claude & Gemini. You’re welcome.
23.04.2025 13:33 — 👍 0 🔁 0 💬 0 📌 0
“KIP-1150 introduces Diskless Kafka topics that write directly to S3 instead of replicating between brokers.”
“Even using the expensive S3 Express (which a week ago lowered its prices by more than 50%) still saves 73% compared to traditional Apache Kafka.”
/ht @ananthdurai.bsky.social
The 217th edition of Data Engineering Weekly is out. Read more here
www.dataengineeringw...
The problem statement is real, and I'm curious to know what Bauplan's secret sauce is to solve this problem.
16.04.2025 21:16 — 👍 1 🔁 0 💬 0 📌 0This looks like an exciting proposal
cwiki.apache.org/con...
The 216th edition is fresh and out!! Read more here
www.dataengineeringw...
Read more about composable data architecture in my recent blog.
www.dataengineeringw...
The 215th edition of Data Engineering Weekly is out!!
www.dataengineeringw...
On April 1st, Data Engineering Weekly officially acquired datashitposting.com, pinoner in data shit posting. As a bonus of the acquisition, please enjoy this shit post.
02.04.2025 02:29 — 👍 4 🔁 0 💬 0 📌 0
The 214th edition of Data Engineering Weekly is out!!
www.dataengineeringw...
I don’t know about others, but manufacturing and operating code are two parts of the software supply chain. We accelerated the manufacturing process, yielding mixed-quality code impacting operational efficiency.
27.03.2025 10:20 — 👍 1 🔁 0 💬 0 📌 0
The 213th edition of Data Engineering Weekly is out!!. Read more
www.dataengineeringw...
From WSDL to Claude MCP, creating structured, documented interfaces for remote services has been a decades-long problem. Both try to answer the question: "What can this service do?" "How do I talk to it?" and "What should I expect in return?"
19.03.2025 21:32 — 👍 0 🔁 0 💬 0 📌 0
I tried to build a mobile app in react native using repl.it and bold.new; Both failed to build the project with react native dependency. It seems no level of intelligence can solve the dependency hell.
17.03.2025 20:55 — 👍 2 🔁 0 💬 0 📌 0
