Ananth Packkildurai

The Missing Layer in Your AI Stack: Context, Not Just State From SQL to Semantics: The Rise of the Context Graph for AI Agents

As we move from dashboards to autonomous agents, something breaks.

Systems of record capture what happened, not why.

Why data platforms need Truth Registries + Context Graphs for the agentic era 👇
www.dataengineeringw...

#DataEngineering #AgenticAI #Graphs #LLMs

Data Engineering Weekly's 254th edition is out. Context Graph is the new talk of the town!!

The companies that build the most boring data stack often win the market!!!

Prove me wrong.

Data Contracts: A Missed Opportunity The Conversation We Should Have Had—Before Thought Leadership Replaced System Design

Data Contract: There was no shortage of activity around the topic. Definitions were proposed and refined. Conceptual boundaries were drawn and redrawn.

I pen down a reflection of the Data Contracts here

www.dataengineeringweekly.com/p/data-contr...

How to build a scalable shopping agent?
Here's a wild thought:
What if—and hear me out—we let humans click that Buy Now button? Just throwing ideas out there.

Data Engineering Weekly #252 The Weekly Data Engineering Newsletter

This week, it is mostly about Multi-Agent Architecture. Do you think the data infrastructure is ready for a multi-agent architecture? Where is the gap?

A Critique of Iceberg REST Catalog: A Classic Case of Why Semantic Spec Fails How a Semantically Correct API Becomes Operationally Unreliable at Scale

Is semantic Spec Good enough to run an enterprise system? I listed challenges to adopting the Iceberg Rest Catalog

DEW - The Year in Review 2025 From Digital Plumbers to Architects of Intelligence: The 7 Paradigm Shifts That Defined 2025

Continuing our yearly tradition of Year in Review Data Engineering Weekly, we published the 2025 Year in Review. What do you think is the most notable trend of 2025?

www.dataengineeringw...

Look at the tech stack IBM now controls:

🐧 Compute: Red Hat (Linux/OpenShift)
☁️ IaC: HashiCorp (Terraform)
💰 FinOps: Kubecost
🌊 Streaming: Confluent (Kafka)
🧠 Vector/AI: DataStax (Cassandra)
⚡ Query Engine: Ahana (Presto)
🔄 Ingest: StreamSets

Data Engineering Weekly #247 The Weekly Data Engineering Newsletter

LinkedIn moves FishDB to Rust, DoorDash builds AI swarms, and Dropbox masters context engineering. 🤯 Data Engineering Weekly #247 is packed with system design deep dives from the best engineering teams.

If the Data Catalog is the answer for AI, the question was wrong.

The Dark Data Tax: How Hoarding is Poisoning Your AI Storage is cheap. Attention is finite. Hallucinations are expensive. It’s time to stop building Data Lakes and start managing Data Metabolism

We stopped asking if data was useful because storage got cheap. Now, "Dark Data" is actively poisoning your AI context windows with hallucination vectors.

Read about the Data Sustainability index

The open source companies built their success on top of open-source platforms, benefited from community contributions and adoption, but now must abandon open-source principles to survive commercially.

Data Engineering Weekly #244 The Weekly Data Engineering Newsletter

🚀 The 244th edition of Data Engineering Weekly dives into:

AI agents as execution engines, LLM inference economics, databases for AI, personalization, and product evidence.

Read more 👉 www.dataengineeringw...

#DataEngineering #AI #LLMs

Cricket has been India’s greatest force in overcoming centuries of colonial suppression. Today’s Women’s World Cup win echoes the spirit of 1983 — a triumph that will inspire generations to come. 🇮🇳🏆

Thinking Like a Data Engineer A Journey Beyond Code — Toward Systems, Curiosity, and Confidence

This is the most personal essay that I have written in Data Engineering Weekly. I shared a few key moments in my life and how fortunate I was to meet mentors along my professional journey, which shaped my career.

Revisiting Medallion Architecture: Data Vault in Silver, Dimensional Modeling in Gold How to Balance Flexibility and Performance in a Modern Data Platform

🚀 Data Vault vs. Dimensional Modeling vs. Medallion Architecture — When viewed through a modern enterprise data lens, these techniques interlock.

I break down how in Part 2 of my “Revisiting the Medallion Architecture” series.

Fivetran and dbt form a strong foundation for modern data infrastructure, known for bringing simplicity to complex engineering workflows. That said, calling it “open” data infrastructure feels like a stretch.

Should we update the definition of an "Analytical Engineer"?

Engineering Growth: The Data Layers Powering Modern GTM Building privacy-preserving pipelines that unify zero-, first-, second-, third-, and fourth-party data into a coherent GTM ecosystem.

As a data engineer, you can't treat zero-party (consent) and third-party (inferred) data the same way. This distinction is critical for building systems that are scalable, private, and trustworthy.

Here’s my guide:

Could be. Composable CDP has not gained significant market share, as identity resolution is a key component that is often proprietary.

With Census already in with Fiveatran and with dbt, it is most likely to evolve as a composable CDP.

Airbnb: Real-Time Key-Value Store

Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection.

Read the full story here: www.dataengineeringw...

Data Engineering Weekly #239 The Weekly Data Engineering Newsletter

Read the full story here:

Grab: Partner Gateway Metrics at Sub-Second Speed
Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.

Data Engineering Weekly #239 The Weekly Data Engineering Newsletter

💡 Read the full story →

Netflix Muse: Scaling Analytics at Trillion-Row Scale
Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.

Data Engineering Weekly #239 The Weekly Data Engineering Newsletter

🔗 Link in bio: