The Android feels choppy/flow and the touch also doesn't feel super responsive

github.com/tansu-io/tansu

This is something to keep an eye on. Should be doable to add within a pg extension and bundle Kafka into Postgres?

Daylight is amazing. I'm really looking forward to a v2 (whenever it comes) that's a bit more refined

I mean, yeah - it's not a technical piece, it's a financial article.

When I hear the Kafka community, I think about engineering talk more than anything else

With greater competition and commercialization, it seems bound to happen.
I'm of the opinion we're bound to see consolidation in the space soon, because there's too many companies chasing too little of a market: bigdata.2minutestreaming.com/p/event-stre...

Then I don't see why a relational database can't maintain a pending queue too. Just process the lowest-id job

What are you referring to?

My intuition is rather that discussion has somewhat died down, in general.

the data bible dropping soon

re: fairness - I admit I'm not as familiar with. Can you teach me how some of these systems ensure fairness? Do they refuse to give messages to performant workers and hold them off for slower ones?

There is some fundamental misunderstanding here.

pgmq is not based on top of SQS. It only provides API parity with it. No messages actually go to SQS...

Latency-wise, my tests showed single-digit write and read. 99% use cases don't need less.

I really don't know. I haven't evaluated it yet.

ok, so what is missing?

> move data fairly, fast, and reliably

It is fast. It is reliable.
It is not fair (the simple implementation).

Is fairness the missing piece?

(Just want to streamline the discussion)

100%. I'm replying in facts.

I simply perceived your intro/outro belittling the Postgres argument to clickbait to reach front-page of HN as such, hence fought back.

GitHub - tansu-io/tansu: Apache Kafka® compatible broker with S3, PostgreSQL, SQLite, Apache Iceberg and Delta Lake Apache Kafka® compatible broker with S3, PostgreSQL, SQLite, Apache Iceberg and Delta Lake - tansu-io/tansu

The problem with building that is that... it exists! github.com/tansu-io/tansu

Pub-sub doesn't exist (much).

So yeah, unless somebody builds the `pgmq` equivalent and makes it work nice enough, it would be extra work that probably isn't justified.

It strongly depends on how much of that "complicated system" you truly need though!

I disagree with your take that it's "complicated". You can write a queue/pub-sub in an hour.

We have to separate the two, though:
- queues exists in battle-tested libraries like pgmq and the older pgq

So this is already built. Unless you want complex routing a-la Rabbit, its fine to use Postgres!

"You Don't Need Kafka, Just Use Postgres" Considered Harmful - Gunnar Morling · gunnarmorling discussions.morling.dev · Discussion #336 "You Don't Need Kafka, Just Use Postgres" Considered Harmful - Gunnar Morling Looking to make it to the front page of HackerNews? Then writing a post arguing that "Postgres is enough", or why "you ...

I think the blog makes a lot of straw mans and misses the point.

I definitely agree with some points made there and remain reasonable - but I think that under my mentioned constraints, PG is likely the right choice

I posted a rebuttal here if you care enough to read github.com/gunnarmorlin...

Well... that would pretty easily be alleviated by just running a separate instance, right? I am not advocating for co-locating it.

In any case, I think one would need to expose some dumbed down API for pub-sub users to avoid them doing more than just using the pub-sub, at least in the apps.

What scale are we talking about here?

Doesn't sound like a bad idea at all. When you've got close to a billion users, might as well connect them!

Just use Postgres until it breaks 🧘‍♂️

Thanks, it's mine. It comes off this post topicpartition.io/blog/postgre... where I focus on how PG solves so many problems at smaller scale

What has changed since?

This was posted 15 years ago

(part 1/2)

**taps the sign**

way, way too many

Small Data Small Data Small data appears to be a very exciting movement that is moving the overton window away from Big Data onto much simpler and cheaper solutions ...

when I say “storage is cheaper now” this is what I mean

topicpartition.io/definitions/...

KIP-1150 in Apache Kafka is a big deal (Diskless Topics) TL;DR KIP-1150 introduces Diskless Kafka topics that write directly to S3 instead of replicating between brokers. It literally reduces costs by 97% (from $1.8M to $20K annually for a 1GiB/s cluster) a...

“KIP-1150 introduces Diskless Kafka topics that write directly to S3 instead of replicating between brokers.”

“Even using the expensive S3 Express (which a week ago lowered its prices by more than 50%) still saves 73% compared to traditional Apache Kafka.”

/ht @ananthdurai.bsky.social

all in 2 minutes, on the best place for Kafka on the world wide web 👇

blog.2minutestreaming.com/

a new 2 minute streaming post is sitting patiently in your inbox...

open it to learn when:
• Kafka decides what messages are visible to Consumers
• acks=all Producers receive responses