Buf

Getting Started with the Buf Schema Registry Our interactive workshop is the best way to learn the ins and outs of our centralized registry for tracking and evolving Protobuf APIs.

Working with Protobuf? We’ve got the online workshop for you! Our engineers are hosting a 1-hour session that shows how to make depending on Protobuf schemas as easy as using go get, npm, mvn, or pip.

Thurs Aug 21 9 AM PDT / 12 PM EDT / 5 PM BST

Sign-up today at: buf.build/events/bsr-w...

Introducing hyperpb: 10x faster dynamic Protobuf parsing that’s even 3x faster than generated code Today we’re announcing public availability of hyperpb, a fully-dynamic Protobuf parser that is 10x faster than dynamicpb, the standard Go solution for dynamic Protobuf. In fact, it’s so efficient that...

Announcing hyperpb, a fully-dynamic Protobuf parser that achieves 3x performance over generated Go code! Our hyperspeed dynamic parsing turns once-unscalable products into ordinary, even essential components of your stack.
Check out our benchmarks! ⚡

buf.build/blog/hyperpb

How to bring schema-driven governance to streaming data Our interactive workshop is the best way to learn the ins and outs of Bufstream, our Kafka-compatible, schema-first streaming data platform.

Streaming data got a data problem? Don’t miss our workshop on schema-driven governance for Kafka. Our engineering and product teams will show what you can do when brokers understand the shape of your streaming data. Q&A throughout!

Thursday, July 10 9 AM PT / 12 PM ET / 5 PM BST

Why Schema-Driven Streaming is the Future of Kafka | Brij kishore Pandey Why Schema-Driven Streaming is the Future of Kafka Anyone who’s managed Kafka at scale knows the pain: -> Managing multiple clusters -> Dealing with frequent schema mismatches -> Wasting ...

"If you're aiming to build robust, modern data pipelines without Kafka’s complexity, schema-first is the way forward." Couldn't have said it better ourselves.

www.linkedin.com/posts/brijpa...

Don’t miss our interactive workshop on bringing schema-driven governance to Kafka. Our engineering and product teams will deep dive into how it’s done. Plus, we’ll answer your questions throughout.

📷 Thursday, July 10 📷 9 AM PDT/12 PM EDT/5 PM BST

buf.build/events/works...

YouTube video by Abhishek.Veeramalla Data Quality Concerns & Storage Cost challenges with Apache Kafka

Another day, another great video on the data quality issues that plague Apache Kafka and other Kafka-compatible platforms. Abhishek Veeramalla goes in depth on schema validation, plus Kafka’s high storage and networking costs. Spoiler alert: Bufstream is the answer.

www.youtube.com/watch?v=oR4F...

YouTube video by Databricks Unifying Human-Curated Data Ingestion and Real-Time Updates with Databricks DLT, Protobuf and BSR

See how Clinician Nexus governs data quality with the Buf CLI + Buf Schema Registry and keeps costs low with Kafka-compatible Bufstream. Results: consistent validation, quick updates & complete audit trail. #DAIS #Kafka

www.youtube.com/watch?v=qZLG...

Tip of the week #9: Some numbers are more equal than others The first 15 field numbers are special: most runtimes will decode them much faster than the other field numbers. When designing a message type for decoding performance, it’s good to use these field nu...

TL;DR: The first 15 field numbers are special: most runtimes will decode them much faster than the other field numbers. When designing a message type for decoding performance, it’s good to use these field numbers on fields that are almost always present.

buf.build/blog/totw-9-...

Oneofs are a disaster. Protovalidate has fixed them. Instead of using oneofs, you can now use the new (buf.validate.message).oneof Protovalidate annotation. As long as you're validating your messages with Protovalidate, (buf.validate.message).oneof does...

Oneofs are a disaster. So we fixed them. buf.build/blog/fixing-...

Buf Announces Support for Databricks Managed Iceberg Tables We’re announcing a major step toward our vision of a single schema definition in Protobuf that governs your entire data flow — from API contracts to streaming topics to analytical tables. Bufstream no...

Big news from #DataAISummit! Bufstream now supports Databricks Managed Iceberg Tables in private preview, bringing together Buf's schema-first approach with Databricks' industry-leading data governance and optimization capabilities. Read all about it!

buf.build/blog/buf-ann...

Tip of the week #8: Never use required Don’t use 'required' to modify fields—you won’t be able to get rid of it later when you realize it was a bad idea.

TL;DR Don’t use required, no matter how tempting. You won’t be able to get rid of it later when you realize it was a bad idea.

buf.build/blog/totw-8-...

Tip of the week #7: Scoping it out buf convert and protoscope are powerful tools for examining and debugging wire format dumps.

TL;DR: buf convert is a powerful tool for examining wire format dumps, by converting them to JSON and using existing JSON analysis tooling. protoscope can be used for lower-level analysis, such debugging messages that have been corrupted.

buf.build/blog/totw-7-...

How to bring schema-driven governance to streaming data Our interactive workshop is the best way to learn the ins and outs of Bufstream, our Kafka-compatible, schema-first streaming data platform.

Don’t miss our interactive workshop on bringing schema-driven governance to Kafka. Our engineering and product teams will deep dive into how it’s done. Plus, we’ll answer your questions throughout.

Thursday, May 29 9 AM PDT / 12 PM EDT / 5 PM BST

Tip of the week #6: The subtle dangers of enum aliases Enum values are a poorly designed feature and shouldn’t be used.

TL;DR: Enum values can have aliases. This feature is poorly designed and shouldn’t be used. The ENUM_NO_ALLOW_ALIAS Buf lint rule prevents you from using them by default.

buf.build/blog/totw-6-...

Bufstream is now on the AWS Marketplace We're excited to announce that Bufstream is now available on the AWS Marketplace. Enterprise customers can purchase through their AWS account and accelerate their deployment of Bufstream.

We’re excited to announce that Bufstream is now available on the AWS Marketplace. Enterprise customers can purchase through their AWS account and accelerate their deployment of Bufstream.

buf.build/blog/bufstre...

How to bring schema-driven governance to streaming data Our interactive workshop is the best way to learn the ins and outs of Bufstream, our Kafka-compatible, schema-first streaming data platform.

WORKSHOP: Schema-Driven Governance for Streaming Data

Learn about Bufstream, our Kafka-compatible #streamingdata platform. This workshop will show what’s possible when you understand the shape of your streaming data.

Thursday, May 29 9 AM PDT / 12 PM EDT / 5 PM BST

buf.build/events/works...

Tip of the week #5: Avoid import public/weak Avoid import public and import weak.

TL;DR: Avoid import public and import weak. The Buf lint rules IMPORT_NO_PUBLIC and IMPORT_NO_WEAK enforce this for you by default.

buf.build/blog/totw-5-...

Cheap Kafka is cool. Schema-driven-development with Kafka is cooler. If you're looking for a modern drop-in replacement for Apache Kafka to save costs and complexity, Bufstream is probably your best bet. However, we've got a bigger mission here. Buf wants to bring sche...

Engineers shouldn’t have to define their network APIs in OpenAPI or Protobuf, their streaming data types in Avro, and their data lake schemas in SQL. A unified schema approach dramatically reshapes our world, solving data quality problems at the source.

buf.build/blog/kafka-s...

Read all about it in our latest blog post: buf.build/blog/bufstre...

A setup quite literally available nowhere else. No other product in the market even comes close.

Instant failover. Scales all the way from 0 to 100 GiB/s with no operational toll. 1/3 the cost of an Apache Kafka stretch cluster (which won't even work at these loads).

Bufstream: A multi-region, active-active Apache Kafka-compatible cluster tested to 100 GiB/s writes and 300 GiB/s reads.

A single-region Apache Kafka cluster handling this load would be challenging. A stretch cluster is unthinkable.

We will leave the cost of that to your imagination 😊

This setup is enough to handle 100 GiB/s writes and 300 GiB/s reads with active-active producers/consumers running in two regions simultaneously. It provides the best RPO/RTO guarantees in the industry with Spanner’s p99.999% multi-region uptime SLA and GCS’ p99 15 minutes replication SLA.

A detailed blog post is coming soon, but to give out some details:

• 108 n2d-standard-32 brokers with Tier 1 networking.
• Dual-region GCS bucket with Turbo Replication.
• 9 node multi-region Spanner cluster with read-write nodes in us-west1 and us-west2; witness nodes in us-west3.

Can you tell the difference between the end-to-end latencies of a multi-region and single-region Bufstream cluster?

On the left is the single-region cluster from our last post.

On the right is a test we completed with 100 GiB/s writes replicated through two (!) cloud regions entirely via GCS.

Check out our blog to see out how we did this: buf.build/blog/bufstre...

Bufstream: 100 GiB/s of writes and 300 GiB/s of reads at 350ms latency in GCP.

For a cost that’s literally 25x lower than the cheapest Confluent Cloud offers.

Bufstream’s stateless design gives you all of this, with the operational simplicity that lets you sleep at night.

What if I told you that a 1 GiB/s Kafka topic streamed directly into your S3 data lakehouse as an Iceberg table could cost you... $10/hr?

Bufstream does it. It's literally too good to be true.

I spent 20 hours researching them. Here's their story (2 minute read) 🧵