placeholder

Scaling Kafka at Honeycomb - Honeycomb

See how Honeycomb reduced Kafka cost by a total of 87% per megabyte/sec of throughput w/ AWS’s new Amazon EC2 lm4gn instance family and Confluent Tiered Storage.

Click to view the original at honeycomb.io

Hasnain says:

This was a great technical read.

"We hope hearing how we managed our Kafka-based telemetry ingest pipeline is helpful to you in scaling your own Kafka clusters. We’ve grown 10x in two years while TCO1 for Kafka has only gone up 20%—in other words, an 87%2 cumulative reduction in cost per MB of incoming data. We do not have any engineers dedicated full time to the care and feeding of Kafka, and have preserved about the same number of engineers fluent in Kafka debugging as needed through training with incidents. This means our toil has not significantly increased even as the data scope increased."

Posted on 2021-12-01T06:25:51+0000