How We Saved Millions in SSD Costs by Upgrading Our Filesystem - Heap
During COVID we experienced rapid growth in the amount of data we ingest. This post details some of the problems this caused us and how we solved them.
Hasnain says:
This is some pretty cool work enabled by new state of the art compression algorithms.
“When all was said and done we observed the following impact:
* Total storage usage reduced by ~21% (for our dataset, this is on the order of petabytes)
* Average write operation duration decreased by 50% on our fullest machines
* No observable query performance effects”
Posted on 2021-11-10T04:16:06+0000