sqlite-zstd: Transparent dictionary-based row-level compression for SQLite - An sqlite extension written in Rust to reduce the database size without losing functionality - phiresky's blog
Motivation (or side side projects) While working on my startup (DishDetective) I started a side project of an automatic time-tracking tool (timetrackrs), where I try to collect a lot of information about my habits for later analysis. The main component collects data from my computer about what
Hasnain says:
This was a pretty interesting technical read. I’ve thought of doing similar stuff at the application layer in the past (a la Managed Compression @ meta) but it’s cool to see something like this for SQLite and in a manner that doesn’t need any application level changes.
“For some use cases, sqlite-zstd is great. It can reduce the size of your database by 50 to 95%. The performance impact is there, but considering most operations still run at over 50k per seconds you’ll probably have other bottlenecks. There’s other optimizations to be done
The same method should work for other databases, with barely any modifications required for e.g. PostgreSQL. I’m not sure why no one has done this before or maybe I just couldn’t find it.”
Posted on 2022-08-03T05:45:14+0000