Hunting orphan objects: 45% off our ClickHouse storage bill and a near data-loss
Tinybird significantly reduced its cloud storage costs by 45% after cleaning up orphaned S3 objects. The company faced challenges with data loss during the cleanup process, but successfully recovered all data. This experience highlighted the importance of improving operational safety and recovery procedures.
- ▪Tinybird cleaned up orphaned S3 objects, leading to a 45% reduction in storage costs.
- ▪The cleanup process nearly resulted in the loss of legitimate data, which was ultimately recovered.
- ▪The company improved its garbage identification tooling to better manage orphaned objects.
Opening excerpt (first ~120 words) tap to expand
We were paying for petabytes of S3 objects nothing was reading. Last month we cleaned them up and our object storage bill dropped ~45%. We also almost lost real data along the way. Here's what happened.How do we deal with cloud orphan objects?At Tinybird, we run large-scale ClickHouse® clusters backed by object storage. Like many teams operating distributed storage systems at scale, we’ve spent a lot of time thinking about replication, consistency, and failure recovery.One issue that kept growing in the background was cloud storage garbage: objects that were no longer being used, but also never deleted.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Tinybird.
Discussion
0 commentsMore from Tinybird
