Let's Talk about Benchmarks
SpacetimeDB initially claimed to process over 100,000 end-to-end transfer transactions per second, but later discovered and fixed performance issues that had made the original benchmark misleading. After improving its V8 threading model, SpacetimeDB now achieves 303,920 ± 4,712 transactions per second, surpassing its earlier results and outperforming Rust in their TypeScript benchmark. The company addressed criticism from competitors, defended the validity of its benchmark methodology, and emphasized its technical improvements and transparency.
- ▪SpacetimeDB initially reported processing over 100,000 transactions per second, but later identified and resolved internal performance issues.
- ▪After optimizing the V8 threading model, SpacetimeDB achieved 303,920 ± 4,712 transactions per second in its TypeScript benchmark, nearly doubling its original performance.
- ▪In the same benchmark, TypeScript outperformed Rust, which recorded 265,541 ± 940 transactions per second.
- ▪SpacetimeDB responded to criticism from competitors like PlanetScale, asserting that many claims against its technology were factually incorrect.
- ▪The company updated its benchmark visuals with error bars and introduced a 2x zoom feature to better display performance differences on standard screens.
Opening excerpt (first ~120 words) tap to expand
In March, we announced SpacetimeDB 2.0 alongside a benchmark that claimed SpacetimeDB could process over 100,000 end-to-end transfer transactions per second. The video and our posts on social media picked up a lot of attention, and in some cases drew criticism for what some felt was misleading benchmarking, especially from competitors whose products did not perform well in our testing. Recently, we discovered and fixed several issues in SpacetimeDB, which unfortunately do make the benchmark numbers we published at that time misleading, and for that we apologize. Thing is, they're just not misleading in the way you might expect. The problem wasn't with the benchmark, but with SpacetimeDB itself. It wasn't living up to its true potential.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Spacetimedb.
Discussion
0 commentsMore from Spacetimedb
