Silk: A silky smooth fiber runtime for ClickHouse
Silk is a stackful-fiber library and scheduler designed for ClickHouse, a database engine. It aims to improve the performance of I/O-bound and highly concurrent workloads, such as object-storage I/O and distributed cache lookups. Silk provides a cooperative fiber scheduler with a per-CPU scheduler that uses io_uring for asynchronous I/O and steals work between cores when local queues run dry.
- ▪Silk is a C++ library that gives a cooperative fiber scheduler, backed by a per-CPU scheduler that uses io_uring for asynchronous I/O.
- ▪The library is designed to execute high-concurrency networking I/O and high-currency file I/O with low overhead.
- ▪Silk's benchmarks show roughly 3.6 nanoseconds per fiber yield with cross-CPU work stealing and 7.6 microseconds for an io_uring ping-pong.
Opening excerpt (first ~120 words) tap to expand
TL;DR # Silk is a stackful-fiber library and scheduler with a NUMA-aware work-stealing loop, io_uring as the I/O ground truth, and zero heap allocation in the steady-state hot path. We built it for ClickHouse, and the first place we aim to integrate it is in our distributed cache. What are fibers? What is Silk? # Fibers are a lightweight user-space execution unit, somewhat like threads. Unlike threads, fibers participate in cooperative multitasking instead of the preemptive multitasking that threads use; allowing fibers to yield their work instead of block on it. This particular behavior is best suited for asynchronous I/O, which is becoming more of a bottleneck in distributed systems as CPUs grow faster and clusters grow larger.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at ClickHouse.
Discussion
0 commentsMore from ClickHouse
