A Rust-Python thing I am working on. Apache 2 licence
Nairobi OS is a high-performance distributed data science infrastructure designed for efficient processing of large datasets. It utilizes a Rust-based refinery daemon and kernel-level features to achieve low overhead and high throughput. The system provides a Python API for ease of use while maintaining performance through specialized components.
- ▪Nairobi OS leverages io_uring and Huge Pages for zero-copy data ingestion.
- ▪It includes a high-performance Rust core for data analytics and a headless rendering engine for visualization.
- ▪The system is designed to operate in constrained environments such as Edge and IoT.
Opening excerpt (first ~120 words) tap to expand
English | 简体中文 | Español | Deutsch Nairobi OS Overview Nairobi OS is a high-performance, distributed data science infrastructure designed for extreme resource efficiency. It enables processing of massive datasets in constrained environments (Edge, IoT, Serverless) by leveraging a specialized Rust-based refinery daemon. By utilizing kernel-level features such as io_uring, memfd, and Huge Pages, Nairobi OS achieves sub-millisecond IPC overhead and zero-copy data pipelines. Key Features Zero-Copy Ingestion: Hardware-accelerated data loading using io_uring and 1GB Huge Pages. Hardware-Accelerated Visualization: Interactive Jupyter plotting via the Lagos Vision engine (wgpu and egui). Fused Analytics Pipeline: Ingest, crunch, and correlate data in a single D-Bus round trip.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at GitHub.