Show HN: Raptor – fast, energy efficient small file uploads to S3
Raptor is a dependency-free, native executable and serverless backend that enables fast, energy-efficient small file uploads to Amazon S3, outperforming the standard S3 CLI. It bypasses HTTP, TLS, and TCP by using UDP with RaptorQ encoding and optional WireGuard encryption, reducing latency and CPU usage. The system is designed for small files and leverages AWS serverless components for scalable, low-overhead file transfer.
- ▪Raptor transfers small files to S3 using UDP and RaptorQ encoding, avoiding the overhead of HTTP, TLS, and TCP.
- ▪The system uses a serverless AWS backend with SQS, Lambda, DynamoDB, and S3 for reconstructing and storing files.
- ▪Raptor's architecture minimizes CPU usage, improving energy efficiency and reducing datacenter load.
- ▪The CLI and Lambda functions are AOT-compiled for performance and fast startup times.
- ▪WireGuard encryption and Proxylity UDP Gateway handle secure packet delivery and routing.
Opening excerpt (first ~120 words) tap to expand
Raptor: Fast Small File Transfers to S3 This dependendcy-free native executable and serverless backend delivers small files to S3 fast. Much faster than the S3 CLI. Perhaps most importantly, it does so using a tiny fraction of the CPU time, saving energy and keeping our datacenters (and planet) a little cooler. Background Moving small files to S3 is slow, and largely remains so. AWS has done significant work to improve S3 throughput, including the Common Runtime (CRT) transfer client and CLI transfer configuration. But these optimizations only go so far. S3's API is based on HTTPS, and the expected floor for small object latency is ~100-200 ms per request.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at GitHub.