Fastconstmap: A faster map from strings to integers in Python
Daniel Lemire introduced a new Python library called fastconstmap that provides a constant map from strings to integers. The library claims to store each key in only 9 bytes, dramatically reducing memory usage compared to standard dictionaries. It also offers faster lookups and supports serialization for reuse.
- ▪A standard Python dict with one million string keys can consume over 100 bytes per key.
- ▪fastconstmap stores each key in just 9 bytes, greatly lowering memory requirements.
- ▪Lookup operations with fastconstmap can be up to twice as fast as with a regular dict in some cases.
- ▪The library allows maps to be serialized to disk or transmitted over a network for later reuse.
Opening excerpt (first ~120 words) tap to expand
Daniel Lemire@lemireIn many applications, you need a map from strings to integers. In python, you might do it like so... d = {"apple": 100, "banana": 200, "cherry": 300} If you have 1 million keys, that can use a lot of memory!!! Like over 100 bytes per key! I have published a new library that key. That's right. Just 9 bytes. You use it like so: from fastconstmap import ConstMap d = {"apple": 100, "banana": 200, "cherry": 300} m = ConstMap(d) m["apple"] # -> 100 m.get_many(["banana", "cherry"]) # -> [200, 300] It can be significantly faster (e.g., 2x in some cases) than the a standard dict.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at X (formerly Twitter).
Discussion
0 commentsMore from X (formerly Twitter)
