Pgrust update: at 67% Postgres compatibility, and accelerating

pgrust update: at 67% Postgres compatibility, and accelerating It’s been a week since I published the original pgrust post. pgrust is my attempt to rewrite Postgres in Rust. My ultimate goal is to build a database that is safer to work with so that I can work on improving Postgres. pgrust now passes about two thirds of the queries in the Postgres regression suite, up from one third a week ago. The codebase is over 450k lines. I have 8 Codex accounts running in parallel at $200/month each, and 280 PRs have merged in the last two weeks. I expected progress to slow down by now. Instead the opposite has happened. I’m now moving an order of magnitude faster against the Postgres test suite than I was when working on the foundational work, even when weighting against token usage. Building a Factory The easiest way I can describe what the experience has been like is by comparing it to building a factory. If you’ve ever played the game Factorio, you know exactly what I’m talking about. My role in building pgrust has become less about hand writing code and more about designing a system that produces the output I want, then optimizing that system to move faster. Things will break all the time and it’s my job to fix the system so it doesn’t break again. A lot of my work has been dealing with bottlenecks in the factory and there’s been a number of really interesting ones I’ve dealt with. For most of the last week, my laptop has been at 80–100% CPU. As it turns out, running multiple instances of the Rust compiler while running multiple instances of a database at the same time can consume a lot of CPU. Even with my beefy macbook pro with 14 CPUs, I’m still frequently maxing out the CPU. When you’re running 20 agents at one, it’s not hard to max out the CPU on your computer. I ended up spending a lot of time over the past week optimizing CPU usage. I did some profiling into where the CPU was going during builds. From the profiling I did, compiling the parser made about half of the time it took to build pgrust. Moving the generated grammar code into its own compilation unit dropped the time it took to rebuild the rest of pgrust from about 60 seconds to 36 seconds. That sounds small, but it gets multiplied by every test every agent runs. After this change, I could run more agents at once. One issue I was not expecting was running out of disk. Because I’m using git worktrees, each agent will get it’s own directory it works in. Initially that was resulting in every agent getting its own build artifacts as well. With each agent building pgrust multiple times over it’s lifecycle, I was ending up with over 100GB of build artifacts before I knew it. I now have a dedicated target directory for all my agents to share to ensure build artifacts are shared across agents. The biggest win overall ended up being setting up a CI workflow. Previously to move quickly, I would have my agents make small changes, run a subset of the tests, and then merge their code into main. This meant the agents could move quickly, but also resulted in the build breaking all the time. Often at the end of the day, it would take a couple of hours of agents running in the background to merge all the code together and fix any merge conflicts and bugs that were introduced. I now have a GitHub merge queue set up that runs before code makes it into main. This allows my agents to still maintain a fairly high velocity, while also making sure the code that makes it into main passes all checks I…

This excerpt is published under fair use for community discussion. Read the full article at malisper.me.