Sophon PFG-1: a monolithic-3D AI ASIC with 330 GB of on-die DRAM and no HBM
Energy-Constrained Ceiling on Model Size As transistor scaling slows and data-center power becomes the binding constraint, the practical ceiling on deployable model size is set not by silicon area but by the energy infrastructure — the power a grid, campus, or rack can deliver and cool. An architecture can dominate one regime without dominating the other, so we treat each in turn. Inference (serving) ceiling — linear in N, memory-bound Under a fixed power budget, the largest model an architecture can serve is fixed by its energy per generated token.
- ▪Energy-Constrained Ceiling on Model Size As transistor scaling slows and data-center power becomes the binding constraint, the practical ceiling on deployable model size is set not by silicon area but by the energy infrastructure — the powe
- ▪An architecture can dominate one regime without dominating the other, so we treat each in turn.
- ▪Inference (serving) ceiling — linear in N, memory-bound Under a fixed power budget, the largest model an architecture can serve is fixed by its energy per generated token.
Opening excerpt (first ~120 words) tap to expand
8. Energy-Constrained Ceiling on Model Size As transistor scaling slows and data-center power becomes the binding constraint, the practical ceiling on deployable model size is set not by silicon area but by the energy infrastructure — the power a grid, campus, or rack can deliver and cool. A model's lifetime energy splits into two regimes that scale differently and are bounded by different figures of merit: a recurring inference (serving) cost that is memory-bound and grows linearly with parameter count, and a one-time training cost that is compute-bound and grows roughly quadratically with model size at compute-optimal data. An architecture can dominate one regime without dominating the other, so we treat each in turn.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Phantafield.
Discussion
0 commentsMore from Phantafield
