Prefill and decode do not have to run the same weights. This repo holds prefillers: NVFP4 transformer blocks that are streamed off the SSD one layer at a time, paired to complement low-bit GGUF for decode.
The point is that the two phases want different things. Decode is memory-bound, has to be on device for fast generation and is accelerated by being small. Prefill is compute-bound, so there’s little point in compressing it below NVFP4 (compute-native on Blackwell) as long as it doesn’t occupy much memory, which we achieve via offloading.
Accuracy and time-to-first-token against device weight size
Auxiliary prefillers recover ~50% of the accuracy that aggressive quantization gives up, at close to no extra memory cost. Moreover, past a few thousand tokens, the offloading streaming disappears under compute, so it is not paid for in latency either
That’s actually a great idea.
It’ll need a big batch size to hide the latency though (so this only worth it past 8K context, or so?) and then it needs to reload the quantized weights onto the GPU.
It does suck that all paths to better inference are so bad. Bigger GPUs are crazy expensive. Good IGPs are rare. CPU RAM is cost prohibitive. Even NVMe drives are crazy expensive now. Hence there’s only so much that can be done if users have absolutely no options to buy.
But there’s good news, the whole scientific field seems to put a lot of effort into better quantization, which means you can run better models with lower resources. The last big innovations were:
So I am just waiting for stream smaller weights for the same quality, giving me better results in the end.