New NVIDIA paper says AI models do not always need huge amounts of past conversation history (context) to work well. They show that regularly shortening the history (compaction) keeps accuracy almost the same on hard, long tasks.
New NVIDIA paper says AI models do not always need huge amounts of past conversation history (context) to work well. They show that regularly shortening the history (compaction) keeps accuracy almost the same on hard, long tasks.
Sure if compaction was free then I’d do it all the time. One advancement I’d like to see is MoE models that use fewer active parameters/experts when compacting. We already have DeepSeek 4.1 that uses a different number of active parameters depending on what its doing (8b for prefill and 16b for decode) so it doesn’t seem unreasonable to use fewer for compaction. It’d be like using a REAP model, but without having to load the model separately. Or even just use the MTP/draft model for compaction.