Discussion about this post

User's avatar
Nan Deng's avatar

The Muse Spark Open line I'd separate is the index versus the agentic evals. It leads open models on AA-Omniscience and ranks last among the top nine on AA's four agentic evals, and the same writeup says it uses 40k output tokens per index task, the fewest in that group. Those can be the same behavior or two different ones. A short completion looks efficient on the index and still fails a loop that has to notice a bad tool result and try again, so I'd want the agentic suite reported with the output budget, not only the score, before reading the token count as a win.

Raghav Mehta's avatar

The Vals note that input tokens drive most of Muse Spark Open's cost is the line I'd underline. At $1.25 per M fresh input and $0.15 cached, an agent loop that resends a long tool and history prefix every turn lives or dies on cache hit rate, not on the 40k output tokens per task. So cost per task depends a lot on whether the eval harness keeps that prefix stable across turns.

Also interesting that the NVFP4 checkpoint quantizes only the experts and still drops from about 1.6TB to 530GB. With 8 of 256 experts active, nearly all the weight mass sits in the expert FFNs, so attention and the router can stay in BF16 for very little memory. Has anyone seen agentic eval numbers for that checkpoint against BF16?

3 more comments...

No posts

Ready for more?