Discussion about this post

User's avatar
Nan Deng's avatar

Active Memory across days is the claim I'd separate from the benchmark table. A model can look strong on a fresh finance or legal prompt and still fail once that memory is carrying last week's private notes into a new session. No system card was referenced at launch, so there is no public description of what the memory is allowed to retrieve or how a user deletes it. Are the third-party evals testing that memory path, or only a stateless prompt?

Stefan Mihaylov's avatar

The $0.20 cached input price matters more than it looks. At 7.69M input tokens per task, your cache hit rate will decide most of what a $4.52 run costs.

4 more comments...

No posts

Ready for more?