Active Memory across days is the claim I'd separate from the benchmark table. A model can look strong on a fresh finance or legal prompt and still fail once that memory is carrying last week's private notes into a new session. No system card was referenced at launch, so there is no public description of what the memory is allowed to retrieve or how a user deletes it. Are the third-party evals testing that memory path, or only a stateless prompt?
The $0.20 cached input price matters more than it looks. At 7.69M input tokens per task, your cache hit rate will decide most of what a $4.52 run costs.
Using the same model as both the generation agent and the CUA verifier worries me a bit, since the two can share blind spots. Before counting its passes, I'd feed the verifier a handful of runs I know are broken and see how many it actually flags.
Active Memory across days is the claim I'd separate from the benchmark table. A model can look strong on a fresh finance or legal prompt and still fail once that memory is carrying last week's private notes into a new session. No system card was referenced at launch, so there is no public description of what the memory is allowed to retrieve or how a user deletes it. Are the third-party evals testing that memory path, or only a stateless prompt?
Genuinely clarifying contribution. Looking at historical parallels provides much needed grounding for present debates.
The $0.20 cached input price matters more than it looks. At 7.69M input tokens per task, your cache hit rate will decide most of what a $4.52 run costs.
Using the same model as both the generation agent and the CUA verifier worries me a bit, since the two can share blind spots. Before counting its passes, I'd feed the verifier a handful of runs I know are broken and see how many it actually flags.
A sharp observation. The tension between theoretical elegance and operational robustness is always where genuine insight lives.