TAG
#benchmarks
Stories connected through a shared idea, actor, technology or theme.
Microsoft research highlights failures outside standard AI benchmarksThe discussion focuses on failures that emerge when AI systems are tested beyond conventional benchmarks and on the importance of examining the underlying data. It argues that evaluation must include surprising and real-world failure modes rather than relying only on headline scores.→Claude Sonnet 5.5 launches with a major coding-efficiency jumpAnthropic says Sonnet 5.5 is over 30% faster and costs up to 30% less per task than Sonnet 5. It scored 70.6% on Terminal-Bench 4.0, compared with 10.3% for Sonnet 5, while retaining a 1-million-token context window.→Era by Eon exposes hidden-knowledge failures in enterprise agentsThe benchmark evaluates 12 agents on questions involving hidden facts distributed across generated company systems. The strongest agent answered 18 of 24 hidden-knowledge attempts correctly, while all agents together solved only two of 84 hardest attempts.→AV-Comparatives tests endpoint security against AI-assisted attack toolingAV-Comparatives evaluated 14 endpoint-security products against 50 multi-stage attack scenarios, including tooling built with AI-assisted development techniques. Eleven products achieved its highest Certified Leader status under a 92% pass threshold.→Harvey raises $550 million and expands its legal-AI stackHarvey said the round was co-led by Lightspeed and Diffusion and would support proprietary model development and enterprise expansion. The company also introduced a post-trained open-weight model and Harvey LAB benchmark, while reporting coverage said it acquired agent-security startup Guardrails AI.→Vals AI raises $40 million to expand model evaluationVals AI raised $40 million in Series A funding led by a16z, with participation from HRT Ventures, Next Ladder Ventures and existing investors. The company evaluates models on domain-specific tasks in areas including finance, law, healthcare and coding.→CAR-bench tests whether agents recognise their limitsThe benchmark tests multi-turn task completion, hallucination resistance when capabilities are missing and disambiguation of ambiguous requests. Agents must chain tools and policies while interacting with a simulated user in a changing environment.→Travel-planning agents face a constraint-heavy evaluationThe challenge evaluates multi-day travel plans against spatial, temporal and financial constraints using tools such as flights and hotels. Its second track tests specialised tool-use plugins for commercial agent platforms, with final results scheduled for presentation at IJCAI-ECAI.→