TAG
#evaluation
Stories connected through a shared idea, actor, technology or theme.
Microsoft research highlights failures outside standard AI benchmarksThe discussion focuses on failures that emerge when AI systems are tested beyond conventional benchmarks and on the importance of examining the underlying data. It argues that evaluation must include surprising and real-world failure modes rather than relying only on headline scores.→Era by Eon exposes hidden-knowledge failures in enterprise agentsThe benchmark evaluates 12 agents on questions involving hidden facts distributed across generated company systems. The strongest agent answered 18 of 24 hidden-knowledge attempts correctly, while all agents together solved only two of 84 hardest attempts.→Twenty-one countries call for mandatory frontier-AI testingThe statement says frontier developers should publish transparent safety protocols and permit qualified independent evaluators to assess risks. It also proposes international coordination on oversight without restricting access to AI’s benefits.→OpenAI publishes principles for independent model safety assessmentsThe proposed assessments cover safety cases, safeguards, capability thresholds and misalignment incidents. OpenAI says evaluators should have access proportionate to their claims, disclose methods and retain editorial independence, while the company may request sensitive redactions.→Microsoft agent-learning 0.9.0 adds reproducible policy replay with input lineageagent-learning 0.9.0 adds policy audit and replay with consumed-input lineage. The release records which inputs contributed to a policy execution so the run can be reproduced and examined later.→ScienceIDE turns scientific software into training environments for agentsScienceIDE uses expert-defined cases, acceptance criteria and verification to generate interaction trajectories for supervised fine-tuning, reinforcement learning and evaluation. The accompanying PhAI-IDE models show gains on selected scientific-code repair and general reasoning benchmarks.→Pydantic AI Harness 0.30 adds live trajectory steering and GitHub workflow integrationThe release introduces TrajectoryJudge, which reviews a live run on a cadence and can steer it mid-run. It also adds filesystem traversal events, explicit OpenTelemetry decisions and a hosted engine definition for running repository-defined Pydantic agents in GitHub Agentic Workflows.→Google DeepMind pilots double-blind evaluations for proprietary modelsGoogle DeepMind announced a pilot of double-blind evaluations for a proprietary frontier model, running Gemini Flash Lite against confidential benchmarks in Google Cloud Confidential Space. Google says cryptographic separation keeps model weights private from evaluators and test prompts private from Google, with the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons participating.→Vals AI raises $40 million to expand model evaluationVals AI raised $40 million in Series A funding led by a16z, with participation from HRT Ventures, Next Ladder Ventures and existing investors. The company evaluates models on domain-specific tasks in areas including finance, law, healthcare and coding.→TRUST-AI convenes on trustworthy AITRUST-AI is scheduled for 15 and 16 August and received 45 submissions, with accepted short papers intended for online publication through CEUR-WS. The workshop focuses on trustworthy AI as systems move into a wider range of services and practices.→