Read Microsoft's diagnostic benchmark with its limits
The 2025 result compared an AI system and physicians on a specific set of difficult case records.
Originally posted · Published here · 1 source · Resume reading
Microsoft reported in June 2025 that its AI Diagnostic Orchestrator, paired with OpenAI o3, solved 85.5% of a benchmark drawn from 304 New England Journal of Medicine cases. The comparison group of 21 physicians averaged 20% on completed cases [1].
The conditions matter
The physicians worked without colleagues, textbooks or generative AI. Microsoft explicitly noted that those resources can be part of normal clinical practice [1]. The result therefore describes a particular evaluation, not a general finding that AI is four times better than doctors at healthcare.
Microsoft also said more testing was needed on common presentations and in real clinical environments. Its announcement described the benchmark and orchestrator as research demonstrations [1].
What to take from the comparison
The useful question is how a system performs on a defined task under stated conditions. Before comparing percentages, check which cases were used, what information participants received and which tools each side could access.
Our original caption compressed the result into a broad accuracy claim. The narrower reading is more informative: this configuration performed strongly on this diagnostic benchmark. It does not establish that a consumer chatbot should diagnose or treat a person without clinical care.
Keep this guide
Filed under AI agents
Sources
Next in this stack
Keyword LOOP
Five rules for building an AI agent that works
Keyword STARTER
Write skill metadata an agent can recognize
Keyword SIMPLEFIRST
Choose an AI workflow before building an agent