Abstract:
Microsoft announced that its artificial intelligence medical diagnosis system Diagnostic Orchestrator has achieved better results than human doctors in complex case tests. Microsoft AI CEO Mustafa Suleiman said that the company is moving towards the goal of medical superintelligence and predicts that in the next few years, the performance of artificial intelligence systems in the medical field is expected to reach a level that is indistinguishable from human experts.
The research results were published in the international academic journal "Science" on October 22. Microsoft compared its self-developed AI diagnostic system with human doctors through a series of medical cases and emergency department data, and established an evaluation benchmark called SDBench. Different from traditional tests that only require answering a medical multiple-choice question, this benchmark requires participants to gradually analyze the condition, ask questions, select examination items, and finally give a diagnosis result, trying to simulate the diagnostic process of a clinician more realistically.
The core idea of Diagnostic Orchestrator is to let multiple AI agents work together to simulate a consultation team composed of doctors with different professional backgrounds. The system will determine what questions to ask and what tests to conduct next based on the patient's symptoms and medical history, and will continuously adjust its judgment based on newly acquired information until a final diagnosis is formed. Microsoft hopes to break through the limitations of a single AI model in complex medical reasoning.
In tests based on hundreds of case reports published in the New England Journal of Medicine, the system's diagnostic accuracy was about 83%. By comparison, general practitioners who had access to reference materials were accurate about 36% of the time, and doctors who couldn't access references were about 22% accurate. In another set of cases from the emergency patient database of Beth Israel Deaconess Medical Center in Boston, the AI system achieved an accuracy of 80%, while the average accuracy rate of human doctors participating in the comparison was about 47%.
In addition to higher diagnostic accuracy, Microsoft also said that the system was often able to produce diagnostic results at a lower estimated cost in tests. One reason is that the system can select inspection items based on existing information, rather than performing a large number of inspections indiscriminately. However, these cost results are based on uniform estimation methods across studies and do not mean that the system has been proven to directly reduce actual hospital operating expenses in real-life healthcare settings.
The system currently provides underlying capabilities by OpenAI's GPT-5, but Microsoft emphasizes that it is not just the basic model itself that really plays a role, but also the orchestration mechanism responsible for organizing the diagnosis process, coordinating multiple agents, and verifying the inference results. The research team said that even if the basic model continues to be upgraded, multi-agent collaboration may still bring additional benefits. The system was initially developed when GPT-4.1 was still an advanced model, and subsequent tests also showed that improvements in model capabilities do not necessarily weaken the value of the collaborative diagnosis architecture.
However, these results still need to be viewed with caution. Testing has mainly focused on past cases and selected complex diagnostic cases, and it has not been able to prove that the system can achieve the same performance in real clinical settings. Research also needs to further evaluate its reliability in the face of common and common symptoms, as well as its performance under real-life conditions such as incomplete information and changing conditions. Microsoft also admitted that for the system to be truly put into clinical use, it must undergo rigorous clinical verification.
Suleiman sees this research as an important step for Microsoft towards "medical superintelligence." What Microsoft calls medical superintelligence not only allows AI to surpass human doctors in certain diagnostic tasks, but also hopes that future AI systems can provide capabilities that surpass the traditional medical system in many key aspects of medical services.
Dominic King, who is in charge of Microsoft's AI medical business, said that diagnosis is only part of the company's medical strategy. Microsoft also hopes that AI can help coordinate clinical care, develop treatment plans, and handle complex issues involving patient communication and human interaction in medical services. Related research and company articles also explore how AI can take on broader tasks in the healthcare system than just identifying diseases based on symptoms.
Microsoft plans to work with Beth Israel Deaconess Medical Center to conduct tests in a hospital environment to further observe how the system performs in real-life medical procedures. King said work could begin in the coming months, but Microsoft has not yet announced a firm timeline. As for commercialization, the company similarly did not give a specific date or product launch plan. The medical field involves strict regulatory, compliance and patient safety requirements, and the system must complete necessary verification before it can gradually expand its application scope.
Suleiman predicts that there will be some impressive practical deployment cases of intelligent systems in the coming year, and believes that AI will significantly change the way medical services are provided. However, he emphasized that artificial intelligence does not mean that doctors will disappear from the medical system. A doctor's job involves more than just diagnosing, it also involves processing ambiguous information, building trust with patients and families, and making comprehensive judgments based on specific circumstances. Microsoft hopes that AI can reduce the time doctors spend on repetitive and tedious tasks, allowing them to focus more on patient care.
If Microsoft's research results can be verified in a real clinical environment, the multi-agent AI diagnostic system may become an important auxiliary tool for doctors to deal with difficult cases, and have the opportunity to improve the identification efficiency of complex diseases. But moving from laboratory and past case testing to daily medical practice still requires clinical validation, regulatory review and long-term safety assessment. At this stage, this research shows the potential of AI-assisted diagnosis, but it cannot be concluded that artificial intelligence can universally replace doctors.
Comments