Abstract:
On September 30, Claude developer Anthropic released a report on the 29th, saying that GLM-5.3 has strong vulnerability exploitation capabilities, but there are still gaps in security protection, which may lower the threshold for malicious attackers to use such capabilities.
In the ExploitBench test, GLM-5.3 completed end-to-end exploits 50 times out of 410 attempts, and Claude Mythos Preview completed 56 times. The test was conducted against offline targets in an isolation sandbox, and Claude, who participated in the capability comparison, turned off security protection. The Mythos Preview was released in April this year and is not the latest model of Claude.
ExploitBench breaks the vulnerability exploitation process into 16 stages, and gradually checks what steps the model can take after getting a known flaw.
On September 17, CAISI, a subsidiary of the National Institute of Standards and Technology, released an independent evaluation report, ranking GLM-5.3 as the one with the strongest network security capabilities among the open weight models released at that time. Compared with the strongest batch of American models at the time, it still had a significant gap: estimated by the comprehensive indicator of the CAISI network security benchmark, it was about four months behind.
This assessment covers tasks such as vulnerability discovery and exploitation, and compares both public models and versions available only to audited users.
Anthropic also used simulated scenarios to test whether the model would take on malicious tasks, without executing the generated code or connecting to a real system. GLM-5.3 rejected all direct malicious commands; after changing the prompt method or modifying the model, the proportion of accepting tasks increased to 64% to 100%.
Claude, who retained protection, did not accept malicious tasks in this set of tests. Because it does not open weights, researchers cannot modify the model using the same method for comparison.
In the GLM-5.3 release notes on August 14, Zhipu introduced three layers of protection: identifying and intercepting abusive requests outside the API, checking task intentions during the reasoning process, and then letting the model itself learn to reject dangerous requests. The company also made it clear at the time that after opening up the weights, what remains valid among these three layers is the security alignment of the model itself.
In that release note, Zhipu also proposed that powerful defensive tools should not only be in the hands of a few institutions, and planned to provide free credits to open source project maintainers, support security audits, and add code audit functions to ZCode.
Comments