Google researcher says Windows NT kernel design still dwarfs Linux

Abstract:

Laurie Kirk, a researcher who has worked as a reverse engineer at Microsoft for four years and now works at Google, has recently reignited the decades-long debate between Windows and Linux. She publicly stated that the Windows NT kernel is still an "engineering miracle" from an engineering design perspective, and even dwarfs Linux in many aspects.

Kirk emphasized that what she admired was not the Start menu, Copilot or advertisements of Windows 11, but the underlying NT kernel architecture of Windows. She believes that NT has established a relatively complete object model and security system from the beginning, while Linux has gradually added security mechanisms such as Capabilities, Namespaces, and SELinux on the basis of traditional Unix, so it appears more dispersed overall.

Kirk said on social platforms that if you give the simplest description from the perspective of a programmer, NT is closer to an "object-oriented" design and has a very strong security model from the beginning; in contrast, many of Linux's security capabilities were gradually added later. She believes that if a future-oriented operating system kernel is designed from scratch today, the final form will probably not look like Linux, but will be closer to NT, and may even be similar to some BSD branch.

Windows NT is actually the technical foundation for all major Windows versions since 1993, including today's Windows 11. Microsoft hired Dave Cutler in 1988, who had previously been responsible for VMS operating system development at DEC. According to Microsoft, a small team of former DEC engineers led by Cutler spent about six months developing the specification before actually starting to write code. The initial goals included portability, multi-processor support, and C2-level security certification.

Retired Microsoft engineer Dave Plummer also participated in the discussion. He pointed out that NT is not the first time Cutler has designed an operating system kernel from scratch. He has previously participated in RSX-11M and VMS, so NT is actually the third time Cutler has built a kernel from scratch.

However, NT does not simply replace VMS with a Windows shell. It was originally called NT OS/2, an operating system that emphasized portability. It was first developed for the Intel i860 processor, and then moved to the MIPS architecture. Microsoft eventually changed NT's main product direction from OS/2 to Win32. An important background was that Windows 3.1 sold 16 million copies in just six months.

One of the core things Kirk admired was NT's "object" design. In Microsoft's own technical terminology, NT is not an "object-oriented operating system" implemented in languages ​​such as C++ in the traditional sense, but uses an "object-based" architecture. Resources such as processes, threads, files, devices, registry keys, mutexes, jobs, and access tokens are all managed as objects.

NT's internal object manager is responsible for creating and destroying these objects, maintaining the object namespace, tracking which objects each process holds, and managing corresponding access rights. Different system components use these objects through the interfaces provided by the component to which the object belongs, so when the internal implementation of the underlying component changes, it can avoid affecting other components to a certain extent.

The easiest place for ordinary Windows programmers to come into contact with this design is the "handle". When an application opens a file, Windows does not simply hand the file itself to the application, but returns a handle and records the permissions that have been granted to the handle. When the program operates on the file later, the system will check based on these permissions. If a handle is copied, its permissions can be further reduced, but they cannot be increased out of thin air through the copy operation.

Kirk believes that this relatively unified management method for different system resources is very elegant and is also one of the important advantages of the NT architecture.

NT's security model is also based on resources, identities and permissions. After a user logs in to Windows, the system creates an access token, which contains the user's security identifier SID, the user group to which it belongs, and related system permissions. Processes started by users usually obtain corresponding access tokens, and each object that can be protected has a security descriptor, which includes an access control list that specifies which SIDs can obtain which permissions or must be denied.

When a process attempts to access an object, Windows matches the access token against the object's access control list and returns a handle with the appropriate permissions. In addition, security descriptors can contain system access control lists for auditing purposes to record successful or failed access to an object.

However, this does not mean that having a well-designed kernel automatically equals an absolutely secure operating system. Windows NT 3.5 once received a C2 security rating, but the certification environment at that time was a stand-alone computer without a network connection, and Microsoft also strengthened the default permissions for files and registry. As Windows enters the Internet era, drivers, default configurations, and compatibility requirements accumulated over decades will also have a huge impact on system security.

One of Kirk's main criticisms of Linux is that Linux uses multiple cooperating mechanisms for security and permission management. The questions she raised included UID, GID, ACL, cgroups, various security policies, SELinux, and file system permissions, etc., and believed that there was a lack of a unified and intuitive permission relationship diagram between these mechanisms.

However, this complexity is also part of the way Linux is designed. UID and GID are used to identify users and user groups, while file permissions and ACL are responsible for protecting files. Capabilities can separate the capabilities of traditional root accounts, such as allowing a web server to bind port 80 without obtaining full root permissions. Namespaces allow processes to see independent file system mounts, processes or network environments. Container technology is built on this mechanism. cgroups are responsible for limiting the use of resources such as CPU and memory, and SELinux and other Linux Security Modules can impose additional security policies on this basis.

Many of these mechanisms of Linux were indeed added gradually during the development of the kernel. For example, SELinux was originally proposed by the US National Security Agency in the form of an independent patch in 2001. Linux subsequently established the Linux Security Module framework so that different security models can be accessed through unified kernel hooks. Today, the Linux kernel already supports multiple security mechanisms such as SELinux, AppArmor, Smack, TOMOYO, and Landlock.

Therefore, there are actually significant differences in the design philosophies of the two operating systems. NT tends to centralize management around a unified object model, while Linux pays more attention to composability, letting different mechanisms take on different tasks, and allowing distributions, administrators, and application scenarios to combine themselves.

Kirk believes that this difference is especially worth revisiting today with the rapid development of artificial intelligence agents. The core question she raised was: "What exactly is an AI agent allowed to do?"

Traditional programs usually execute one by one according to instructions explicitly issued by the user, while AI agents may continuously perform thousands of operations in a few minutes, generate and run code by themselves, read files, and concatenate multiple permissions to complete complex tasks. Therefore, the operating system not only needs to determine "who" is running the program, but also needs to more clearly define what resources an AI agent can access, within what scope it operates, and how to record these behaviors.

Kirk believes that NT's object model can allow the operating system to establish clearer permission relationships for different resources and provide a more centralized and clear audit record when the AI ​​agent loses control. However, she also admitted that this is still an architectural hypothesis rather than a proven conclusion.

In fact, Linux has provided more and more tools for this problem. Landlock added in Linux 5.13 allows even unprivileged processes to actively limit the file system and network resources they can access. These limits can be inherited to child processes and can only be tightened further, not relaxed by the child process. Combined with seccomp, Namespaces, cgroups and various LSM mechanisms, Linux can also strictly isolate AI agents.

Microsoft itself is moving in a similar direction. Microsoft announced Microsoft Execution Containers, or MXC, at the 2026 Build conference, allowing developers to declare what resources the AI ​​agent can access, and Windows to enforce these restrictions during runtime. MXC can also create independent user accounts for agents, allowing the system to attribute each operation performed by the agent to a specific identity. Windows 11's existing Agent Workspace also uses ACL to restrict agent accounts so that their permissions do not exceed the user's own.

In other words, Microsoft is now using traditional NT mechanisms such as SIDs, access tokens, and ACLs to solve the problem of AI agent permissions. This is exactly where Kirk believes that the NT architecture has advantages. However, this does not mean that Microsoft has proven that Linux is not capable of AI agents. While Microsoft itself is advancing its AI operating system, it has also warned that AI agents may create new malware and security risks.

Kirk then proposed his most interesting alternative historical idea: Microsoft should actually launch an "Open NT" back then.

The Open NT she envisioned does not necessarily need to be completely open like GPL software, but allows large enterprises to modify specific components in the kernel while maintaining the basic security and compatibility standards set by Microsoft. For example, when Amazon was building the EC2 cloud computing platform in the early days, it could derive a version called "AmazonNT" from Open NT and modify the scheduler, network stack or memory allocator on its own, while still complying with the core compatibility and security specifications defined by Microsoft.

In fact, Microsoft has indeed tried a similar model to a certain extent in the past. Microsoft's Shared Source program has provided access to Windows source code to approximately 1,600 enterprise customers, universities and government agencies. In 2001, the Austrian Ministry of the Interior became the first European government to obtain the Windows XP source code.

In 2006, Microsoft also launched the Windows Research Kernel, which allows university researchers to modify NT's scheduler and memory manager for teaching and research. However, these projects are still limited source code sharing in nature and do not allow companies such as Amazon to create and commercially distribute their own Windows NT branches.

As for why Microsoft has not further opened up NT, the article believes that it may involve licensing income, intellectual property rights, technical support costs, and Microsoft's most important commitment to Windows compatibility. Allowing third parties to modify the kernel for a long time is likely to mean that Microsoft needs to face a large number of compatibility and security issues between different versions.

Linux developers have questioned the Open NT vision from another angle. David Airlie, who has long been involved in the development of the Linux kernel graphics subsystem, believes that the real problem is the long-term cost of maintaining a kernel branch. Even if the source code is fully open, if companies still need to maintain their own modified memory managers or schedulers 20 years later, they must continue to invest in dedicated engineering teams.

Airlie pointed out that a large number of companies have tried to fork Linux versions, but after a few years they often find that the cost of maintaining their own branches is too high, and ultimately choose to submit changes back to the mainline. The reason why different JVM implementations in the Java ecosystem can exist for a long time is because they have clear commercial customers and revenue sources behind them; while an NT branch that only serves internal enterprises may be difficult to bear such costs in the long term.

If Amazon does modify NT's scheduler, then the security fixes that Microsoft releases each month will have to be re-merged, tested, and verified. The longer time passes, the closer this branch becomes to an operating system kernel team that needs to be maintained independently. The Linux model is to submit a large number of changes required by enterprises back to the mainline as much as possible, and to be jointly maintained by the entire community.

NT itself is not without historical baggage. Some engineers pointed out that the consistent design of VMS was not completely transferred to modern NT. The Windows registry has long been one of the most controversial system components among engineers.

Another controversy arises from the graphics system. During the Windows NT 4.0 period, Microsoft moved Window Manager, GDI and graphics drivers into the kernel space to improve graphics performance. This means that a severely problematic graphics driver can directly cause the entire operating system to crash. Windows 2000 then further added a large number of mechanisms such as Windows Driver Model, Plug and Play, power management, WMI and job objects.

Therefore, the NT kernel in Windows 11 today is no longer the same kernel as when NT 3.1 was first released in 1993. What Kirk really praised was some of the basic architectural concepts when NT was founded, rather than thinking that all designs of modern Windows are inherently superior to Linux.

Judging from historical results, Linux eventually achieved a position in the server and cloud computing fields that NT failed to achieve. Linux's open license enables any organization to modify the kernel, run it on a variety of hardware, and submit improvements back to the mainline. The subsequent emergence of containers, development tools, and huge ecosystems such as Android have further strengthened this development model.

What’s interesting is that Microsoft has been working hard to make Windows run Linux better in recent years. Windows Subsystem for Linux continues to receive performance and network improvements, and Microsoft has also launched features such as WSL Containers, allowing users to run Linux containers directly in the Windows environment. Google has also begun to add WSL support to its own AI tools.

At the same time, Kirk also mentioned BSD. She believes that if the operating system kernel is redesigned today, in addition to NT, a BSD-like architecture may also appear. She believes that the overall structure of BSD is relatively neat, and it has isolation mechanisms such as Jails in its early days.

FreeBSD Jails can limit the file systems, users and network environments that a process sees, while Capsicum is closer to the capabilities model emphasized by Kirk. After entering Capability Mode, the process cannot access the global namespace at will and can only use permissions explicitly granted to itself through file descriptors. Interestingly, the Capsicum project was completed at the University of Cambridge and received research funding from Google.

From this perspective, what Kirk is really discussing is not who is better in all scenarios, Windows or Linux, but a more basic issue: how the operating system should represent and control "permissions."

NT's idea is to let resources have clear types, attach access permissions to resources, and check and record them uniformly by the operating system as much as possible. Linux starts from the more flexible Unix foundation and meets the needs of different environments through multiple mechanisms that can be combined. Neither method alone can ensure system security.

The emergence of AI agents is making this issue even more important. In the past, the most important question for operating systems was to confirm "which user" is performing the operation; in the future, it must further answer who an AI agent can represent, what resources it can access, how long it has permission, what operations it can perform, and whether the system can accurately prove what it has done.

Windows NT did not become the dominant player in the server and cloud computing world, and that position was ultimately taken by Linux. However, it has been more than 30 years since the advent of NT 3.1. Windows is still using the original design concepts of objects, handles, access tokens and ACLs, and Microsoft is now beginning to use these mechanisms to solve the permission isolation problem of AI agents.

Therefore, the "Open NT" envisioned by Laurie Kirk may only exist in fictional history forever, but as computers begin to actively perform more and more tasks for humans, an increasingly real problem is facing all operating systems: When software begins to act on behalf of humans, how far should we allow it to go?