Republished with permission from WatchGuard Technologies, Inc.

WatchGuard


Rootkits 101:
Rotten to the Core

by Lisa Phifer, Vice President, Core Competence Inc.

From Maslan to Mailbot, F0r0r to CoolWebSearch, malware programs have grown increasingly stealthy. Last year, use of "rootkit" techniques to evade conventional virus and spyware defenses skyrocketed 400 percent. Microsoft's Malicious Software Removal Tool now finds a rootkit on 14 percent of infected PCs, including half of all unpatched XP machines. Unfortunately, this recent growth represents just the tip of the iceberg, as rootkits are expected to play a major role in hardening next generation viruses.

Helping malware

Rootkit refers to techniques and tools used to cloak remote-controlled trojans, keyloggers, DDoS zombies, spam 'bots, adware, viruses, and other malicious programs, evading detection by security scanners, operating system utilities, and end users. Rootkits such as Hacker Defender and FU are not themselves malicious. They are simply a means to an end. Malware developers use rootkits to install "stealth" processes, files, registry keys, accounts, services, and device drivers, without the user's knowledge or consent. Rootkits not only inhibit discovery; they can also make unwanted programs difficult or impossible to remove.

Primitive rootkits date back to the early 90s, when intruders started over-writing UNIX commands like "ps" and "ls" and "netstat." Those revised command executables filtered the intruder's processes, files, and sessions out of response text, hiding them from system administrators. In other words, early rootkits were used to "get root" and then evade detection. It was not until the late 90s that comparable tools were developed for Windows NT.

However, malware rarely used rootkits until they were popularized by the Sony DRM scandal. Sony included an Extended Copy Protection (XCP) program on its music CDs. Although XCP had a legitimate goal (Digital Rights Management), it was installed without disclosure or consent and used rootkit techniques to deter removal. Worse, trojans such as Breplibot.b started taking advantage of the Sony-installed rootkit. By late 2005, CERT advisories, lawsuits, and adverse publicity forced Sony to recall XCP-protected CDs and issue an XCP removal tool. Nonetheless, the rootkit XCP used is still found on 5 percent of PCs cleaned by Microsoft's Malicious Software Removal Tool.

Hiding in plain sight

As rootkits evolved, they found better ways to elude the grasp of anti-virus, anti-spyware, host intrusion detection, vulnerability assessment, and system diagnostic tools. To understand how rootkits do their dirty work, let's examine several methods used to create hidden operating environments for Win32 (NT, 2000, XP, 2003) malware.

Like early UNIX rootkits, basic application rootkits overwrite regular Windows user-mode application binaries with trojaned binaries. To hide a given piece of malware effectively, the rootkit might need to overwrite many applications. This sledge-hammer technique is relatively crude and easily spotted by checksum-based PC security programs.

Instead of overwriting binaries, User Mode rootkits can hide objects by intercepting (hooking) application calls to high-level Windows API functions. Most Windows applications rely on stock functions for a wide range of basic tasks, such as displaying dialog boxes or driving devices such as hard drives and printers. The stock Microsoft functions are grouped conceptually and stored in shared libraries, referred to as Dynamic Link Libraries (DLLs). Every developer can learn the locations of functions within DLLs -- and every attacker can exploit this same knowledge. For example, many applications use the kernel32.dll FindFirstFile function to explore files. A rootkit may change the application's Import Address Table, replacing pointers to the location of kernel32.dll with rootkit DLL pointers. When the FindFirstFile function is invoked, the rootkit uses the kernel32.dll's FindFirstFile function, but filters the result to hide filenames that provide evidence of the rootkit's presence. This Windows API hooking technique can also be used to hide processes, sockets, services, and registry keys.

More sophisticated User Mode rootkits exploit the slightly lower-level Native API which invokes functions provided by the operating system's ntdll.dll library. For example, Task Manager (iexplore.exe) uses the ntdll.dll NtQuerySystemInformation function to get a list of active processes. A rootkit can hook Task Manager's call to ntdll.dll, then strip malware processes from the returned list.

Some User Mode rootkits patch a few bytes of in-memory Windows API code to insert a jump to the rootkit's DLL. As described above, the rootkit filters output objects before returning control to the compromised code, which then returns data to the application that invoked the Windows API or Native API. A malicious hacker can use in-memory patching to hide files, processes, services, drivers, registry keys and values, open ports, and disk space usage. (For a first-hand look at a rootkit called Hacker Defender using this technique, also known as inline function hooking, see the LiveSecurity videos, "Malware Analysis: Rootkits," parts 1 and 2.)

User Mode applications and DLLs access the kernel by using system calls to reach device drivers. Drivers have direct access to kernel data objects, including the Master File Table, the Registry Hive, and the kernel's active process table. Kernel Mode rootkits hook these system calls, patch these device drivers, or modify kernel data objects to alter results at the lowest level. For example, system call hooking can modify the process list returned by the kernel to ntdll.dll, instead of changing the list returned by ntdll.dll to Task Manager. Due to complexity and tight-coupling, Kernel Mode rootkits must take care to avoid crashing the compromised PC.

An increasingly popular technique used by Kernel Mode rootkits is Direct Kernel Object Manipulation (DKM). Instead of intercepting query and enumeration API calls, DKM modifies the kernel's own data structures. For example, a DKM rootkit might remove a malware process from the kernel's process table. It might use syntactically-invalid names to hide folders and files from the Windows NT file system (NTFS). Or it might manipulate the Registry Hive to prevent enumeration of hidden registry keys; for example, embedding a NULL character in a key's name or value, hiding what follows from Windows API RegEnumValue or Native API NtEnumerateKey queries.

These methods can be found in many Windows rootkits, including AFX (kernel32.dll hooking and patching), Vanquish (kernel32.dll patching), Hacker Defender (ntdll.dll patching), and FU (DKM). For instance, FU (a play on the UNIX "su" command) loads the driver msdirectx.sys, which can then be used to hide malware processes and device drivers. Unlike most rootkits, FU does not attempt to hide itself, which may explain why FU is the most oft-removed rootkit encountered by Microsoft's Malicious Software Removal Tool.

Hiding in the shadows

New stealth attack vectors are always being developed to better conceal both the rootkit and associated malware.

For example, the rootkits described thus far conceal malware by hiding persistent objects (stored on disk somewhere) or dynamic objects (held in memory). Malware developers are now starting to hide persistent objects in locations such as video card EEPROM, bad disk sectors, and other locations overlooked by most (if not all) security scanners.

Another new method is to modify the boot sequence of the compromised PC, loading a clean Virtual Machine (VM) on top of an infected host operating system. This essentially puts the victim computer in "the Matrix," since the clean virtual version of the machine answers all queries from the victim computer about itself. Files, processes, remaining hard drive space, registry keys, and almost everything else that could reveal that a compromise has occurred, appear normal. But outside the virtual environment (which the administrator did not install, and thus might not realize is there), the real, compromised OS directs the computer. SubVirt is one proof-of-concept rootkit that works by installing a Virtual Machine Monitor. A guest OS using Linux/VMWare or Windows/VirtualPC to run in a virtual machine cannot see SubVirt. But SubVirt (and any associated malware) has access to all VM activity and data.

Researchers warn that hardware-supported CPU virtualization extensions such as Intel VT-x might also be compromised in a similar fashion. Malware developers could conceivably load a rootkit hypervisor onto physical memory pages that cannot be seen by the multiple virtual operating systems running on that physical platform.

And finally, Windows Vista is set up to allow only digitally-signed code into the kernel. Will this stump attackers? Maybe not. At Black Hat Las Vegas in August 2006, security researcher Joanna Rutkowska introduced stealth malware, code-named Blue Pill, that allowed her to root Vista by placing it (unwittingly) into a virtual environment, subject to a hypervisor. However, since then Microsoft acknowledged the validity of her technique and has been developing ways to defeat it.

Rootkits are here to stay. Fortunately, as rootkits evolve, so do countermeasures that detect rootkits. Unfortunately, in this continuing game of cat and mouse, today rootkit developers are ahead of defense tools. But there are techniques and tools you can use to spot these stealthy interlopers, and many of the countermeasures cost you nothing. I'll list them in my next LiveSecurity column. ##

Direct link to Part 2.
"Rootkits 201: Countermeasures and Defenses"


      Copyrightę 2006, WatchGuard Technologies, Inc. All rights reserved. WatchGuard, LiveSecurity, Firebox and ServerLock are trademarks or registered trademarks of WatchGuard Technologies, Inc. in the United States and other countries.