Protecting Windows Servers against DoS Attacks
By Dave Piscitello, President, Core Competence
Windows servers are the lifeblood of many small and medium business networks. Besides hosting Web and business applications, Windows servers often host network and security management and logging systems. Most small-to-medium businesses (SMBs) dutifully protect their servers against TCP/IP attacks in general, and denial of service attacks in particular, by placing them behind Internet firewalls. However, many SMBs overlook the additional layer of security they can achieve by configuring the Windows OS to detect and protect its TCP/IP "stack" against attacks. These measures are especially important in defending servers from Denial of Service (DoS) attacks that originate from internal networks.
Origins of internal attacks
Internal attacks originate from disgruntled employees, right? Actually, a more likely source of internal attacks is malicious (viral) code delivered via e-mail. In addition to code designed to propagate the virus, e-mail message bodies and attachments can contain many kinds of malicious code. In particular, the virus strains known as blended threats can turn compromised computers into DoS zombies. Infected laptops are also vectors for malicious code. Mobile workers who fail to maintain virus and malware protection while on the road can expose the organization to an internal attack when they connect to internal networks, either directly or via VPN tunnels. Unsecured "convenience" or guest WLANs can provide vectors for employees, guests, and contractors, unless you carefully separate them from internal networks.
Why single out DoS attacks?
(A historical perspective)
Slammer and Nimda offer ample proof that DoS attacks are among the most debilitating attacks an organization can suffer. The most publicized of these attacks came in through ports left open at firewalls, but they potentially can be insider attacks as well. Historically, attacking an opponent behind enemy lines is an excellent way to demoralize his troops, disrupt his supply lines, and distract him from his military objectives. John Hunt Morgan's raids through Ohio during the American Civil War provide a good example of this tactic. A DoS attack on your internal networks is potentially as disruptive. Think of your Internet firewall as your primary line of defense. The DoS attack occurs behind your firewall, circumventing its countermeasures exactly as Morgan's raid did to the Union army's skirmish line.
Some military raids are intentionally disruptive. Morgan's raiders stole every horse within a ten-mile radius of their path, tore down telegraph lines, and in at least one case, burned a town. Similarly, some DoS attacks seek multiple targets, indiscriminately flooding all internal networks and servers. Other raids have one specific target and objective, e.g., the 1945 American paratrooper raid on the Japanese internment camp in Los Baños , Philippines to rescue prisoners and internees. Similarly, an attacker's blended threat code might probe for one specific service (for example, a server hosting Active Directory; or an Intranet Web server) and then launch a DoS attack against only those qualifying systems it discovers.
DoS attacks aren't the only internal attacks you should counteract, but defenses exist to help contain them, so you should consider implementing these measures on your Windows Servers.
Implementing DoS defenses on Windows Server
DoS attacks come in many forms. One of the more prevalent attacks, SYN flooding, exploits TCP synchronization. Windows Server 2000 and 2003 both offer protection against SYN floods, but the advanced configuration of this measure is not enabled by default. To understand how defenses against SYN flooding work, let's begin with a short primer on the attack itself.
In a SYN flood attack, the attacker submits TCP connection requests (TCP SYN packets) faster than your server can process them. Your server must respond to an incoming TCP SYN packet with a SYN-AK packet and wait for a reply ( TCP AK ) to complete TCP's three-way handshake. If no reply is forthcoming, your server suspects that it is lost, and retransmits the SYN-AK. The attacker knows this, so he withholds replies to your server's TCP SYN-AKs and continues to request additional TCP connections, expecting that your server will persist in retrying, which keeps the connection "half opened." Unless you configure your server to detect this hostile activity and respond appropriately, the attacker will force your server into devoting its resources for connections that never complete. That leaves zero resources for connections your legitimate users might wish to make.
To defend against SYN flooding, you can enable SYN attack protection and set several thresholds in the Windows Registry to protect your system against resource depletion.
Use a registry editing tool (e.g., regedit.exe) to make a copy of the Registry, so you can restore it in case you make an error while configuring DoS protection.
From your registry editing tool, navigate to the following Registry Subtree and key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services
Open the Tcpip/parameters key. Locate (or create) a DWORD, SynAttackProtect . Set the value of this DWORD to two (2), to enable advanced DoS protection. Then, to complete the configuration of advanced protection, add the following four DWORDs:
- TcpMaxHalfOpen
- TcpMaxHalfOpenRetried
- TcpMaxPortsExhausted
- TcpMaxConnectResponseRetransmissions
I explain each, below.
- TcpMaxHalfOpen. Use this DWORD value to set the maximum number of TCP connections you will allow in the "half opened" state before you begin applying SYN flood protection. The commonly recommended value for servers is 500. If you run a very busy server, and you receive complaints about blocked connections, you might want to increase this value.
- TcpMaxHalfOpenRetried. DoS attacks send SYNs, but nothing else. Thus, an unusually large number of connections in the "SYN received" state can indicate that a DoS attack is in progress. Use this DWORD value to identify the maximum number of TCP connections in the SYN_RCVD state for which at least one retransmission has been sent. The commonly recommended value is 400.
- TcpMaxPortsExhausted. This DWORD identifies the maximum number of SYNs your server processes before it refuses new connections. Microsoft recommends setting the value to five. Although other resources suggest that you set it to zero (0), five should suffice.
- TcpMaxConnectResponseRetransmissions. When a DoS attacker sends a SYN packet to your server, your server replies with a SYN-AK packet, marks the TCP connection as half opened, runs a timer, increments a counter, and waits for a reply (AK). This process repeats until the count of SYN-AK retransmissions exceeds the value you specify for this DWORD. This DWORD is thus extremely important because it essentially controls how much time and resources your server wastes on bogus connection attempts. Microsoft recommends a value of two (2).
[Note: Some apparently dated pages at Microsoft TechNet state that SynAttackProtect may only be set to zero (0) or one (1). If (1) is chosen, then SYN flood protection is invoked when TcpMaxHalfOpen and TcpMaxHalfOpenRetried values are exceeded. Other Microsoft TechNet pages, and general resources as well, describe the use of the value (2) to invoke the advanced SYN flood protection.]
Conclusions
With a little effort, you can add a useful layer of defense against external DoS attacks and, in the process, shut the door on internal DoS attacks. For additional protection, harden your Windo ws Server's TCP/IP stack as prescribed in Microsoft's Knowledge Base Article #324270 and read some of the resources at the end of this article. Even if your IT security budget feels tight, you can get the edge on the DoS battle with an investment of time instead of cash. ##
Resources
Protect against SYN floods (WinGuides)
How To: Harden the TCP/IP Stack (MSDN)
How to harden the TCP/IP stack against Denial of Service Attacks in Windows Server 2003 (Microsoft Support)
Hardening the Windows TCP/IP Stack (AntiOnline)
Windows TCP/IP Stack Hardening Tool (SecuriTeam)
Threats and Countermeasures Guide (Microsoft TechNet)
Security Considerations for Network Attacks (Microsoft TechNet)
Copyright� 2005, WatchGuard Technologies, Inc. All rights reserved. WatchGuard, LiveSecurity, Firebox and ServerLock are trademarks or registered trademarks of WatchGuard Technologies, Inc. in the United States and other countries.