What's that Entry in My Log?
by David M. Piscitello, Core Competence, Inc.
Logs from Internet firewalls are vital sources of information. They provide a chronology of events that serves two purposes. First, logs provide a good picture of normal user behavior: what applications they use and when, where they visit, and how frequently. This information can help you determine how efficiently your Internet bandwidth is being used or whether it’s being misused. It can help you confirm that the outgoing security policy you seek to enforce is correctly implemented.
Logs serve another, equally important purpose. Log entries identify denied or blocked incoming traffic from the Internet, so you can learn quite a bit about the kinds of mischief or malice that’s directed at your site or company. Mischievous activity is rampant on the Internet. You have no doubt noticed dozens (if not hundreds) of deny messages in your logs. If you ignore all of these, you run the risk of overlooking a configuration error that’s preventing your users and customers from getting the most from your Internet connection. You also risk overlooking warning signs of an impending attack.
How should you respond to events that the Firebox logs? Unfortunately, there’s no quick reference table to guide you. But here’s a sample of the kinds of activities I see at my site, how I interpret them, and the measures I take when curious entries appear in my logs.
What’s normal?
Normal activity, even mischievous activity, will vary from site to site. Review
your logs regularly (I do so daily, but I have a small site). Get a good sense
of how frequently your firewall is being probed, and for what purposes. While
you are trying to get a sense of what is normal, log all incoming and outgoing
traffic, allows and disallows. Yes, this creates a lot of log data, but how can
you identify abnormal activity if you don’t know what’s normal?
While establishing what is normal, disregard who is probing your firewall. The only who information you have is an IP address, and the source IP addresses in packets used to scan your network are probably bogus: they are typically stolen identities used by the attacker but belonging to another site, or the IP addresses of hosts the attacker has already compromised. Worry about services attackers seek out: block incoming requests or protect these appropriately, and your site may stay productive and your critical information intact.
Let’s focus on incoming traffic. If you run a stringent security policy, you probably block all non-essential incoming connections to ports, thus keeping attackers at bay. Hopefully, you’re blocking the sample set of ports I’m about to talk about for incoming traffic, or you have good reason and alternative measures to prevent their misuse. Each example below illustrates an attack stage – a scan, service enumerations, an exploit, and backdoor/Trojan communication.
Scans
Hackers use the ping command line utility to scan blocks of
IP addresses and identify reachable hosts on the Internet, a.k.a. prey.
By default, your Firebox not only denies incoming ICMP Echo messages
used by the ping utility (Type code 8 in log entries 41178-416388,
below), but silently discards these as well as any ICMP message, including redirects
(in log entry 198088, below, type code 5).
"Silently" means the Firebox returns no information to the
ping's source. Attackers
focus their activity on hosts that respond, so keeping silent is one way to
avoid more attention.
411788 06/26/01 21:27:15 firewalld[74]
deny in eth0 28 icmp 20 115 63.26.93.47
64.53.71.51 8 0 (Ping)
412498 06/26/01 21:32:16 firewalld[74]
deny in eth0 28 icmp 20 115 63.27.236.238
64.53.71.51 8 0 (Ping)
413968 06/26/01 21:37:35 firewalld[74]
deny in eth0 28 icmp 20 116 63.207.61.34
64.53.71.51 8 0 (Ping)
415068 06/26/01 21:45:16 firewalld[74]
deny in eth0 28 icmp 20 112 216.93.21.99
64.53.71.51 8 0 (Ping)
416388 06/26/01 21:53:45 firewalld[74]
deny in eth0 28 icmp 20 118 63.162.51.134
64.53.71.51 8 0 (Ping)
. <snip> .
198088 06/29/01 11:53:44 firewalld[78]
deny in eth0 56 icmp 20 64 64.53.71.49
64.53.71.51 5 1 (default)
68898 06/30/01 12:11:47 firewalld[78]
deny in eth0 64 icmp 20 110 195.36.253.87
64.53.71.51 8 0 (Ping)
What should you do about these? Concede that scans are a fact of life -- everyone gets “ping-scanned.” I become concerned if I see an increased intensity or persistence of scans from a block of addresses beyond what I’ve established as normal. The security practice called Predictive Analysis classifies this kind of escalation as a possible close-in threat, a precursor to an attack. I reassure myself that my incoming traffic policies are as stringent as I intend by running a port scan against my Firebox. Depending on how persistent scans have become, I may put an Intrusion Detection System or LAN analyzer like Ethereal outside my firewall for awhile to capture traffic thrown at my network. Packet captures and IDS’s will reveal more information about the composition and streams of packets than your firewall logs, and provide complementary data to your logs should you initiate a criminal investigation.
Service enumeration … or
mis-configuration?
Block incoming NetBIOS/SMB packets (Ports 135-139, through the SMB packet filter
in your Firebox). Your logs will likely show denied incoming Name Query
requests:
38098 06/30/01 02:17:37 firewalld[78]
deny in eth0 78 udp 20 113 64.24.44.24
64.53.71.51 137 137 (default)
38118 06/30/01 02:17:39 firewalld[78]
deny in eth0 78 udp 20 113 64.24.44.24
64.53.71.51 137 137 (default)
189598 07/02/01 10:44:37 firewalld[78]
deny in eth0 78 udp 20 113 64.213.59.130
64.53.71.51 137 137 (default)
189628 07/02/01 10:44:39 firewalld[78]
deny in eth0 78 udp 20 113 64.213.59.130
64.53.71.51 137 137 (default)
189648 07/02/01 10:44:40 firewalld[78]
deny in eth0 78 udp 20 113 64.213.59.130
64.53.71.51 137 137 (default
This may be an attacker intentionally directing NetBIOS Name Service Registration packets (Port 137) at your network, hoping to elicit a response from your Domain Controllers. Or it may be a PC or laptop connected to a cable or dialup modem, with no firewall capabilities, unwittingly issuing Name Registration messages over a broadcast medium. Nuisance, or probe? Frankly, I worry when I don’t see these entries in my log, and double-check my configuration to verify that I still have ports 135 through 139 blocked.
Service enumeration attempts for *NIX systems are just as common, but much more likely to be hacking than misconfiguration. I see these more frequently; in fact, daily:
58008 06/28/01 19:19:42 firewalld[78]
deny in eth0 44 tcp 20 236 203.66.232.114
64.53.71.51 38711 111 syn (blocked port
58028 06/28/01 19:19:45 firewalld[78]
deny in eth0 44 tcp 20 236 203.66.232.114
64.53.71.51 38711 111 syn (blocked site)
58048 06/28/01 19:19:46 firewalld[78]
deny in eth0 40 tcp 20 236 203.66.232.114
64.53.71.51 38711 111 rst (blocked site)
10668 06/29/01 17:21:53 firewalld[78]
deny in eth0 60 tcp 20 40 202.31.233.57
64.53.71.51 2880 111 syn (blocked port)
20778 06/29/01 20:28:49 firewalld[78]
deny in eth0 60 tcp 20 47 61.73.62.114
64.53.71.51 4380 111 syn (blocked port)
40188 06/30/01 02:56:26 firewalld[78]
deny in eth0 60 tcp 20 50 4.22.115.5
64.53.71.51 3231 111 syn (blocked port)
53358 06/30/01 07:19:32 firewalld[78]
deny in eth0 60 tcp 20 47 24.9.187.104
64.53.71.51 2672 111 syn (blocked port)
142618 07/01/01 17:11:58 firewalld[78]
deny in eth0 60 tcp 20 46 217.57.101.211
64.53.71.51 2105 111 syn (blocked port)
172598 07/02/01 04:53:16 firewalld[78]
deny in eth0 60 tcp 20 45 202.153.44.9
64.53.71.51 3877 111 syn (blocked port)
184328 07/02/01 09:39:46 firewalld[78]
deny in eth0 40 tcp 20 21 61.218.196.106
64.53.71.51 111 111 fin syn (blocked port)
224218 07/02/01 22:34:54 firewalld[78]
deny in eth0 60 tcp 20 43 210.90.192.164
64.53.71.51 2212 111 syn (blocked port)
235918 07/03/01 03:12:22 firewalld[78]
deny in eth0 60 tcp 20 43 192.203.200.167
64.53.71.51 2070 111 syn (blocked site)
3248 07/03/01 15:33:57 firewalld[78]
deny in eth0 60 tcp 20 42 195.54.109.11
64.53.71.51 1678 111 syn (blocked site)
16818 07/05/01 10:05:02 firewalld[78]
deny in eth0 60 tcp 20 44 210.95.42.253
64.53.71.51 951 111 syn (blocked port)
In these cases, script-kiddies are looking for a *NIX host that’s running portmapper (rpcbind) at TCP port 111. For more on this attack, see Rik Farrow's article.
This log “thread” also shows that different TCP scans have been run against my Firebox. While the most common TCP scan uses the SYN (connect request) packet, both RST (Reset connection) and FIN (Finished, or close connection) flags can be used not only to identify listening ports, but to fingerprint the operating system being scanned. As you become familiar with probes and the tools attackers use, you’ll get a feel for what tools are being used against you. I’m guessing that the attacker at 203.66.232.114 is using the popular nmap scanner Rik has previously discussed.
DNS
The Domain Name Service is a popular target. The lack of authentication in DNS
makes it ripe for attacks, including cache poisoning, DNS response spoofing, and
buffer exploits that lead to root compromises (visit here
for more info). DNS zone information itself is a gold mine of information for an
attacker, and there are many tools available to initiate transfers, including DNSCMD.EXE
from the Windows Resource Kits. A zone transfer request like these from any but
your authorized secondary DNS server is a sure sign you’re being probed:
124598 07/01/01 10:35:12 firewalld[78]
deny in eth0 60 tcp 20 57 216.28.32.88
64.53.71.51 1878 53 syn (default)
243168 07/03/01 06:06:19 firewalld[78]
deny in eth0 60 tcp 20 52 212.188.63.245
64.53.71.51 3286 53 syn (default)
40958 07/05/01 12:56:12 firewalld[78]
deny in eth0 60 tcp 20 54 209.136.35.2
64.53.71.51 4688 53 syn (default)
47478 07/05/01 13:40:21 firewalld[78]
deny in eth0 60 tcp 20 53 209.194.244.42
64.53.71.51 2968 53 syn (default)
Zone Transfers are performed using TCP port 53 (client requests use UDP 53), so block incoming traffic to TCP port 53 if you run a single DNS for internal and external names, or allow incoming requests only from authorized secondary servers (perhaps your ISP). Larger networks should consider a split-DNS configuration, where you maintain a protected DNS with internal name records behind your Firebox, and an external DNS with only those name records you absolutely must reveal, like mail and Web hosts. I encourage anyone who administers a DNS to read Securing an Internet Name Server by Cricket Liu. For the merits of outsourcing DNS, see David Bonn's LiveSecurity article.
Root kits on your network?
Log entries like this one reveal potentially worrisome information and may
demand further action:
575678 06/27/01 15:13:45 firewalld[74]
deny in eth0 48 tcp 20 114 65.15.157.88
64.53.71.51 3084 27374 syn (default)
575698 06/27/01 15:13:48 firewalld[74]
deny in eth0 48 tcp 20 114 65.15.157.88
64.53.71.51 3084 27374 syn (default)
575738 06/27/01 15:13:54 firewalld[74]
deny in eth0 48 tcp 20 114 65.15.157.88
64.53.71.51 3084 27374 syn (default)
7778 06/28/01 14:58:51 firewalld[78]
deny in eth0 48 tcp 20 109 63.59.2.114
64.53.71.51 2104 27374 syn (default)
7808 06/28/01 14:58:57 firewalld[78]
deny in eth0 48 tcp 20 109 63.59.2.114
64.53.71.51 2104 27374 syn (default)
7858 06/28/01 14:59:09 firewalld[78]
deny in eth0 48 tcp 20 109 63.59.2.114
64.53.71.51 2104 27374 syn (default)
. <snip> .
14878 06/28/01 15:38:30 firewalld[78]
deny in eth0 48 tcp 20 112 66.30.8.58
64.53.71.51 3904 27374 syn (default)
14898 06/28/01 15:38:33 firewalld[78]
deny in eth0 48 tcp 20 112 66.30.8.58
64.53.71.51 3904 27374 syn (default)
. <snip> .
15018 06/28/01 15:38:52 firewalld[78]
deny in eth0 48 tcp 20 112 66.30.8.58
64.53.71.51 3904 27374 syn (default)
Incoming connection requests to ephemeral ports are always suspicious. These ports are usually numbered 1024 or higher, are not assigned to well-known services, but are used to bind clients to applications. The “best practice” here is: suspect any incoming request to an ephemeral port. In a situation like this, I first check a list of vendor-proprietary ports used by applications I permit (WatchGuard services, for example). I next visit a ports database and learn that port 27374 is used by a Linux root kit/backdoor installed by the Ramen Internet Worm, and the Windows SubSeven Trojan as well. I have Linux and Windows hosts on my network, so, I ran a quick scan to see if there’s a host on my internal network listening to this port. If I had found one, I’d have searched for online resources that describe Ramen and SubSeven and would have followed the prescribed countermeasures.
Wrap up
In this small sampling of log entries, I’ve identified
several kinds of denied traffic and suggest responses these should evoke from
you, the poor sod stuck with managing the firewall. Your mileage will vary, as
will the attacks on your network.
Regrettably, in some circles hacking has become a cool thing. There’s so much nuisance traffic. You can’t react to every blocked packet by doing reverse lookups on IP addresses, identifying the administrative contact, and complaining, and you wouldn’t like the responses if you did. You also can’t ignore your logs entirely. Today, you must be responsible for protecting your site and sensitive information. And the way legal trends are heading, soon you’ll be accountable for preventing your systems from injuring others. Keep in touch with your logs, make your network look as much like a “black hole” as possible by blocking all unnecessary services (in both directions), and your job will be much easier. ##
This article came about partly because of suggestions from LiveSecurity subscribers Boris Wetzel, Shanta Jayawardana, and Joanne Barthel. Was this article useful to you? What would you like our experts to write about? Let us know at lsseditor@watchguard.com.
To view all previous LiveSecurity articles, log in with your WatchGuard user name and password at www.watchguard.com/support.
Copyright© 2001, WatchGuard Technologies, Inc. All rights reserved. WatchGuard, LiveSecurity, Firebox and ServerLock are trademarks or registered trademarks of WatchGuard Technologies, Inc. in the United States and other countries.