Debugging Mobile User (IPsec) VPN
by David Piscitello, Core Competence, Inc.
Though VPN technology has been around longer than most people realize, it hasn't come into broad usage until the last couple of years. In addition, the IPsec standard is not exactly an example of seamless elegance. The result is that VPN technology, while improving steadily, is still complex and time-consuming to install.
My previous article, "Deploying Mobile User (IPsec) VPN," offered help and guidance for administrators in the early stages of providing remote users VPN access to a network. This article covers the next phase: if you've begun installing VPN and the connections aren't coming together, here's where you might find help. I'll suggest some preventive measures that can ease VPN implementation, and show examples of how to interpret your Firebox logs when remote access fails.Troubleshooting
MUVPN Connections
If the security policy implemented on your Firebox
blocks incoming ICMP messages, it may be hard to confirm network connectivity
between remote users and a Firebox before you attempt to create VPN tunnels.
Have the remote user ping the closest point he can reach on the public side of
the Firebox (for example, your Internet access router); as an alternative, try
to ping the public IP address of the remote user from behind the Firebox (beware
that the client's address may change over time!). Other speed bumps to watch for:
- Make sure that UDP traffic is flowing on port 500 between the two devices.
- Watch out for (and eliminate) any use of NAT or PAT along the route between your two devices.
- Beware of intermediate devices that block ports 50 or 51.
If you get this far and come up short, you’ll have to revert to using packet analysis. The good news is that Gerald Combs’ freeware packet analyzer, Ethereal, parses the unencrypted bits of IKE and IPsec, and it’s available for Windows NT/2000 and *NIX. Install this or your favorite more expensive packet analyzer on your external network, between your Firebox and Internet router.
Refer back to your planning sheets and confirm that systems have the same security policies for IKE and IPsec.
What the logs may show when
remote access fails
Differences in configurations between the
MUVPN client and the Firebox may prevent your remote user from establishing
IPsec tunnels and accessing your Trusted Network. Firebox and MUVPN client logs
may help you diagnose the problem.
Should your remote user change or mistype his password, you’ll see the following sequence of log entries:
38228 06/13/01 15:32:50
iked[91]
FROM
207.246.161.212 AG-HDR
ISA_SA ISA_KE ISA_NONCE ISA_ID
38238 06/13/01 15:32:50
iked[91]
TO
207.246.161.212 AG-HDR ISA_SA ISA_KE ISA_NONCE
ISA_ID ISA_HASH
38248 06/13/01 15:32:50
iked[91]
CRYPTO ACTIVE after delay
38258 06/13/01 15:32:50
iked[91]
FROM
207.246.161.212 IF-HDR
-2B5EB394 ISA_NOTIFY
38268 06/13/01 15:32:50 iked[91]
Received AUTH_FAILED message, mess_id=0x00000000
In this case, log entry 38268 isolates the problem for you.
In the process of managing remote users, you may run into a situation where the IKE encryption selected by the remote access client software differs from the default DES encryption used by the Firebox. This might be more commonly encountered in situations where another IPsec client (for example, the SafeNet client), is installed by a remote user. While the Firebox and SafeNet client are compatible, users who manually configure this client may choose 3DES for IKE encryption. You might see this log sequence:
40998 06/13/01 15:40:16
iked[91]
FROM 207.246.161.212 AG-HDR
ISA_SA ISA_KE ISA_NONCE ISA_ID
41008 06/13/01 15:40:16 iked[91]
Proposal is unacceptable. mess_id=0
41018 06/13/01 15:40:16 iked[91]
Sending NO_PROPOSAL_CHOSEN message
41028 06/13/01 15:40:16
iked[91]
Error processing (sa)
41038 06/13/01 15:40:16
iked[91]
Aggressive Mode processing failed
The log entries 41008-41018 show that the IPsec client and Firebox could not agree on the parameters of the IKE Phase 1 SA.
The final example is subtle, and it’s a good one to examine. Do you recall that you assign a static IP address per remote user at the Firebox? Well, what if you change IP assignments and one of your remote users fails to update his configuration file? Let’s look at the log entries for a situation where user “lisa” has changed her Virtual IP address on the Trusted Network from 207.86.152 .186 to 207.86.152.188:
47048 06/13/01 16:00:37
iked[91]
FROM 207.246.161.212 AG-HDR
ISA_SA ISA_KE ISA_NONCE ISA_ID
47058 06/13/01 16:00:38 iked[91]
TO 207.246.161.212 AG-HDR ISA_SA
ISA_KE ISA_NONCE ISA_ID ISA_HASH
47068 06/13/01 16:00:38
iked[91]
CRYPTO ACTIVE after delay
47078 06/13/01 16:00:38
iked[91]
FROM 207.246.161.212 AG-HDR
ISA_HASH
47088 06/13/01 16:00:38
iked[91]
Phase 1 completed as responder
47098 06/13/01 16:00:38
iked[91]
Deleting old phase 1 SA for 207.246.161.212
47108 06/13/01 16:00:38
iked[91]
Deleting SA: peer 207.246.161.212
47118 06/13/01 16:00:38
iked[91]
my_cookie 22F81615A7322B8F
47128 06/13/01 16:00:38
iked[91]
his_cookie F9C35BC2D5ACCCDF
47138 06/13/01 16:00:38
iked[91]
Drop Host 8 207.86.152.186 ipsec_users lisa succeeded
47158 06/13/01 16:00:38
iked[91]
User "lisa" at 207.86.152.186 logged out
47168 06/13/01 16:00:38
iked[91]
lisa unbound
47178 06/13/01 16:00:38
iked[91]
Updated lisa000 channel for lisa rgw
47198 06/13/01 16:00:38 iked[91]
User "lisa" at
207.86.152.186 logged in
47208 06/13/01 16:00:38
iked[91]
Add Host 8 207.86.152.186 ipsec_users lisa succeeded
47218 06/13/01 16:00:38
iked[91]
lisa bound to 207.246.161.212
Log entry 47198 shows user “lisa” has successfully established an IPsec tunnel to the Firebox (this log sequence also shows that “lisa” has previously connected, so you now have a picture of what the process of clearing old IPsec tunnel “state” looks like). Note that the Firebox has bound “lisa” to the Virtual IP address 207.86.152.188 upon completion of IKE Phase 2.
47618 06/13/01 16:01:47
iked[91]
FROM 207.246.161.212 QM-HDR*-4B7E7383
ISA_HASH ISA_SA ISA_NONCE ISA_ID ISA_ID
47628 06/13/01 16:01:47
iked[91]
Getting IPSEC preferences as Responder propnum=1, mode=(Tunnel), laddr=64.53.71.51,
raddr=207.246.161.212
47638 06/13/01 16:01:47
iked[91]
TO 207.246.161.212
QM-HDR*-4B7E7383 ISA_HASH ISA_SA ISA_NONCE ISA_ID ISA_ID
47648 06/13/01 16:01:47
iked[91]
FROM 207.246.161.212 QM-HDR*-4B7E7383
ISA_HASH
47658 06/13/01 16:01:47
iked[91]
Load outbound ESP SA, Algs=ESP_3DES/AUTH_ALG_HMAC_SHA1 Life=3600sec/0KB SPI=c87f1960
47668 06/13/01 16:01:47
iked[91]
207.86.152.128/26 <->
207.86.152.188/32
47678 06/13/01 16:01:47
iked[91]
Load inbound ESP SA, Algs=ESP_3DES/AUTH_ALG_HMAC_SHA1
Life=3600sec/0KB SPI=01048f22
47688 06/13/01 16:01:47
iked[91]
207.86.152.128/26
<-> 207.86.152.188/32
47698 06/13/01 16:01:47
iked[91]
Committing SAs for channel=1 established with QM message_id=4b7e7383
47708 06/13/01 16:01:47 kernel
ipsec: make bundle
for channel 1, 1 in SA's, 1 out SA's
IKE Phase 2 completes, “lisa” and the Firebox set up IPsec SA’s for data, and life is good, right? Look carefully at log entry 47668. Now look at what happens when lisa tries to ping to the FTP server on the Trusted Network (yes, ping is enabled for ipsec_users):
47728 06/13/01 16:01:47
firewalld[74]
deny
in ipsec0 60 icmp 20 32 207.86.152.188 207.86.152.181 8 0 (Ping)
47758 06/13/01 16:01:48
firewalld[74]
deny
in ipsec0 60 icmp 20 32 207.86.152.188 207.86.152.181 8 0 (Ping)
47788 06/13/01 16:01:48
firewalld[74]
deny
in ipsec0 60 icmp 20 32 207.86.152.188 207.86.152.181 8 0 (Ping)
What’s wrong? Remember, during IKE Phase 1, the remote IPsec client identifies itself using the USER-FQDN “lisa.” The Virtual IP address has local meaning to the Firebox and MUVPN client, and is never passed in IKE. The Virtual IP address tells the client what IP address to use as its source address in encapsulated security payload (ESP) packets. It also tells the Firebox to route return packets back to the MUVPN client. The problem here is that the Firebox has a route for 207.86.152.186 for lisa. Thus, all traffic from lisa arrives with a source IP address of 207.86.152.188, but the Firebox is configured to only permit packets from .186
This is actually easier to appreciate in the MUVPN client log than the Firebox log, where each IKE parameter (called transforms) are parsed, and helps make my point that logging at both ends of IPsec VPNs is essential:
2001/06/13
16:02:56.890: AL_IKE--
Start Phase 1 negotiation with peer 64.53.71.51
2001/06/13
16:02:56.950: AL_IKE--
DestID: IPV4_ADDR--64.53.71.51 PresharedKey: xxxxxxxxxxxxxxxxx
2001/06/13
16:02:56.950: AL_IKE--
Protocol -- PROTO_ISAKMP
2001/06/13
16:02:56.950: AL_IKE--
Transform -- KEY_IKE
2001/06/13
16:02:56.950: AL_IKE--
Encryption -- DES_CBC
2001/06/13
16:02:56.950: AL_IKE--
Hash -- MD5_HASH
2001/06/13
16:02:56.950: AL_IKE--
MyID: USER_FQDN--lisa
2001/06/13
16:02:56.950: AL_IKE--
Authentication -- PRESHARED_KEY
2001/06/13
16:02:56.950: AL_IKE--
LifeType -- SECONDS
2001/06/13
16:02:56.950: AL_IKE--
LifeDuration -- 3600
2001/06/13
16:02:56.950: AL_IKE--
GroupDescription -- MODP_768
2001/06/13
16:02:56.950: AL_IKE--
AggressiveMode Exchange Selected
2001/06/13
16:02:57.220: AL_IKE--
AggressiveMode
-- initiator sent out message1
2001/06/13
16:02:58.100: AL_IKE--
IKE message received from 64.53.71.51
2001/06/13
16:02:58.100: AL_IKE--
AggressiveMode
-- initiator received response message1
2001/06/13
16:02:58.100: AL_IKE--
Rcvd Msg [HDR
SA KE NONCE ID HASH]
2001/06/13
16:02:58.210: AL_IKE--
DestID: IPV4_ADDR--64.53.71.51 ProtocolID:17 Port:500
PresharedKey:xxxxxxxxxxxxxxxx
2001/06/13
16:02:58.210: AL_IKE--
AggressiveMode
-- initiator sent out message2
2001/06/13
16:02:58.210: AL_IKE--
Phase 1 negotiation succeeded with 64.53.71.51
2001/06/13
16:02:58.320: AL_IKE--
Sending Notification CONNECTED (0x4000) to peer
64.53.71.51
2001/06/13
16:02:58.320: AL_IKE--
Phase 1 boom counter = 3600
Here, IKE Phase 1 completes. Note that the Virtual IP address never passed between the Firebox and remote IPsec client.
2001/06/13
16:04:06.760: AL_IKE--
Start Phase 2 negotiation with peer 64.53.71.51
2001/06/13
16:04:06.760: AL_IKE--
RemoteHostID: IPV4_ADDR_SUBNET-- 207.86.152.128/255.255.255.192
2001/06/13
16:04:06.760: AL_IKE--
SourceID: IPV4_ADDR--207.86.152.188
2001/06/13
16:04:06.760: AL_IKE--
Protocol -- PROTO_IPSEC_ESP, Nmbr of transforms -- 1
2001/06/13
16:04:06.760: AL_IKE--
Transform -- ESP_3DES
2001/06/13
16:04:06.760: AL_IKE--
Authentication -- HMAC_SHA
2001/06/13
16:04:06.760: AL_IKE--
LifeType -- SECONDS
2001/06/13
16:04:06.760: AL_IKE--
LifeDuration -- 3600
2001/06/13 16:04:06.760: AL_IKE-- EncapsulationMode -- TUNNEL
2001/06/13
16:04:06.760: AL_IKE--
QuickMode --
initiator sent out message1
2001/06/13
16:04:07.140: AL_IKE--
IKE message received from 64.53.71.51
2001/06/13
16:04:07.140: AL_IKE--
QuickMode --
initiator received response message1
2001/06/13
16:04:07.140: AL_IKE--
Rcvd Msg [HDR*
HASH SA NONCE ID ID]
2001/06/13
16:04:07.190: AL_IKE--
Source ID: IPV4_ADDR--207.86.152.188
2001/06/13
16:04:07.190: AL_IKE--
RemoteHost ID: IPV4_ADDR_SUBNET-- 207.86.152.128/255.255.255.192
2001/06/13
16:04:07.250: AL_IKE--
QuickMode --
initiator sent out message2
2001/06/13
16:04:07.250: AL_IKE--
Phase 2 negotiation succeeded with 64.53.71.51
Phase 2 completes. However, the Firebox will deny incoming packets from .188 rather than accepting them because it’s bound lisa to .186. The remedy here is easy: change the MUVPN configuration so that both the Firebox and the client use the same IP address.
Closing Remarks
MUVPNs can be slightly more difficult to configure than Branch Office VPNs, for
two reasons: diversity of computing platforms on which the client software is
installed, and quite likely, less sophisticated users at the client end, which
often results in configuration errors. Plan correctly and you can minimize the
helpdesk calls and maximize your remote access security. And use all the logging
facilities at your disposal.
Remember, however, that making the traffic from remote users and teleworkers secure is only part of the puzzle. Mobile worker laptop security and teleworker broadband connections need firewall and antivirus protection, OS and file system protection, and more. MUVPN is just one element in your entire defense system. ##
Resources:
Revisit Dave's earlier columns
on Security and xDSL
Connections: they are still relevant and helpful.
If you missed the "prequel" to this article, it's in the LiveSecurity archive, entitled "Deploying Mobile User (IPsec) VPN."
Was this article useful to you? Is there a security topic you want our experts to write about? Let us know at lsseditor@watchguard.com.
Copyright© 2001, WatchGuard Technologies, Inc. All rights reserved. WatchGuard, LiveSecurity, Firebox and ServerLock are trademarks or registered trademarks of WatchGuard Technologies, Inc. in the United States and other countries.