SOHO Disaster Recovery
(or, The Story of Bob)
by David M. Piscitello, President, Core Competence, Inc.
[Editor's note: We've sent this article to both SOHO and Firebox users, because many Firebox users also manage SOHOs. If you don't have a SOHO, feel free to skip this piece. If you do use a SOHO, this article contains detailed instructions on how to help a remote user recover if it malfunctions. The article covers all versions of SOHO firmware. That made for a long article, but it also made it a handy one-stop resource. We know you're busy -- feel free to jump to the procedures that pertain to your network. --Scott Pinzon]
This is a story about Bob, and the administrator who must control Bob’s network and security behavior, and must control others like Bob.
Every administrator has a Bob -- perhaps several. Bob’s the power user in your organization. He “knows” Windows, knows how to set up neighborhoods and shares and knows what all those unfathomable Control Panel things do. He impresses the secretaries and drives you crazy. He tweaks and tunes his desktop PC for maximum performance. He’s the first to run “new” applications and services from his desktop PC. He routinely corrupts the Windows Registry on his and fellow workers’ PCs. In the process of recovering his desktop operating system, he (routinely) fails to reinstall anti-virus software. His browser zone settings invite disasters. Bob is a support incident waiting to happen.
In remote offices, he often is the self-appointed SOHO network (ahem) administrator.
Bob and the SOHO Firewall “Incident”
Our Bob works in a branch office. In Bob’s mind, his unparalleled prowess with Windows makes him the obvious choice to manage the firewall. Bob begs and cajoles (or directs you -- my experience is that Bobs frequently hold director level positions) until you relent and assign him the task of managing the SOHO. Pandora’s box is opened, and it’s merely a matter of time before you receive the call:
Bob: “The firewall’s broken!”
You: “Define ‘broken.’ Did you drop it or spill coffee on it?”
Bob: “Of course not. The Watchguard SOHO is a sophisticated piece of hardware. I know that, and I don’t let anyone near it."
You: “So what’s broken?”
Bob: “I read this great column in Home and Firewalls magazine about how important it is to have long passwords and to change them regularly, so I thought I’d beef up our security here at our office. I changed the username and password on the SOHO while I was renumbering our site—DHCP wasn’t working for us. We have some sensitive material here and --”
You (interrupting Bob): “And you can’t remember the password?”
Bob: “I remember the last 3 letters were ‘B-O-B.’ I wrote it down, and had it on a Post-It on my machine. The cleaning lady must have thrown it away… Look, I know what you’re thinking, and don’t worry. I wrote 'password for the server' on the Post-It to throw intruders and questionable insiders off. You know, Home and Firewalls should mention this as a countermeasure --”
You (cutting Bob off): “Doesn’t matter. I can talk you through the process of resurrecting your firewall. You’ll need a regular LAN cable, the kind you use to connect PCs to the SOHO’s internal hublet. Do you have one?
Bob: “You bet!”
You: “Good. Now, I see you were running SOHO firmware version 2.4.19. Now we can do a firmware reset using the factory.exe program I sent you, then reinstall the SOHO firmware I sent you and the current configuration file all organizations should be using. Then…”
Bob (cutting you off): “That won’t work. I lost the SOHO software when my PC froze last week and I was forced to use my recovery disks. I think there’s something wrong with my PC, maybe it’s time you order me a 1.4 Ghz … and the default configuration file won’t do at all. We run some very advanced services here and I’ve changed the filters to allow traffic to port, um, 31337 I think, and several other business critical applications…”
You: “Do you have a copy of the configuration?”
Bob: “No, I keep it on the firewall --”
You: “Let me get this straight. Your office firewall is inaccessible because you changed your password and IP addresses. I assume you don’t remember what IP address you assigned to the firewall. You’ve lost the recovery firmware. You have no backup of the current configuration file so when we restore the firewall your office won’t be able to run the unauthorized applications you permitted. Does that about sum it up?”
Bob: “Geez, when you put it like that, you make it sound like this is all my fault!”
Best practices to eliminate Bob Incidents
Small changes in operating practices and policy will help you contain damages Bob can inflict.
First, don’t concede the write pass phrase of any security appliance to anyone you don’t directly oversee. This is a policy battle for your superior(s) if Bob is a higher-up, and you can use this real-world scenario as an example of why access control is so important.
Next, consider burning recovery CDRs containing everything you (or someone you trust) needs to resurrect a SOHO whose software has become corrupted or whose configuration password gets lost. Put the following on this CD:
-
If you're running firmware version 2.4.19 or earlier, put the firmware reset program, factory.exe on the disk. If you're running system 5.0 or later, put the file SOHO.exe on the disk instead, and skip step 2.
-
The latest firmware level your organization uses
-
A copy of the Advanced FAQ from Watchguard entitled How do I reset the SOHO to factory defaults?
-
A copy of the SOHO User Guide
- A README file that complements this documentation with any specific information, such as a step-by-step procedure to configuring the SOHO according to your organization’s security policy. The README should provide enough information and the appropriate firmware and documentation so that a sufficiently capable employee (i.e., someone other than Bob) can restore a SOHO to a working state without connecting to the Internet. Bob incidents often take remote offices offline. Include an escalation procedure, which should identify the process for returning the SOHO to you or someone delegated with the responsibility for dealing with security hardware control.
Bundle this with an extra straight-through cable. If you avoid over-the-phone confusion about whether a cable is straight-through or crossover just once, you've recovered the cost of the cable.
If the firewall seems "broken" to the folks in your remote office, they will be sorely tempted to plug their network directly into a broadband connection to keep business running. You must do what you can to make the recovery process as convenient as possible, so they have few or no excuses for bypassing the firewall. This is one reason I suggest a CDR instead of an intranet Web page. But if you feel confident your help staff can securely dial into an intranet from any remote office in a pinch, then an intranet Web page with downloadable versions of all of the above works fine.
When your helper does restore the SOHO to its upright and original position, your first priority will be to reassign a new write pass phrase. You can do this remotely, securely, and conveniently for all SOHOs you manage when you’re using VPN Manager, but that's a topic for another column.
WatchGuard recently upgraded the SOHO's firmware to version 5. The actual steps for restoring a SOHO differ markedly, depending on whether you're using before-5.0 or after-5.0 software. Here are both procedures -- use the one that fits your situation.
How to resurrect a SOHO (firmware version 2.4.19)
WatchGuard's technical documentation for resetting a SOHO with factory defaults is quite good, and I recommend you read it (see the link in Step 2, below). Summarizing the steps you should explain in your README…
-
Download factory.exe and the appropriate SOHO firmware image from Watchguard.
-
Print a copy of the technical documentation How do I reset the SOHO to factory defaults?
-
Unplug power from the SOHO.
-
Connect a straight-through LAN cable to LAN port number one on the SOHO, then to the Ethernet NIC port on the PC you’ll use to resurrect the firewall.
-
Reconfigure the IP address of your PC to 90.0.0.5, set the subnet mask to 255.255.255.0, and if necessary, restart Windows on your PC.
-
Run the factory.exe program. When prompted, plug the power into the SOHO. The utility will clean up the SOHO, erase the configuration, and reboot the SOHO.
-
When the reboot is completed, power cycle the SOHO. After about 45 seconds, the SOHO ON light will blink. It’s now in the factory default setting.
-
Configure your Windows PC to use DHCP, and reboot Windows. Be certain your SOHO is still on, and that your PC is still connected to the SOHO internal hublet via a straight-through LAN cable.
-
From a Windows command prompt (MS-DOS window), type ipconfig to confirm your PC’s been assigned an IP address from the SOHO default address space (it’s likely to be 192.168.111.2). Alternatively, you can type ping 192.168.111.1, the address that the SOHO will assign to its own private network interface.
-
Locate the SOHO firmware (on the CDR or in the directory where you downloaded the firmware), double-click the file or icon, and follow the instructions of the update program. The SOHO will reboot.
-
Connect again to the SOHO at http://192.168.111.1. Log in using the default account, then configure the firewall according to the policy you wish to enforce.
- Return the SOHO to its original and upright position, between your remote office PCs and broadband modem or router.
How to resurrect a SOHO (firmware version 5.0.28)
The procedure for restoring the default configuration on a SOHO with the new version 5.0.28 firmware is simpler for your Bobs to perform. The new firmware is organized into three separate partitions in the SOHO’s long-term memory (called flash). A partition called <SYSA> contains the latest version of the SOHO firmware you’ve installed. A second partition contains your configuration data. A backup application stored in a third partition <SYSB> eliminates the need for your Bobs to keep a copy of factory.exe handy.
Here’s a step-by-step synopsis of the new (and much shorter) procedure:
-
Unplug power from the SOHO.
-
Connect one end of the straight-through LAN cable to LAN port number one on the SOHO, and the other end to the WAN port on the SOHO.
- Plug the power into the SOHO. If the SOHO system start code senses the WAN-LAN connection is present, it erases the configuration partition, i.e., it restores the configuration defaults. If you attempt to connect to the SOHO from your PC during this process you might see this screen:
Your WatchGuard device is running from a backup copy of firmware.
Your unit can be restored by either using the UpdateWizard, or by sending the required files using FTP.
Contact http://www.watchguard.com/support to download the latest SOHO update wizard. NOTE: This unit may only be updated with version 5.0.0 (Crestline) or newer firmware. You can revert to older versions by uploading the ROMIMAGE file created by the update wizard when this unit was updated to 5.0.0
Status of firmware
SYSA partition is valid.
/RAM/NVFS file system is NOT valid.
SYSB version 3.7
Help available at http://www.watchguard.com/support
-
When the reboot is completed, power cycle the SOHO. After about 45 seconds, the SOHO ON light will blink. It’s now in the factory default setting.
- If the PC from which you administer the SOHO is configured to use DHCP, open an MS-DOS (Command Prompt) window and type ipconfig /release. When the SOHO reboot is completed, type ipconfig /renew. Now connect to IP address 192.168.111.1 from your Web browser and begin the process of reconfiguring your SOHO. (Otherwise, configure your Windows PC to use DHCP, reboot Windows, then begin Step 6.) Be certain your SOHO is still on, and your PC is still connected to the SOHO internal hublet via a straight-through LAN cable.
The Version 5.0.28 SOHO firmware has another useful “disaster recovery” feature. If your Bob somehow managed to corrupt the SOHO firmware in <SYSA>, the new firmware will detect this problem when the SOHO boots. The firmware will automatically save off your working configuration and boot the SOHO from the backup partition <SYSB>. Your Bobs can use the update software, SOHO.exe, to upload a fresh version 5.0.28 SOHO firmware to <SYSA> without losing their working configuration.
Parting comments
Managing remote office firewalls and mobile users can be a painful and time-consuming aspect of your administrative load. Learn to deal with it efficiently while the numbers are small, because it will only become more difficult if you don’t have administrative control and sound operational practices in place from the outset.
My final word of encouragement: are you familiar with the phrase illigitemi non carborundum? No? To paraphrase it roughly, it means, “Don’t let the Bobs grind you down.”
For more helpful articles, see our LiveSecurity Archive.
Copyright© 2001, WatchGuard Technologies, Inc. All rights reserved. WatchGuard, LiveSecurity, Firebox and ServerLock are trademarks or registered trademarks of WatchGuard Technologies, Inc. in the United States and other countries.