Republished with permission from
WatchGuard Technologies, Inc.
SOHO Disaster Recovery
(or, The Story of Bob)
[Editor's note: We've sent this article to both SOHO and Firebox users, because many Firebox users also manage SOHOs. If you don't have a SOHO, feel free to skip this piece. If you do use a SOHO, this article contains detailed instructions on how to help a remote user recover if it malfunctions. The article covers all versions of SOHO firmware. That made for a long article, but it also made it a handy one-stop resource. We know you're busy -- feel free to jump to the procedures that pertain to your network. --Scott Pinzon]
This is a story about Bob, and the administrator who must control Bob’s network and security behavior, and must control others like Bob.
Every administrator has a Bob -- perhaps several. Bob’s the power user in your organization. He “knows” Windows, knows how to set up neighborhoods and shares and knows what all those unfathomable Control Panel things do. He impresses the secretaries and drives you crazy. He tweaks and tunes his desktop PC for maximum performance. He’s the first to run “new” applications and services from his desktop PC. He routinely corrupts the Windows Registry on his and fellow workers’ PCs. In the process of recovering his desktop operating system, he (routinely) fails to reinstall anti-virus software. His browser zone settings invite disasters. Bob is a support incident waiting to happen.
In remote offices, he often is the self-appointed SOHO network (ahem) administrator.
Bob and the SOHO Firewall “Incident”
Our Bob works in a branch office. In Bob’s mind, his unparalleled prowess with Windows makes him the obvious choice to manage the firewall. Bob begs and cajoles (or directs you -- my experience is that Bobs frequently hold director level positions) until you relent and assign him the task of managing the SOHO. Pandora’s box is opened, and it’s merely a matter of time before you receive the call:
Bob: “The firewall’s broken!”
You: “Define ‘broken.’ Did you drop it or spill coffee on it?”
Bob: “Of course not. The Watchguard SOHO is a sophisticated piece of hardware. I know that, and I don’t let anyone near it."
You: “So what’s broken?”
Bob: “I read this great column in Home and Firewalls magazine about how important it is to have long passwords and to change them regularly, so I thought I’d beef up our security here at our office. I changed the username and password on the SOHO while I was renumbering our site—DHCP wasn’t working for us. We have some sensitive material here and --”
You (interrupting Bob): “And you can’t remember the password?”
Bob: “I remember the last 3 letters were ‘B-O-B.’ I wrote it down, and had it on a Post-It on my machine. The cleaning lady must have thrown it away… Look, I know what you’re thinking, and don’t worry. I wrote 'password for the server' on the Post-It to throw intruders and questionable insiders off. You know, Home and Firewalls should mention this as a countermeasure --”
You (cutting Bob off): “Doesn’t matter. I can talk you through the process of resurrecting your firewall. You’ll need a regular LAN cable, the kind you use to connect PCs to the SOHO’s internal hublet. Do you have one?
Bob: “You bet!”
You: “Good. Now, I see you were running SOHO firmware version 2.4.19. Now we can do a firmware reset using the factory.exe program I sent you, then reinstall the SOHO firmware I sent you and the current configuration file all organizations should be using. Then…”
Bob (cutting you off): “That won’t work. I lost the SOHO software when my PC froze last week and I was forced to use my recovery disks. I think there’s something wrong with my PC, maybe it’s time you order me a 1.4 Ghz … and the default configuration file won’t do at all. We run some very advanced services here and I’ve changed the filters to allow traffic to port, um, 31337 I think, and several other business critical applications…”
You: “Do you have a copy of the configuration?”
Bob: “No, I keep it on the firewall --”
You: “Let me get this straight. Your office firewall is inaccessible because you changed your password and IP addresses. I assume you don’t remember what IP address you assigned to the firewall. You’ve lost the recovery firmware. You have no backup of the current configuration file so when we restore the firewall your office won’t be able to run the unauthorized applications you permitted. Does that about sum it up?”
Bob: “Geez, when you put it like that, you make it sound like this is all my fault!”
Best practices to eliminate Bob Incidents
Small changes in operating practices and policy will help you contain damages Bob can inflict.
First, don’t concede the write pass phrase of any security appliance to anyone you don’t directly oversee. This is a policy battle for your superior(s) if Bob is a higher-up, and you can use this real-world scenario as an example of why access control is so important.
Next, consider burning recovery CDRs containing everything you (or someone you trust) needs to resurrect a SOHO whose software has become corrupted or whose configuration password gets lost. Put the following on this CD:
Bundle this with an extra straight-through cable. If you avoid over-the-phone confusion about whether a cable is straight-through or crossover just once, you've recovered the cost of the cable.
If the firewall seems "broken" to the folks in your remote office, they will be sorely tempted to plug their network directly into a broadband connection to keep business running. You must do what you can to make the recovery process as convenient as possible, so they have few or no excuses for bypassing the firewall. This is one reason I suggest a CDR instead of an intranet Web page. But if you feel confident your help staff can securely dial into an intranet from any remote office in a pinch, then an intranet Web page with downloadable versions of all of the above works fine.
When your helper does restore the SOHO to its upright and original position, your first priority will be to reassign a new write pass phrase. You can do this remotely, securely, and conveniently for all SOHOs you manage when you’re using VPN Manager, but that's a topic for another column.
WatchGuard recently upgraded the SOHO's firmware to version 5. The actual steps for restoring a SOHO differ markedly, depending on whether you're using before-5.0 or after-5.0 software. Here are both procedures -- use the one that fits your situation.
How to resurrect a SOHO (firmware version 2.4.19)
WatchGuard's technical documentation for resetting a SOHO with factory defaults is quite good, and I recommend you read it (see the link in Step 2, below). Summarizing the steps you should explain in your README…
How to resurrect a SOHO (firmware version 5.0.28)
The procedure for restoring the default configuration on a SOHO with the new version 5.0.28 firmware is simpler for your Bobs to perform. The new firmware is organized into three separate partitions in the SOHO’s long-term memory (called flash). A partition called <SYSA> contains the latest version of the SOHO firmware you’ve installed. A second partition contains your configuration data. A backup application stored in a third partition <SYSB> eliminates the need for your Bobs to keep a copy of factory.exe handy.
Here’s a step-by-step synopsis of the new (and much shorter) procedure:
Your WatchGuard device is running from a backup copy of firmware.
Your unit can be restored by either using the UpdateWizard, or by sending the required files using FTP.
Contact http://www.watchguard.com/support to download the latest SOHO update wizard. NOTE: This unit may only be updated with version 5.0.0 (Crestline) or newer firmware. You can revert to older versions by uploading the ROMIMAGE file created by the update wizard when this unit was updated to 5.0.0
Status of firmware
SYSA partition is valid.
/RAM/NVFS file system is NOT valid.
SYSB version 3.7
Help available at http://www.watchguard.com/support
The Version 5.0.28 SOHO firmware has another useful “disaster recovery” feature. If your Bob somehow managed to corrupt the SOHO firmware in <SYSA>, the new firmware will detect this problem when the SOHO boots. The firmware will automatically save off your working configuration and boot the SOHO from the backup partition <SYSB>. Your Bobs can use the update software, SOHO.exe, to upload a fresh version 5.0.28 SOHO firmware to <SYSA> without losing their working configuration.
Managing remote office firewalls and mobile users can be a painful and time-consuming aspect of your administrative load. Learn to deal with it efficiently while the numbers are small, because it will only become more difficult if you don’t have administrative control and sound operational practices in place from the outset.
My final word of encouragement: are you familiar with the phrase illigitemi non carborundum? No? To paraphrase it roughly, it means, “Don’t let the Bobs grind you down.”
For more helpful articles, see our LiveSecurity Archive.
Copyright© 2001, WatchGuard Technologies, Inc. All rights reserved. WatchGuard, LiveSecurity, Firebox and ServerLock are trademarks or registered trademarks of WatchGuard Technologies, Inc. in the United States and other countries.
Copyright © 1996 - 2001
WatchGuard Technologies, Inc. All rights reserved.