On Monday afternoon, the building one of my clients is in lost power. Normally, this wouldn’t be a big deal. Their UPS would tell their servers to shut down, and all would be good. Unfortunately, that’s not what happened. Their UPS failed, and their Windows 2003 Small Business Server shut down in a rather dirty fashion.
Once power was restored, I managed to get into the machine and see what kind of damage was done. Normally, I wouldn’t have worried at all – but this client seems to have the worst luck when it comes to systems.
My worries were well proven. Exchange couldn’t mount the private message store. Some quick digging with ESEUtil showed that the store had shut down dirty (which it should have recovered from), but was missing one of the transaction log files.
Bad news. With that log file missing, it’s pretty unlikely to get the store mounted without some data loss.
Normally in this situation, you have two choices:
- Restore from backup
- Repair the store and lose data
Neither option is terribly good. Restoring on a Monday would mean losing all the e-mail from the weekend and most of a business day. Repairing the store with that transaction log missing would mean losing an unknown amount of data. Oh, and according to Microsoft, the repair process takes and hour for every 3 gigs of data (there’s 15 gigs in that store).
Thankfully, there’s an answer number 3.
When I upgraded this client to SBS and Office 2003, I configured the clients to work in Cached Mode. For those of you who don’t know, Outlook Cached mode keeps a copy of the users mailbox locally to reduce the load on the server, and let the users work when they’re offline. It works something like an OST file did in previous versions of Outlook, only much smoother.
Since I knew I wouldn’t lose any data, I started the repair process and let it run. Microsoft’s estimates weren’t anywhere near accurate for timing – 15 gigs were processed in about 2 hours. The offline defragmentation that is required afterwards took about an hour.
After the process was complete, Exchange was able to start up normally. Of course, the only way I knew anything about the amount of data lost was based on the size of the store – which had shrunk about 5 megs.
Instead of leaving the server up and running, I actually left the Exchange services stopped. That way I could have the users start Outlook and export their mailbox before connecting to the server. Any users who lost data could easily reimport their mailbox (not importing duplicates) to recover any lost data.
I wish the cache synchronization process was a little smarter – it simply assumed the server was correct and removed any items that were in the cache that weren’t on the server.
In the end, I had a long night but the client was fully up and running, with no data loss, within a couple hours of Tuesday morning.
Their new UPS is on order.