Thursday, September 9, 2010

From The Archives

This one happened a couple of years back.....

I had just started a new job as a LAN Administrator at a pretty well known shoe company. I started at a pretty busy time since there were a number of migrations in progress and a few more coming down the road. I come into work 15 minutes early and notice our Exchange cluster is on the secondary server. My comrades are all discussing how the consultants fucked something up but didn't bother to do any troubleshooting. Good thing I didn't bang out sick. While I'm going through the Event logs (you think anyone else would have tried that?). I come to find out the information store crapped out around 3am. That could be for a number of reasons though as we have another group of consultants migrating all of our PST's to a product called Unlimited Mail. Unfortunately to do this you need to migrate them to the users' mailboxes first. So our exchange database is half a Terabyte right now. So I keep going through the event logs and find out that BOTH servers rebooted at 5am. At EXACTLY 5am. Guess why? Can you guess? Did you figure it out? These numb nuts that I work with turned Microsoft's Automatic Updates on.....on ALL of our servers. Not only did the turn it on, they set it to install all the updates and reboot automatically. That's not the best part though. You would think that if you have a cluster setup you may want to stagger the updates if you decide that you have to do them automatically, so that, I don't know.......YOUR ENTIRE EXCHANGE CLUSTER ISN'T REBOOTING AT THE SAME FREAKIN' TIME!!!!!!!!!!! And for THIS reason.........."IT is HELL!!" Please shoot me. Maybe I should become a Cracker Jack vendor at Yankee Stadium or something.

Side Note: I made sure to set up a WSUS server and create a GPO forcing Automatic Updates to Download and Notify. If only the very well paid consultants would have paid attention to detail. All of this could have been avoided.