I don't know about you guys, but I'm super happy that spring has finally arrived. This winter was just cold, snowy and not a lot of fun. Luckily it looks like nothing but sunshine and butterflies from here on out. It'll be time to start busting out the bathing suits before you know it. With the spring comes the rain, it's a necessary evil that brings with it trees and shade which we'll need in the summer. Much to my surprise though, it's been raining indoors. How is that possible, you ask? It's a very easy explanation.
For as long as I can remember, one of the two air conditioning units in my server closet (I can't even call it a datacenter) has been malfunctioning. The room is not very big, but unfortunately the architects didn't take into consideration what was going into the room and didn't plan it out too well. There isn't a lot of space and there are at least a dozen servers and a dozen network switches in there pumping out heat. The AC units we have are barely enough to get by as it is. If one of them goes down, we're in trouble. Since my boss has only been in his new position for a year, he's still in the "SAVE MONEY, LOOK GOOD FOR THE SUITS, KISS AS MUCH ASS AS POSSIBLE" phase of his employment. It's completely understandable. He's thinking about his career and anyone else would do the same thing. The issue I have is where he's trying to save said money.
I have been bringing up the issue of this faulty AC unit almost since the start of my employment. I've worked at companies that had similar AC issues and it never ends well. In fact this particular issue was the reason that I worked 14 straight weekends last summer. Being that I am the only support guy here, 14 straight weekends can really take its toll on you. Both physically and mentally. Especially when it happens during the summer. Instead of being in the car heading to the beach, I'm on a hot subway heading to work.
Fast forward to this week...
The AC has failed for 72 straight days. When I say fail, I mean it starts blowing hot air instead of cold. Or in this case it starts dripping water, which is blown throughout the room. Normally, turning it on and off will resolve the issue. The problem occurs when it happens overnight. The room can get pretty hot. I lost count of how many times I've come in and the server closet has been 90 degrees plus (which is a great operating environment for servers). Now that my new boss is here and he's looking to make an impact I figure he'll want to nip this issue in the bud. <Charlie Murphy voice> Wrong, WRONG! I even went so far as to get a quote from our AC service company on how much it would cost to replace the unit. Their answer, $10k. $10k? That's it? Really? With labor included? WTF?!??! LET'S DO THIS!! Alas, the head honcho said that price is a little too steep. I went on to explain to him that we easily have $750k-1M worth of equipment in that room. Spending $10k to prevent failures is definitely worth it. I also went on to explain that a server outage will cost us money on missed orders, user downtime, etc. Guess what? It all comes down to the money.
So what's your opinion? Do you think it's worth the incurred cost to insure that your network can function properly? I sure as hell do. Unfortunately for me, I'm not the guy who makes the call on this. I'm just the one who has to come in on the weekend to get everything back up and running.
