Fire protection in Server Room

[streaker69]

I have a meeting tomorrow with a fire suppression guy and system design guy. We're going to be looking at doing what the OP had talked about. Capping off a wet sprinkler head and installing a self contained clean system. We're also going to be looking at putting in clean systems in several PLC cabinets, some MCC's and some VFD's cabinets. Probably about a dozen systems in all.

I'll post back with more information after the meeting.

[daffyduc]

we do not own our building so we are stuck with basically what we have....

sprinkler.... we looked into a self contained system but the cost was quickly shot down.

what we did do was have the alarm company come in and install a flow sensor that will immediately cut the power to the servers and from the ups system if the sprinklers are about to go off.... we figured that was the best option in our case.... at least with no power we would not short the servers out and we would not get electrocuted if we were in the room when the puddles form....

not the best option but it was the cheapest....

to be honest though our server room is a concrete walled room with cement floors and a metal roof above... (warehouse server room) so there is not much to burn unless it is an electrical fire....

[streaker69]

we do not own our building so we are stuck with basically what we have....

sprinkler.... we looked into a self contained system but the cost was quickly shot down.

what we did do was have the alarm company come in and install a flow sensor that will immediately cut the power to the servers and from the ups system if the sprinklers are about to go off.... we figured that was the best option in our case.... at least with no power we would not short the servers out and we would not get electrocuted if we were in the room when the puddles form....

not the best option but it was the cheapest....

to be honest though our server room is a concrete walled room with cement floors and a metal roof above... (warehouse server room) so there is not much to burn unless it is an electrical fire....

Cost was shotdown? Sounds like a great management method<sarcasm>. If you were to add up the cost of all the equipment even in a small place, plus all the time associated with rebuilding everything from scratch, plus all the lost time for employees because the systems cannot be accessed I bet it would greatly exceed the cost of a clean system.

What if the fire isn't actually caused by something that cutting the power would help? Your management might want to take another look at it.

I got the quote back today to install a system where our servers are located, which is about 200 cubic feet, which holds the phone system, a standard full size rack, 7 servers, 3 UPii, AC unit and 3 large breaker panels. A clean system for that, with a relay out to tie into the environmental board of one of my UPS' to shut down in case of fire is about $5900. Compared to the cost of the equipment, it is a worthwhile investment.

[Thorn]

Cost was shotdown? Sounds like a great management method<sarcasm>. If you were to add up the cost of all the equipment even in a small place, plus all the time associated with rebuilding everything from scratch, plus all the lost time for employees because the systems cannot be accessed I bet it would greatly exceed the cost of a clean system.

Right. A big part of risk management is to FULLY explain the costs involved. CEO's and the like are cost and ROI driven, which is part of their job. If you don't explain the cost of things like fire suppression systems verses the cost of equipment replacement and data rebuild and employee downtime, then you can't justify anything to them in terms that they understand, which means you haven't done your job.

[daffyduc]

barking up the wrong tree guys

Let me start by saying this is one of the top 10 largest companies in the world... and the cost of downtime perhour is in the $300,000 range.... never mind the rebuild and any other crap we have to deal with.... they spend the budget on BS and then when we bring up the real problems they shoot it down... the line of thinking is why are we going to change something that has been working for years.... yeah welcome to corporate....

the quote was sub $40k to protect 2 racks with aeroK.... that was sub 40K installed and with 3 years maint I believe.... yeah thats a bargain....

to give you an idea of how screwed up this company is we have ups' that will only last about 30 minutes... the warehouse is in the middle of no where... the closest IT guy lives 45 minutes away.... we proposed a natural gas generator (NOTE this company makes GD generators) and they still shot it down......

we then proposed expanding our UPS systems, no dice.....
this is a 24x7 facility that only has IT 9-5 other than oncall....

luckily I just accepted a position in information security for a different company.... so that place is not my problem any more the server/desk side support/network tech/ jack of all trades crap was getting old...... but it was excellent experience....

I will be happy to see them twist in the wind once I am gone.... the only other IT is 2 programmers (one of which is also on his way to another job very soon(apparently negotiating salary right now)) and a manager that was probably useful when decnet was the new it fad.... yeah its going to be fun to watch....

[citrix3k]

Let me start by saying this is one of the top 10 largest companies in the world...

Daffy, can I get a number for them, I will sell them a FS system, pen test and ongoing support

When it comes to presenting a package of measures that are traditionally considered a cost centre such as fire suppressant systems I always take a four pronged approach.

Fear – If you do not do this then lots of bad things will happen, and if you do it then you will be safe.

Competitive edge – Look at our competitors, they don’t do this and if we can show our clients that doing business with us is safer we can attract more clients and strengthen our position in the market. If all our competitors already do this, I would point out how we were losing ground. So essentially creating a profit centre out of a cost centre. If you can sell that concept then you normally get the backing of the Finance Director and the Sales/Marketing director as they love to have any edge they can get. I would always present figures to demonstrate my argument.

Association – If I can I will tie the proposal in with any other project that is likely to get the go ahead. Normally this works well with pen testing new applications, servers et al by making the test a critical part of the project management process. Doing that for fire suppressant system would take a bit of thought though.

ROI – There always has to be a return on investment, even if it is never fully realised like buildings insurance. So my two ROI’s would be continuity in DR and a competitive advantage.

If all that fails , maybe start a small fire in the corner of the server room and ask them what would have happened if you had not been there to put it out

[Thorn]

barking up the wrong tree guys

Let me start by saying this is one of the top 10 largest companies in the world... and the cost of downtime perhour is in the $300,000 range.... never mind the rebuild and any other crap we have to deal with.... they spend the budget on BS and then when we bring up the real problems they shoot it down... the line of thinking is why are we going to change something that has been working for years.... yeah welcome to corporate....

the quote was sub $40k to protect 2 racks with aeroK.... that was sub 40K installed and with 3 years maint I believe.... yeah thats a bargain....

to give you an idea of how screwed up this company is we have ups' that will only last about 30 minutes... the warehouse is in the middle of no where... the closest IT guy lives 45 minutes away.... we proposed a natural gas generator (NOTE this company makes GD generators) and they still shot it down......

we then proposed expanding our UPS systems, no dice.....
this is a 24x7 facility that only has IT 9-5 other than oncall....

luckily I just accepted a position in information security for a different company.... so that place is not my problem any more the server/desk side support/network tech/ jack of all trades crap was getting old...... but it was excellent experience....

I will be happy to see them twist in the wind once I am gone.... the only other IT is 2 programmers (one of which is also on his way to another job very soon(apparently negotiating salary right now)) and a manager that was probably useful when decnet was the new it fad.... yeah its going to be fun to watch....

Sometimes you do exactly what you're supposed to do and the morons in charge still don't get it.

[Barry]

Sometimes you do exactly what you're supposed to do and the morons in charge still don't get it.

If they show their bosses how well they're saving the company money they get a bonus....

[daffyduc]

no if you save the company money your boss gets a bonus then 3 months later tells you that due to the economy you are not getting a raise... again! yeah there is a reason I left

Daffy, can I get a number for them, I will sell them a FS system, pen test and ongoing support

As much as I would like to get you some "business" I am currently inclined to just watch it burn right now

a quick update... I spoke to the person who replaced me at my previous position. they had an incident at another warehouse where a valve malfunction set off the sprinkler in the server room.... it took 10-15 minutes for the sprinkler system to bleed out after the water was shutoff.... all the systems were running via UPS at the time and it sounds like there was nothing salvageable.... The HDD's were encrypted and apparently the decryption info was not well documented.... upon re-installation of new hardware the drives were useless.... they had to send the drives off to a datarecovery company for recovery.... 17 drives (not sure what size).... $180,000 for recovery!!! (or so I am told... this is all second hand info)

never mind the hardware cost and downtime.... just the data recovery cost was more than 4 times what the fire suppression system would have cost (the server room at that location is similar to the one here in knoxville)

yeah the guy called me to find out where that quote was

I think corporate woke up a little....

remember this was not a fire but a system malfunction.... apparently someone broke the glass sensor in the room... at least that is the only way I could see this happening....

food for thought....