STL-A Network Issues 0945 GMT

March 27th, 2009 @ 04:56 AM

Started about 10 minutes ago, we’re looking into the issue and on the phone with STL-A NOC.

Note: this will affect manage.slicehost.com, I corrected the post from STL-B to STL-A

update: 1100 GMT – The issue is still ongoing. I am sorry I don’t have more information but we have our engineers and datacenter network techs on site and we are doing everything we can.

update: 1122 GMT – the issue appears to be resolved. I will update when I have more information as to cause, etc.

Post Event Notes: This outage affected STL-A, not STL-B or DFW. But ns1 and ns2 are located in STL-A and some people saw DNS timeouts until hitting ns3. We’re working on a plan to add more nameservers to help in situations like this. We replaced the router in STL-A and worked on several internal switches to resolve the issue.

Posted by matt_disabled

Comments:

Dash commented on Fri Mar 27 04:59:59 UTC 2009:

Please get this resolved ASAP. All our customers are hitting us badly at the moment.

Mario commented on Fri Mar 27 05:01:40 UTC 2009:

All my servers are down, customers are waiting to download their files, please resolve the problem quickly

Olivier commented on Fri Mar 27 05:02:48 UTC 2009:

Mosso CloudServers are down two. Probably a related issue…

Ramon commented on Fri Mar 27 05:03:52 UTC 2009:

Please try to give us an estimate about how long the downtime will be as soon as you can so we can inform our customers.

Thanks in advance and good luck.

Anonymous commented on Fri Mar 27 05:04:27 UTC 2009:

Enjoying this!

chrisfarms commented on Fri Mar 27 05:06:15 UTC 2009:

eek!

atc commented on Fri Mar 27 05:11:23 UTC 2009:

This affects me, but I love the fact that I had so many different ways of working out what’s wrong (i.e. me or the pipes) before having to contact support.

Good luck with the fix!

neen commented on Fri Mar 27 05:12:21 UTC 2009:

Hi guys, though this issue doesn’t affect me I really appreciate your ability to keep us all updated on issues like this. Thanks a bunch SH, and hope you guys can resolve this quickly for those who are affected by the issue..

kochab commented on Fri Mar 27 05:14:14 UTC 2009:

hi, does this impact also SH dns servers?

Sheldon commented on Fri Mar 27 05:15:24 UTC 2009:

yes kochab, ns1.slicehost.com is down too. ns2.slicehost.com isn’t.

Tom Allender commented on Fri Mar 27 05:20:16 UTC 2009:

Looks like ns1.slicehost.net and ns2.slicehost.net are unreachable from here. ns3.slicehost.net is still available.

raoul commented on Fri Mar 27 05:25:20 UTC 2009:

bugger. all my slices are down. pls resolve asafp.

DElyMyth commented on Fri Mar 27 05:26:31 UTC 2009:

Thanks for the quick update, now we can ony hope the issue will be solved quickly.

I came here through twitter, thanks for setting up all the channels :D

alan commented on Fri Mar 27 05:27:33 UTC 2009:

I was under the impression in particular the nameservers were hosted with 3rd parties???

please resolve this asap, i’ve got a couple of hundred clients running sites on our slices! and it’s kind of mid-day in sunny South Africa.

Duarte commented on Fri Mar 27 05:33:05 UTC 2009:

ns2 is also down.

alutz commented on Fri Mar 27 05:33:40 UTC 2009:

Please resolve the problem, we operate German sites using Slicehost and it is 11.30 here, peak time for orders/bookings. Thank you! al

chrisfarms commented on Fri Mar 27 05:49:31 UTC 2009:

if manager was working would I be able to clone a backed up slice into STL-A?

Paco B. commented on Fri Mar 27 05:54:25 UTC 2009:

Not sure, chrisfarms. My six slices are affected, and they are in STL-A, I think.

dkam commented on Fri Mar 27 05:59:00 UTC 2009:

Hi guys – I agree with chrisfarms – if slicemanager was available I could clone a backup or rebuild my site (mostly an automated process). I vote for a highly available slicemanager.

Manuel commented on Fri Mar 27 06:11:47 UTC 2009:

Also STL-A is not reachable!

John commented on Fri Mar 27 06:14:04 UTC 2009:

Despite the message, I think this covers both STL-A and STL-B, based upon what I’m seeing. Not sure if it also affects DFW1.

Manuel commented on Fri Mar 27 06:15:41 UTC 2009:

STL-A is reachable, DNS are not reachable

:S

Mattijs Naus commented on Fri Mar 27 06:17:13 UTC 2009:

Slices back up and running

Jason commented on Fri Mar 27 06:19:26 UTC 2009:

I guess I am lucky. I am on STL-B and all my sites are up.

Dave commented on Fri Mar 27 06:23:11 UTC 2009:

My slice is back – phew! Thanks for fixing it, Slicehost.

dkam commented on Fri Mar 27 06:27:42 UTC 2009:

FS Corruption. :-( + the manager seems to be disconnected from my host – Says it’s not running when it is, console doesn’t attach.

PickledOnion commented on Fri Mar 27 06:34:18 UTC 2009:

dkam,

Please submit a support ticket from the Slicemanager (under the Help tab) – we will look for you.

PickledOnion

Manuel commented on Fri Mar 27 06:53:01 UTC 2009:

Ufff, just in time!

Thanks!

chrisfarms commented on Fri Mar 27 07:08:11 UTC 2009:

+1 for more DNS backup +1 for Highly Available DNS control panel

Thanks for getting it sorted

dkam commented on Fri Mar 27 08:16:22 UTC 2009:

@PickledOnion – the Xen host I was on was rebooted – fsck fixed most of the rest and we’re good to go. Thanks guys.

dave commented on Fri Mar 27 08:34:43 UTC 2009:

+1 for DNSMadeEasy

Ben Allen commented on Fri Mar 27 11:41:20 UTC 2009:

Perhaps now we can invest in some redundant networking hardware (assuming this would fix the problem) so this type of event becomes impossible or at least near impossible! No offense but Slicehost is a hosting company. Downtime is unacceptable, and in reality not too incredibly hard to prevent. Especially now that you have the pockets of Rackspace behind you. Fix it, ensure it never happens again… etc and so forth.

slicematt commented on Fri Mar 27 15:59:11 UTC 2009:

Ben – redundant hardware was not the cause of the problem and is in place.