Possible network issue affecting STL-A
May 28th, 2009 @ 12:15 PM
We have noted a network issue that is affecting some Slices in STL-A.
We are investigating.
Affecting Level 3 routing only. Engineers are working on it.
Note this may affect the Slicemanager and the main website as well.
UPDATE: 17.52hrs UTC - Just to clarify, the Slicemanager is https://manage.slicehost.com/ so you may see issues attempting to log into your account.
UPDATE: 18.12hrs UTC - issue has been identified. Working with the DC to finalise fix.
UPDATE: 18.22UTC - all issues are fixed.
Please come to http://chat.slicehost.com if you experience any issues.
More information - it was actually an XO issue which created the problems we saw in STL-A. However, the routes appeared to be stalling on Level 3 for various reasons (hence the original note that it was a Level 3 issue).
Posted by paul
Comments:
Jim Elliott commented on Thu May 28 12:34:00 UTC 2009:
Hmm. My slice seems to be unreachable, and so does http://manage.slicehost.com, so this may be more serious than you think.
Dmitry commented on Thu May 28 12:45:18 UTC 2009:
My slice recently became unreachable. Just couple of minutes ago. And the same thing with manage.slicehost.com. Hope that will not last too long.
Ismail commented on Thu May 28 12:55:10 UTC 2009:
Our network department is on the issue. This problem is only effecting Level 3 routing.
ash commented on Thu May 28 14:21:22 UTC 2009:
Slicehost is its own AS, right? Why didn’t you guys pull the bad route? I’m not a BGP guru, so I’m not complaining, just wondering why. Thanks.
jason commented on Thu May 28 19:33:58 UTC 2009:
ash, the issue was caused by a fiber cut on XO’s network and there were problems with upstream routers (off our network) continuing to use stale route data even though BGP updates were made. Until those routers were flushed, people trying to reach stl-a prefixes from networks that still saw the bogus XO route were having trouble.