Cogent maintenance in STL-B at 8AM GMT on July 1st
June 30th, 2009 @ 10:17 PM
One of our connectivity providers, Cogent, will be performing a maintenance at 8AM GMT on July 1st that will affect the STL-B datacenter. In preparation for the maintenance, we have brought down the Cogent link to ensure good connectivity through the maintenance period. All traffic will be able to route through our other providers without issues.
If you have any issues overnight, please contact our support team and we will be happy to help.
Posted by major 0 comments »DFW1: Networking issues
June 29th, 2009 @ 03:24 PM
We’re investigating some problems with the network in our DFW1 datacenter. As soon as we have additional information, we will update this post. We apologize for the inconvenience.
Update as of 20:32 GMT: It appears that almost all of the networking issues have been resolved. We are still awaiting further details and confirmation at this time.
Posted by majorNetworking issues in STL-B
June 28th, 2009 @ 03:17 PM
Earlier today, around 7:45 PM GMT, a large DoS attack affected the network at our STL-B datacenter. It caused increased latency for customers in that datacenter, and it may have caused some customers to briefly lose access to their slices.
The issue was resolved by 7:55 PM GMT and the network is now back to normal.
Posted by majorEmergency Switch Replacement in DFW1
June 25th, 2009 @ 02:45 PM
We’ve found that one of our primary switches in the DFW1 datacenter needs to be replaced as soon as possible. The emergency maintenance is planned for this evening at 2AM GMT (9PM CDT) and it should be very brief.
Since our environment has redundant switching, the downtime should only be a matter of seconds. In the worst case scenario, the downtime should be less than 60 seconds. As soon as the switch is successfully replaced, we will provide an update regarding the network’s status.
We apologize for the short notice for this maintenance and for any inconvenience that it may cause. Thank you for your support!
UPDATE AT 2:45AM GMT: The maintenance is now complete and the switching has been thoroughly tested after the replacement. If you’re experiencing any on-going issues, please contact us at any time.
Posted by majorMaintenance on STL-A
June 2nd, 2009 @ 01:07 AM
We've entered the planned maintenance window for STL-A. You may experience brief connectivity issues as routes are updated. Should you experience any problems, please come by the chatroom or contact support. We will update this page with any updates and once this maintenance has been completed. Update: 1:15AM CST The maintenance window is now finished. We apologize for any issues you suffered during this maintenance. We'll continue to monitoring the network. If you experience any issues, please come by the chatroom or contact support. Posted by sabrinaPossible network issue affecting STL-A
May 28th, 2009 @ 12:15 PM
We have noted a network issue that is affecting some Slices in STL-A.
We are investigating.
Affecting Level 3 routing only. Engineers are working on it.
Note this may affect the Slicemanager and the main website as well.
UPDATE: 17.52hrs UTC - Just to clarify, the Slicemanager is https://manage.slicehost.com/ so you may see issues attempting to log into your account.
UPDATE: 18.12hrs UTC - issue has been identified. Working with the DC to finalise fix.
UPDATE: 18.22UTC - all issues are fixed.
Please come to http://chat.slicehost.com if you experience any issues.
More information - it was actually an XO issue which created the problems we saw in STL-A. However, the routes appeared to be stalling on Level 3 for various reasons (hence the original note that it was a Level 3 issue).
Posted by paulResize and Rebuild issues - STL-A & STL-B
May 27th, 2009 @ 10:38 AM
We are experiencing a technical issue (a booboo!) with resizes and rebuilds in the STL-A and STL-B DCs.
The techs are there right now fixing it.
Apologies for any hassle this is causing.
Will update soonest.
UPDATE: Still working on it. All relevant people are investigating and fixing.
UPDATE: Main issues fixed. Backups and resizes will be queued for the moment. Will update when fully restored.
UPDATE: All fixed. There may be a small delay while we work through the queued backups and resizes.
Please do come into our chatroom (http://chat.slicehost.com) if you experience any issues.
Paul.
Posted by paulSTL-B Network Maintenance
May 13th, 2009 @ 01:27 PM
Last night we upgraded the software on all of our secondary, internal routers. Tonight, starting around 0300 GMT, we’ll be failing over to the secondary devices and upgrading the primaries. We do not expect any customer impact and the Slicehost/Rackspace NOC will be monitoring the upgrade process. If you experience any networking difficulties, please contact our support team or stop by chat.slicehost.com. Thank you!
Posted by mattSTL-A Network Issues
May 11th, 2009 @ 09:30 AM
We are investigating a DoS attack in STL-A.
UPDATE: The issue has been resolved but we will, of course, keep an eye on it.
Posted by paulSTL-A Network Issue
April 24th, 2009 @ 05:25 AM
Monitoring alerted us to an issue in STL-A, NOC investigating the issue.
UPDATE Appears good for most people now.
Posted by mattEmergency Switch Replacement STL-A
April 17th, 2009 @ 12:27 PM
We have determined one of our switches in the STL-A datacenter needs to be replaced. We plan on executing this maintenance tonight at 11:00PM CST. However, if the situation degrades anymore, we will need to do an emergency maintenance immediately.
The planned maintenance window will begin on Friday, April 17 at 11:00PM CST (04:00 UTC) and end at 12:00AM CST (5:00 UTC). We expect the actual downtime to be less than 30 minutes and many customers may only experience network latency.
We apologize for the short notice and any problems this may cause you. If you have any questions, please let us know. We appreciate your understanding and patience as we deal with this issue. Thank you for your support!
Update 11:40PM CST (4:40 UTC)- Maintenance has been completed successfully. We’ll continue to monitor for any irregularities.
Posted by mattSTL-A Network Issue
April 10th, 2009 @ 06:15 AM
Some people are reporting difficulty connecting to STL-A network. We’re investigating the issue now.
UPDATE – This appears to be affecting people coming in over XO network.
UPDATE – We escalated the issue, the NOC engineering team was onsite. They were working on the core router during a non-impacting maintenance window. We’re investigating to determine what caused the problem.
UPDATE 1320 GMT – Some people still having issues, investigating.
UPDATE 1345 GMT – Still working, appears to be EU customers.
UPDATE 1420 GMT – Still working, confirmed via comments not exclusive to EU.
UPDATE 1445 GMT – Should be resolved, issues seems to have been with Cogent in STL-A affecting some customers.
Posted by mattNetwork issues - STL-A
April 2nd, 2009 @ 03:58 PM
21.50 UTC:
We are experiencing issues with the provider network in the STL-A DC.
We have escalated it and will update as soon as know anything.
Update: Appears to be resolved - was outside of our network: http://internetpulse.com/
Still investigating though.
Update: Issue has be escalated but it is now resolved.
Posted by paulLevel3 and Cogent Issues
March 27th, 2009 @ 03:56 PM
Level3 and Cogent seem to be having some network wide problems that may affect customers. Check www.internetpulse.com for more info. Rackspace NOC has escalated with Level3 and we are working the issue.
Posted by mattSTL-A Network Issues 0945 GMT
March 27th, 2009 @ 04:56 AM
Started about 10 minutes ago, we’re looking into the issue and on the phone with STL-A NOC.
Note: this will affect manage.slicehost.com, I corrected the post from STL-B to STL-A
update: 1100 GMT – The issue is still ongoing. I am sorry I don’t have more information but we have our engineers and datacenter network techs on site and we are doing everything we can.
update: 1122 GMT – the issue appears to be resolved. I will update when I have more information as to cause, etc.
Post Event Notes: This outage affected STL-A, not STL-B or DFW. But ns1 and ns2 are located in STL-A and some people saw DNS timeouts until hitting ns3. We’re working on a plan to add more nameservers to help in situations like this. We replaced the router in STL-A and worked on several internal switches to resolve the issue.
Posted by matt