Slicehost

We just received word of a datacenter maintenance event scheduled this Friday. It will affect most customers, although not those who signed up this week. During the 4-hour window, the service impacting activities are scheduled to occur between 0800-1000GMT.

• routing config changes on core routers
• software updates on core routers
• replacing interface card with a defective port

The expected impact is 3-4 outages of 5 minutes or less. We’ll be monitoring during the window, remember to check status.slicehost.com, slicechat and our twitterstream for updates.

Update

These are the time frames to expect network outages if everything goes as planned:

0800 to 0810 08330 to 0835 0900 to 0910

FYI:

• Ubuntu Gutsy (7.10) is available
• Gentoo is now 2007.0
• CentOS is now 4.5 (5 is coming soon)
• Bug fixes for all images

The datacenter network maintenance that caused the outage yesterday was rolled back and will be performed again tonite (Nov 3 0800 GMT). It was supposed to be non-impacting, as is tonite’s maintenance. The Cisco bug that caused the problem has a work around and it will be employed this evening.

We’ll be monitoring things on our end – should there be problems remember to check the chatrooms, Twitter or status.slicehost.com. This maintenance is in preparation for the core cutover scheduled for next week.

Relaying more information as we receive it: the cause of the outage was due to a Cisco IOS bug dealing with HSRP. Unfortunately, it should have been circumvented and can be attributed to human error during the maintenance procedure. We’re meeting with the datacenter leads next week to discuss the issues of the past week and what is being done to correct them. There’s also discussion of moving the core cutover up, keep an eye on the blog/forum for any news on that front – we’ll send an email if it is happening sooner than expected. Again we are truly sorry for an unacceptable round of outages. As always, please contact us if you have any questions or concerns.

For those of you just tuning in, there was another network outage from 0900-1100GMT. We’re still piecing together what exactly happened but here is what we know so far:

• there was a non-impacting maintenance schedule by our datacenter in preparation for the core upgrade taking place in about a week. They were inserting a switch in parallel with the existing core switches.
• starting at 0900 GMT when the maintenance was completed, the outage started.
• after working with Cisco, the issue was resolved around 1100 GMT

This is an unacceptable level of service and people have expressed concerns in the chatroom about the recent outages. We agree completely and cannot offer more than our sincerest apologies at the moment. We are gathering more information and reviewing our options, updates will follow. In the meantime, if you have any questions or feel like venting, you can contact us via email or call me directly (314.266.3502). If I don’t answer, leave a message and one of us will get back to you shortly.

As I’m sure most people noticed, we had 2 major bouts of network trouble – yesterday afternoon and one today. It appears they were distributed denial of service attacks directed at multiple customers. The amount of traffic was filling both of the datacenter’s incoming pipes. There aren’t any excuses for why this took 2 extended outages to figure out and we’ll be addressing that issue with our provider. All I can offer are our sincere apologies and a promise that this will get better.

During the outages a lot of questions came up regarding multiple carriers, routers, etc. So to answer a couple of those:

• we have redundant transport via XO and Verizon, however this won’t help since traffic will end up filling both.
• we also have redundant routers using VRRP to failover on our side as does the datacenter.

Again, we’re extremely disappointed with how this was handled, but here is some good news. In the next couple of weeks, the datacenter’s core network (not ours, but how we connect to them) will be upgraded. We’re assured this will result in a more stable setup for growth.

Secondly and this was the announcement we referred to earlier this week, we are putting the final touches on a second facility. This space uses separate carriers and we will be managing the network, but everything else stays the same. We hope this expansion allows us to accelerate our growth and offer a higher level of redundancy for customers who need it.

Thanks to everyone for their patience during the past 2 days. For future reference, aside from the chatrooms, the slicehost twitterstream and the network status page are worth bookmarking for quick updates.

Appears to be an upstream issue at the moment, we’ll have more details later in another post. Our apologies for the downtime and flurry of pages/emails/calls you all likely received.

Update 2145 GMT – issue appears to be resolved.

Heads up – we’ll be switching to new versions of our backend software and SliceManager Monday morning. manage.slicehost.com site may experience minor outages as we move things into place and squash any last minute bugs. There will be no service impact to your Slices during this time, but the webconsole, rescue mode, adding slices and upgrades will be unavailable while the site is down.

We’ll be around Monday to keep an eye on things, if after logging into the new site you notice any problems – please let us know. Once things have settled down, we plan to resume new signups and have some exciting news to share.

Background info – Pickled Onion has a new page on the wiki up with more details.

I’m sure all of you Slicers-to-be have noticed the waitlist estimates growing longer over the past few weeks and we feel your pain. The new customers are pouring in and we’re working hard to get people in the door as fast as possible.

Getting the bad news over with: although you can still add yourself to the waitlist, we are temporarily halting new invitations for slices to people on the list. Why? We’re planning to deploy new backend software and an updated version of SliceManager this week. As all of you know, rolling out a new piece of software is a daunting chore and we’ll be focused on this 100%.

Good news – once completed, the new software and some pending announcements will allows us to accelerate growth and hopefully blow through the waitlist quicker than ever.

Ultimately, we didn’t want to give people false time frames for getting a Slice. Our goal is to get everyone aboard ASAP and all of us are working to get it done. We thank everyone for their patience and holding out for us, the notes of encouragement and enthusiasm really pump us up. We plan to make things even better in the coming months!

It was just brought to our attention in the chat room that our form mailer at www.slicehost.com has been queuing mail for a few days instead of delivering it. If you did not receive a response, we apologize and hope to get back to you soon. Feel free to resend a message if needed.

You should have received an email today if any of your Slices need a new IP. The wiki has instructions and examples for various distributions.

One of our biggest requests is multiple IP addresses for a Slice. It’s something we’ve been working toward for sometime. While not quite here yet, we are taking preliminary steps to reach that goal. A required hurdle is the transition away from some of our current IP allocations. We have a process in place to minimize how this impacts our customers, allowing them to use both IPs for a bit to smooth the cutover. If your Slice IP needs an address change, you’ll receive an email soon with detailed instructions.

• a bit of manual work on your Slice to modify the network config
• updating external DNS records pointing to the new IP
• removing the old IP after the cutover

It’s a short-term pain, but ultimately a good thing for all Slicehosters. It grants us portability, more IPs and hopefully multiple IPs for those who need it.

UPDATE: Use this forum thread for comments/questions.

WIKI: DNS updates

We’ve just enabled a new feature that will allow you to resize your Slices.

• via the SliceManager, choose a new Slice size
• resize will begin as soon as resource availability permits
• data is copied from the old slice, preserving your IP
• the synchronization process was designed to minimize downtime
• a notification email is sent on completion and status may be viewed in SliceManager

• upgrades are prorated based on the difference in Slice price and remaining time in the current bill period
• the speed of the process and amount of downtime will vary depending on your system activity and the composition of your filesystem
• the original copy of the Slice is kept for up to a 12 hour verification period

We just sent an email to a batch of customers requesting address and phone number updates and did not BCC. It was a misclicked paste, we truly apologize for this, since privacy is one of our personal pet peeves. Please contact us if that has caused you any trouble.