The past 2 days: what went wrong, and what's being worked on

Report a post

Thank you for taking time to help Etsy! Please note that you will not receive a personal response about this report. We will review this post privately...

Why are you reporting this post?

Any additional comments?

Edit Post

Edit your post below. After editing, the post will be marked as edited and the date & time of the last edit displayed.

Close

What is this?

Admin may choose to highlight awesome community posts that are friendly, answer questions, and offer informative links.

What does it do?

Highlighted posts are placed at the top of each page in a thread for greater visibility.

This topic has been closed.

Original Post

Hi all,

On Saturday night, Haim, Mike, and I headed to the datacenter to perform some heavy maintenance on the servers. We planned to install a new, larger switch, several new image servers, and a reorganization of the network architecture. This required system downtime.

Despite a well-documented plan (with charts and everything!), the work took much longer than anticipated. It ended up seeping overnight into Sunday afternoon, as almost everything that could have gone wrong did go wrong. We eventually got the site online, but there were still problems.

The create servers were not functioning -- they are the servers that facilitate anything involving image uploads. This affected the listings process, the editing process, leaving feedback, profile changes, and shop appearance page. Additionally, everything was running slowly, and some images were slowed to the point of timing out.

After 24 hours of working at the datacenter, we could not work any further on the issues without some rest. There is a point at which your brain ceases to function properly, and we hit that point.

After roughly 3 hours of rest, Haim and Mike continued work on the remaining problems. The main problem was pinpointed to our load balancing machines, which were sending data back and forth manically. It was sorted out, and now the site is just about at the point it was before we began the maintenance. We will spend the next days pushing it beyond that point.

Here are some things that were affected by the outage:
• All treasury lists were lost and cannot be retrieved
• Views for the last several days may have been lost for some listings – these will be retrieved later this week.
• Some email-related confirmations may not be functioning

Due to the issues experienced during the maintenance, the new image servers were not installed as planned. We will be installing them once everything else is working properly. The images are now loading about the same way they were before the maintenance, which is sometimes slow. The new servers should help us out in that department.

We recognize these issues as a failure on our part, and we’re deeply sorry for the disruption to the tens of thousands of people who tried to access the site during these times. Those with showcase dates during the effected days (Sunday and Monday) will be issued a make-up day next month.

Needless to say, we will be doing our best to prevent this kind of thing from happening again. We’re working now on getting fully operational and facilitating future growth. Keeping the site up and functioning quickly is our #1 priority, and we’re working day and night toward this goal.

If you are having any issues that are not described in the list above, please let us know here, and we’ll look into them as soon as possible.

Again, we are deeply sorry for the weekend’s problems, and we appreciate truly appreciate your patience.

Posted at 3:57pm Apr 23, 2007 EDT

Responses

Thanks so much for the info. We appreciate it. Wishing you the best in getting everything squared away. : )

Posted at 3:59pm Apr 23, 2007 EDT

Thanks for the info, RD.

Posted at 4:00pm Apr 23, 2007 EDT

Thanks RD - much appreciated ;)

Posted at 4:01pm Apr 23, 2007 EDT

cgstudio says

thanks!

Posted at 4:02pm Apr 23, 2007 EDT

Thanks for the update!

Posted at 4:03pm Apr 23, 2007 EDT

Thank you for all your hard work!

Posted at 4:04pm Apr 23, 2007 EDT

mLee says

Thanks for the info Chris! Sounds like an awful way to spend a beautiful Sunday.

Posted at 4:04pm Apr 23, 2007 EDT

juliery says

very interesting read. hope you guys all got some well-deserved rest yesterday and today! best of luck with what's next. ;)

Posted at 4:04pm Apr 23, 2007 EDT

SoCaliGirl says

Thanks RD!

Posted at 4:04pm Apr 23, 2007 EDT