Our New Infrastructure

2 | published by Scott Szarapka on Monday, May 02, 2011

If there's one thing I've learned over the years working in IT, it's that you're only as strong and secure as the infrastructure you live on. One of the first goals I set for us after coming on board was to shift our primary LemonStand services from their managed server to a completely custom and self-managed solution. Based on our needs assessmemt for both our primary services as well as an upcoming project, we settled on RackSpace's Cloud Servers service for our infrastructure's new home. Their API is fantastic, and will allow for some quick provisioning, expansion and backups as well as scaling. We're looking forward to some exciting new features (both internal and public) that we'll be building now that we've transitioned!

We came to the decision on Rackspace about two weeks ago, and since then we've been planning, testing, deploying and testing some more, and then testing again, then breaking things, followed by some additional planning, more testing and even more testing, which leads us to Sunday night (May 1st). On Friday we put the big yellow LemonStand stamp of approval on our new infrastrucutre, for the last week or so we had been running a nice and stable pre-production environment that was an identical mirror of our production environment. The moment we actually decided that we were happy and comfortable with what we've built, it breaks. Thankfully through our extensive planning and testing we knew exactly what the problem was, where it happened, why it happened, and how to fix it.

Now, if any of you live in Vancouver, you'll note that Saturday was a very, very important day, a Canucks playoff game! So we can totally write Saturday off - there was definitely no work being done on this transition! Sunday afternoon we once again tested some more, ensuring that each critical service was functional in it's new home, after a few dozen tests, we were satisfied and began changing our DNS records.

It's 5:08 AM PST - Monday, May 2nd and we're happy to report that as of now, we've had a perfect transition to our new RackSpace Cloud Servers based infrastructure. This was truly a team effort, and only through extensive testing and planning could we transition quickly, effeciently and without error to ensure that we minimize the impact to you, our awesome customers!

Over the next few weeks we'll be releasing some of the policies and guidelines that we use to deploy LemonStand servers for those that are interested.

Share this blog post! →