Our Server Had Babies!

First, I hope everyone's safe. My heart aches for what happened in Paris last week. But like my French friend Orane said, we must go on like normal. We can't let them win.

So on to today's topic: Our server had babies!

Well, not exactly. But where there were previously 3 of them, we now have 7. Or so.

Before, we had mainly one server that ran everything, with two secondary database servers, just so we had instant backups of our primary database, in case a hard drive started to fail. It's served us pretty well for a while, but it also has some obvious drawbacks.

In order to scale, you can either make the server bigger, or you can add more servers. Making a server bigger in-place is not really possible, so you'd have to set up a new server that's bigger, then move things over. That's complicated.

That's why having a structure with one load balancer in front and multiple application servers is better, because then you can add more application servers as the traffic increases. Each individual application server is disposable, you can remove them, add more, create them over from scratch, as needed.

But the setup itself adds complexity. More moving parts means more things that can break. The loadbalancer itself can fail.

When I moved to new servers last year, I wanted to go with multiple hosts, but the complexity was not worth it at the time. So we ended up with a much simpler one-machine-for-everything setup. And apart from some recent overload due to some server-to-server notifications, it's worked well. But it was time for a change.

The biggest concrete reason it was time for a change was that I wanted to hand over management of our primary database to someone who has a team dedicated to this. That's Amazon and their database service. Amazon can run an entire database cluster with replication, backups, up-to-the minute snapshots, and make sure we never run out of disk space the way we did the other day. It lets me sleep easier at night knowing they've got that covered.

Moving to a new database server means moving to a new datacenter. You want to have your application servers and your database servers close to each other, since they may talk 100 times for a single page load. And moving to a new data center is a great opportunity to redo the entire server infrastructure and get it the way I wanted it from the start.

So we now have two dedicated application servers (which is more than we need), two dedicated background job servers (which is way overkill right now), plus dedicated servers for another (non-critical) database, testing, and a few other bits and pieces. 

Already, I'm feeling how nice it is to be able to try out something on one of the application servers, without having it take down the whole site. Or I can spin up a copy of the main database and try something out without having to impact production. It's really neat, and we should be a lot more stable and fast going forward.

Of course, with any transition as massive as this, there are going to be some mishaps and fallouts, but I think (knock-on-wood) that we're pretty much there now.

Solving Puzzles

This type of work is complex for sure, and some of it is idiosyncratic and arcane, but it's also intensely logical. There's a part of me that really enjoys figuring things out. Just get in there, read, try, think, google, and figure out how it works, and how to do what I want. Just one step at a time. Cracking one piece of the puzzle at a time.

I think it's a great way to learn things and to make things happen. It's what I did with bartending and mixology a while back. It's what I've done with music production and songwriting. My friend Jeff who's worked in the music industry his whole life said he's never seen anyone pick up music production as quickly as I did. I'm honored.

I'm not sure why I'm sharing this, honestly. It's not to brag. It's just something I've discovered about myself by doing this. It's always worth it to look out for these things that you're really great at and that you really enjoy, and make a note, so you can choose to use it in your life.

New Features in Simplero

That's pretty much it. This infrastructure project has been pretty all-consuming for me these past couple of weeks, weekdays and weekends both. I feel like I need a vacation :)

Random Links from around the Interwebs

Why New York Subway Lines Are Missing Countdown Clocks by The Atlantic. The article is way too long, but it's a fascinating illustration how technology projects go off the rails. You think you can just add one more requirement here, and a little requirement there, and suddenly something that could be done for $100,000 ends up costing $200 million and never gets finished. Getting technology projects actually finished is a never-ending process of fighting complexity. It's how I've been able to singlehandedly create a software solution that's able to hold its own against players hundreds of times bigger.

Secure passwords are important. I love this little generator.

Taylor Swift talks about making the 1989 album. So awesome.

Lots of love and peace to you all, and a happy Movember :)
—Calvin