I’ve been a bit quiet because I’ve been working on (hopefully) completely invisible stuff involving backend server scalability.  What does that mean, you ask?  In practical player-facing terms, it means I’m trying to get the lobby and registration system robust enough to first invite the rest of the waiting list in,1 and then to open the beta up completely, allowing people to hear about the game, visit the site, pay their money, and get into the beta instantly.  Between 10 and 500 people sign up a day, depending on how much press the game is getting at the time, and I’m very curious to find out how many of those will join the beta if it’s available immediately, and they aren’t told they have to wait.

I’m currently working on the text-mode robot client.  I’ve got it logging into the lobby and pretending it’s a full SpyParty client, and it can chat, and invite people to games.  I posted on facebook and twitter asking for suggestions of what the robots should say to each other as they’re hanging out in the lobby.

Next up, I’m going to get the robots playing fake matches and games and reporting the game results to the lobby, and logging in and out a bunch.  Then, I’m going to hack up the bees with machineguns load testing app to launch a bunch of EC2 micro instances with multiple robots running on each, and aim them all at my test server.  I expect chaos.

Once I’ve got the obvious stuff fixed, and have figured out how many clients my system can host at a given machine size, I’ll do some tests with beefier machines to make sure it scales linearly.  There are four basic machine resources: CPU, memory, network, and disk…I assume I’ll run out of memory first, but I don’t know for sure.  I was originally going to try to get the entire backend infrastructure running in the cloud, but I think for the near term I’m going to just make sure I get a machine that can accept way more clients signing up and playing than I think I’m going to get when I open up the beta, and hope it holds up.  If it dies due to too much traffic, well, I guess that’s a good problem to have, at least!  I did this same kind of thing with when I started taking beta signups, and I stress tested and optimized it well enough that it didn’t even break a sweat, so hopefully the beta launch itself will go as smoothly. If not, then if Blizzard can do it, so can I! Yikes.

Then, once I’ve got the backend scalable for the robots, I’ll start inviting much larger groups into the beta to loadtest with live players.  There are some features I’m going to need to add to make this work, like chat rooms and colored chat text, and it’s going to be pretty raw to start with, but it should work okay.  The beta testers have been very forgiving of my incredibly primitive lobby so far, so hopefully that attitude will continue!

After everybody’s in, I’ll shut off the signups and let it stew for a few days, and if it’s working, I’ll open it up!  I hesitate to give time estimates on this stuff, because I’ve never hit a date, but most of this will be happening over the next few weeks.

Asserts!

Programming is complicated, and handling errors makes it even more complicated.  Oftentimes, it’s good programming practice to not handle some types of errors gracefully, but simply assert that you aren’t in a state you shouldn’t be in, and if the assertion fails, you exit with an error.

assert(1 + 1 == 2);  // integer arithmetic better work or we're hosed

Of course, during development, the impossible is not only possible, but likely, and so your asserts fire.  And, often in complicated code, you’ll assert things that should be true, but aren’t catastrophic if they’re not true, and so you usually pop up a dialog that gives you the choice to break into the debugger, abort the program, or ignore the assert (just this once, or forever).  The problem is, once you let yourself have that ignore option, you can get lazy, and start using asserts as popup reminders of things to fix.  This tends to be really bad on a large team, because the game is asserting every 2 seconds, and you’re just hitting ignore all the time to other people’s asserts, and it becomes a habit.  However, on an indie sized team, which in my case means exactly one programmer, one can use asserts this way and they can still be useful.  I can get a feel for how often certain types of things are going wrong while I’m testing, and I can remember what I was doing when a specific assert fired most of the time…it goes from a purely quantitative tool to a qualitative tool.

I also can leave the asserts on in beta builds, but in a way that fires silently once and then auto-ignores-forever, and then have them shipped off over the network to the server, so I can see what kinds of things are going wrong on player machines, and how frequently they go wrong.

I tend to be pretty liberal with asserts in my code, and so they fire a lot, and in turn the server logs a lot of them.  About 30000 of them so far in the beta.  Here they are, sorted by frequency:

1. currently at 16681 people as of this post [↩]