When you’re making an online multiplayer game, you have to worry about the community that builds itself around your game.  In the beginning the worry is, “Will there be a community at all?”  If you’re lucky and don’t screw things up too badly, and people like your game, a community will start growing around it, and as the designer, it’s your job to nurture it and help it stay healthy.  The analogy with parenting is pretty obvious.  And, just like with parenting, while you have some influence over the community’s health, in a lot of ways all you can do is set a good example with your own behavior and the feel of your game, and hope for the best.

One of the aspects of SpyParty that I’m most proud of is its community.  It’s really hard to describe how it feels to play the game, or hang out in the lobby, the forums, or the TeamSpeak channel, or watch a stream, and see all these people who were strangers to each other before they started playing your game…and they’re friends now.  What’s more, they’re welcoming new people into the community with open arms, setting their own good examples and helping it stay healthy!  Let’s just say it’s a pretty great feeling.  It’s an even greater feeling when you know you’re making an insanely hardcore player-skill competitive game, and they’re still all helping each other out, both beginners and elite players.

Another aspect of SpyParty that’s incredibly important to me is diversity, in terms of allowing people to choose characters that fairly represent a wide array of different colors, body shapes, ages, genders, sexual orientations, and physical abilities.  In fact, I have a goal to make SpyParty the “most diverse game ever”, in the sense that you will be able to kick somebody’s ass straight into next week as the Spy while playing as a queer octogenarian of color in a wheelchair.  You might move a little slower, but you’re low to the ground, so it’s easier to hide that listening device!  I am very lucky that the fiction of SpyParty is compatible with making an intense player-skill game that can still be diverse in this way.1 You are unlikely to see this kind of playable character plausibly slashing dragons with a sword or shooting space aliens, but as a Spy at a cocktail party?  I say it’s perfect cover!

You can see a start to see this diversity in the initial batch of new characters we’ve revealed:

They’re a good start, diversity-wise.

I’ll talk more about this “in-game diversity” in another post.

Today, I want to talk about a complementary kind of diversity, that of the community.  I don’t have any data on this, but I think these two kinds of diversity go hand-in-hand, because if the game is setting a good example with its design, gameplay, and aesthetics, I think it’s probably more likely the community will follow suit.

But, a more important part of community diversity is the community valuing inclusiveness and speaking up for it.  Below is an example from last week in the beta forums.  The influx of new people from the open beta happened, and one of the community members felt the need to post a note to the new players.  Some mature discussion happened, including some respectful disagreement, and people arrived at a consensus.  This is not how conventional wisdom says it’s supposed to work in hardcore multiplayer gaming communities on the internet, but yet, it can if everyone wants it to work that way.

I thought this thread was a perfect example of the SpyParty beta community in action.  Some of the players posting in this thread are at the top of the leaderboard, the best players in the game.  Way better than you or me, and I made the damned game.  But they’re also respectful, humble, mature, and open to the fact that there are other ideas in the world that may not come naturally but are worth the effort to consider, and they’re working to keep the community a healthy environment for all kinds of people…a healthy environment to ruthlessly shoot each other2 over and over again.

If you’d like to be a part of this kind of community, you are welcome to join us!

1. There are other important kinds of diversity SpyParty probably can’t explore effectively, like class and income diversity. [↩]
2. and the occasional innocent civilian [↩]

I just realized that since I finally opened the beta, I can now post the release notes for the new builds here publicly without getting a torrent of “Send me an invite!” blog and facebook comments and tweets! Yay, I am so happy to be out of the invites business!

I’ll do a post with numbers from the open beta launch sometime next week, but it’s going great, and I’m super happy and relieved. I was really worried about the community, and all the new players are blending into the community really well, and it all feels very healthy. Phew!

This build was almost exclusively bugs and minor features for supporting lobby rooms with lots of people in them. Most of these were discovered as the result of the human load test we did right before opening the beta. The main problem was that with lots of people in the lobby, you couldn’t really mouse-over somebody to reliably invite them into a game, so I struggled with how to solve this until somebody in the lobby said, “hey, there’s a problem because the player under my mouse keeps changing”, which pointed me to a reasonable and obvious solution: keep the player under your mouse fixed, even if they’re moving around in the sort order (due to other players joining or leaving, or due to them changing state). This is slightly weird, because it means you can get into a state where the lobby has scrolled such that there’s a blank space above or below, but it works pretty well.

Edge behavior for fixed-point mouse selection.

I created a new “release notes” tag, so you’ll be able to find all of the posts about new builds. I’m also going to start “streaming the release notes”, meaning I’m going to start up a live stream on the SpyParty twitch.tv channel, and walk through the changes in the build on video. Then I can put that up on the SpyParty YouTube  channel for later view. The idea here is to capture a bit of the coolness of a well-edited video of changes, like the Overgrowth folks do, but without spending the time editing the video. It’ll be rough, but it will keep me from triggering my perfectionism neurons and fiddling forever with the clip. So, hopefully it’ll be a good comprimise. I wanted to do a stream for this build, but it was 2:30am when I finished it, so I couldn’t pull it off. Luckily, drawnonward and canadianbacon were streaming when I was working on it, and they convinced me to add 3 Known Missions on Veranda for craziness. You can see that video here, it’s pretty close to the kind of release notes streaming I’m going to do…casual, chatting in TeamSpeak, but previewing the features in real time.1

Here are all the changes for version 0.1.2758.0:

• k3 enabled on veranda…k3 hard tells anyone?
• true up the triangles and the text in selected but not necessarily enabled missions
• better /who /stats formatting
• /help take command for single line help
• shrink lobby a bit more to avoid red text and make /away visible
• fix 00:00 event time bugs in pending mission state
• stop accidentally filtering suspected da cast events
• make lobby details a tiny bit more compact…user /stats username for full stats
• practice mode on lobby escape menu
• invites beep on whisper settings or chat settings
• make –console –logstderr dump log to console window for debugging startup problems
• better /who stats, idle, away on own line for copy
• /statsroom /whoroom /wr
• sort by column in lobby by clicking on the column titles
• sort /who clients correctly, by current lobby sort
• new fixed-point list ui for list stability, the player under your mouse will stay under your mouse, even as they change state, it’s weird, but will avoid misclicks
• fix bug with not setting changed number on destroys
• don’t output warning if can keep current completion string
• fix whisper (lobby) bug
• correct single-line chat paging with room masking…goes back in time in all rooms simultaneously, which is slightly weird
• beeps respect room mask too
• allow tab room changes in lobby settings and room chooser
• make lobby say messages room based, so can be masked
• only change chat usernames for completion if clients join/leave
• clients stick around exiting if leave lobby, and leaving room for 2 seconds
• sort lobby list by state with invites and exits in prev state, playing by timestamp

Those are just the raw notes I post in the private beta forums,2 so they might not mean much if you’re not a player. The way to solve that problem is obviously to become a player!

Up next, a couple super-minor fixes to this build, and then a brand new mission! I haven’t added a mission in a long time, so this will really change things. I’ll talk more about that in another post, but I’m hoping to have it stood up this week.

1. And, watching it myself from his point-of-view I just found a couple bugs with the update system! [↩]
2. …which I might make publicly read-only some day, they’re such a wealth of information about the game! [↩]

I made a link for you:  http://www.spyparty.com/beta

Oh man am I tired.

Update:  The loadtest was a success, thanks everybody!  It was fun to hang out and chat, and a bunch of other indie game developers showed up as well, including Davey Wreden (Stanley Parable), Alexander Bruce (Antichamber), and David Rosen (Overgrowth), which made it in to a bit of an indie lovefest loadtest!  Fun seemed to be had by all!

I wasn’t worried about the server’s performance this time, since the robots pounded pretty hard on it, I was mostly interested in how a crowded lobby would feel as a player.  We hit 170 people at one point, which meant we filled up Headquarters and people created overflow rooms, so that was good.  Chatting was crazy and hectic, as you’d imagine.  I need to move the Practice button from the bottom of the lobby list or no one will ever see it.  I found a few different things like that I need to fix, but best of all, I got a suggestion for fixing the lobby list moving around while you’re trying to select somebody, which is awesome!  I ♥ my players!

Beta tester KY called this “Mount Loadtest”

Original Post:  I’ve been trying to write this blog post for three days.  I am…let’s just say…a little frazzled.

As you may have read, I recently finished optimizing the lobbyserver using robots in the cloud1 to loadtest it.  I then moved everything (including this site) to a new faster server, all in preparation for opening up the Early-Access Beta.  You’ll also notice, if you clicked through that last link, I’ve disabled beta signups now, and I’ve finally sent all the invitations, including around 5000 to people who had never been invited, and over 13000 to people who had been invited before, but hadn’t responded.  If you missed signing up for the beta, don’t worry, you’ll be able to play very soon, just check back here or follow the game on one of the social media outlets you can see on the sidebar.  I still need to send a few hundred here and there for various reasons, but I’m done with the bulk of the invitations, thankfully!  I am so happy to be out of the “invite business”, and almost into the “just buy the game and play it now” business.  Getting mail delivered on the internet in 2013 is a disaster.

On the topic of almost opening the beta, and all the new people playing the game, I want to run a loadtest with actual humans tonight.  Yes, I know, this is short notice, but I’m barely treading water here, so it’s the best I can do right now!  I need to test with lots of humans in case the robots didn’t do a good enough job pretending to be humans, testing-wise.  Trust me, they were very boring to talk to.

Sorry this is not a great time for most of the US or the world.  If you can’t log on that late, come earlier, and maybe everybody who can’t go late will make their own crowdsourced loadtest before the official one!  But, if you can be on at 10pm, whether you decide to come early and play or not, please be there!

Here’s the player graph for the lobby over the past week.  This is averaged data, so it’s a little low, but I’m told by zerotka, the  resident SpyParty historian, that the max was 53 simultaneous at that spike on Sunday evening.  Let’s see if we can beat that.  Over 100 would be awesome, because the room size limit is 100 so it’d be nice to test that code with humans.

I may try to stream the loadtest at the SpyParty twitch.tv channel.  Not sure how exciting watching a lobby list and chat is going to be, and not sure if I’ll be panicking and fixing things and have to turn it off, but we’ll see.  I’m also going to play kcmmmm‘s 5000th game with him tonight at 10:30pm, and we’ll try to stream both sides of that.

Come play if you can!  Thanks!

1. ROBOTS IN THE CLOUD!!!! [↩]

Check out Loadtesting for Open Beta, Part 1, Part 2, and Part 3 to read the previous installments of this epic tale!

It’s been a while since the last update in this series, sorry about that!  At the end of Part 3, I mentioned the SimCity launch giving me pause about my goal of testing the SpyParty lobbyserver to 1000 simultaneous robots.  Well, I got scared enough after their launch that I increased my optimization target to 2000 simultaneous robots on my old and slow server, and then I also decided to bite the bullet and upgrade the server hardware after I hit 2000 to give myself some extra headroom.  I really don’t think I’m going to hit these numbers at Open Beta launch or even for a long time after that, but I’d rather err on the conservative side and have it purr along nicely.

Since I waited so long to post this Part 4, I can’t really give a play-by-play of all the optimizations I did as they happened, so I’m going to give the general arc I followed, and then talk about some of the interesting stops along the way.

iprof, atop, oprofile, et al.

As I mentioned at the end of the last post, I’d fixed some of the huge and obvious things with the network bandwidth usage, so it was time to start profiling the CPU usage.  There are lots of different kinds of profilers, but the one I use the most is based on Sean Barrett’s iprof.  I’ve modified it a fair bit over the years,1 but the core of the system is still the same.  It’s a runtime profiler that requires instrumenting your code into blocks, it’s efficient enough that you can leave it on all the time as long as you don’t stick a “prof block” in an inner loop, and you can generally see where you’re spending your time hierarchically.  It can draw to the screen, but I also have it output to a string, and so on the lobbyserver I can have it output to the log after a spike, and also catch a signal I send and it’ll force a prof dump.  Here’s an example:

2013/04/17-16:12:10: 85.156 ms/frame (fps: 11.74)  sort self - current frame
2013/04/17-16:12:10: zone                                                     self     hier    count
2013/04/17-16:12:10:  ProcessMessages                                      59.2910  59.2910     1.00
2013/04/17-16:12:10: +Send                                                 17.8164  18.4493  1120.69
2013/04/17-16:12:10: +ClientsUpkeepAndCloseLoop                             2.3034   2.6989   793.15
2013/04/17-16:12:10:  Log                                                   1.3559   1.3559    25.97
2013/04/17-16:12:10:  unpack_bytes                                          0.7311   0.7311    26.56
2013/04/17-16:12:10:  unpack                                                0.6674   0.6674  3551.71
2013/04/17-16:12:10: +ClientsPacketLoop                                     0.5492   3.7160   792.58
2013/04/17-16:12:10: +ClientsUpdated                                        0.4023  10.9843     0.56
2013/04/17-16:12:10: +SendQueuedClientRoomMessages                          0.2524   7.3086     1.00
2013/04/17-16:12:10:  FindClientByID                                        0.2494   0.2494   267.05
2013/04/17-16:12:10: +JournalQueuedSave                                     0.2374   0.3882     1.43
2013/04/17-16:12:10: +Tick                                                  0.2112  25.6313     1.00
2013/04/17-16:12:10:  iprof_update                                          0.2051   0.2051     1.00
2013/04/17-16:12:10: +JournalSavePrep                                       0.1560   0.5442     1.43

As you can see, it’s pretty easy to read, and you can drill down on individual blocks and see who calls them and who they call:

2013/04/17-16:12:10: 85.156 ms/frame (fps: 11.74)  sort graf - current frame
2013/04/17-16:12:10: zone                                                     self     hier    count
2013/04/17-16:12:10:     LoginReply                                         0.0006   0.0006     0.01
2013/04/17-16:12:10:     JOINING                                            0.0007   0.0008     0.03
2013/04/17-16:12:10:    +TYPE_CLIENT_GAME_ID_REQUEST_PACKET                 0.0011   0.0014     0.44
2013/04/17-16:12:10:    +NewWaitingForJoinClients                           0.0019   0.0019     0.01
2013/04/17-16:12:10:    +TYPE_CLIENT_PLAY_PACKET                            0.0086   0.0087     0.12
2013/04/17-16:12:10:    +TYPE_CLIENT_INVITE_PACKET                          0.0213   0.0225     2.37
2013/04/17-16:12:10:    +TYPE_CLIENT_IN_MATCH_PACKET                        0.0280   0.0282     0.49
2013/04/17-16:12:10:    +JournalQueuedSave                                  0.0563   0.0571     1.18
2013/04/17-16:12:10:    +RoomsChanged                                       0.0852   0.0939    15.38
2013/04/17-16:12:10:    +NewInLobbyClients                                  0.1155   0.1342    14.55
2013/04/17-16:12:10:    +ClientsUpkeepAndCloseLoop                          0.1362   0.2093   150.55
2013/04/17-16:12:10:    +TYPE_CLIENT_MESSAGE_PACKET                         0.3122   0.3170     8.87
2013/04/17-16:12:10:    +SendQueuedClientRoomMessages                       6.7122   7.0013   501.99
2013/04/17-16:12:10:    +ClientsUpdated                                    10.3361  10.5720   424.67
2013/04/17-16:12:10: -Send                                                 17.8164  18.4493  1120.69
2013/04/17-16:12:10:     unpack                                             0.6329   0.6329  3362.06

This is super useful.  The biggest downside to it is that it’s not thread-aware, but I’ve made it thread-safe via the brute force method of having it ignore all threads that aren’t the “main”.  My code is mostly single-threaded, but the threadedness increased a fair bit during these optimizations, so I hope to eventually modify iprof to be thread-aware without losing too much simplicity and performance.  However, until I make those modifications, any background thread activity will show up attributed to one of these main-thread blocks.  You can still get useful data, you just have to be aware of this.  For example, ProcessMessages in the loop above is hiding a WaitForMultipleObjectsEx call on Windows, or a call to select/epoll on POSIX, so it’s not actually taking that much active time on the main thread.

I also used oprofile, which is a nice sampling profiler on Linux that can profile per-thread using just the debug information in an application, and atop for keeping track of things happening on the machine as a whole.

Here’s a list of the stuff I ended up optimizing:

• I was originally sending out the chat messages to all clients as they came in, but I started queuing them up and sending them all out at once to reduce send calls.  Of course, once you do this, you have to make sure you don’t overflow the network packet if you have queued a lot of messages that tick, so that makes the code more complicated and harder to modify, which is a tradeoff one often has to make while optimizing, and it’s why you want to put off most optimization until you need it…although you should have a rough plan for how you’ll optimize a piece of code in the future even if you write it the dumb way first.
• I made more threads, including putting network sending and receiving on separate threads, making a separate thread for logging, and a thread for saving files to the disk.  There were already threads for talking to the database and Kerberos, for receiving network packets, and for checking for new client builds.  These are all relatively simple threads to add, because they’re all just throwing data into a queue on one thread and taking it out on another, although multithreading a program always makes it harder to understand.  I discovered a fair number of deadlock bugs in libjingle, the library I’m using for NAT traversal and some cross platform threading stuff, and I’ve fixed some of them.  I’ve veered far enough from the original libjingle code that I’m probably just going to have to put my version up as a fork, sadly.
• I timesliced the login phase for the clients.  Previously, when a client would log in, I’d process a bunch of stuff immediately, including some authentication stuff which can be somewhat time consuming.  In a load test where hundreds of clients log in to the server at the same time, this would bog down, so I now process a maximum of 20ms worth of clients each tick.  This makes some clients wait a bit longer before they’re logged in, but doesn’t result in a positive feedback loop where there’s a really long tick, so a lot of packets will have arrived while it was happening, so the next tick is really long too, etc.
• Like the player list packets, I also made the room list packets incremental, and able to span multiple network packets.  This way all the lists of players and rooms that the lobby sends to the clients can be differential and arbitrarily long, so there’s no more hard limit on the number of clients that can join the lobby.  I think there’s actually a bug in this code, but I’ve only ever seen it once, even after tens of thousands of robot sessions, so I just hope it shows up more at some point.
• I switched the POSIX networking inner loop in libjingle from select to epoll.  This was not so much an optimization as it was simply to allow more than 1024 sockets to work at all.  epoll is also a lot faster, but I’m currently kinda using it in a dumb way, so I’m not benefiting from that speed boost much yet.
• There were also a bunch of smaller traditional code optimizations, like using maps to cache lookups, using free lists to avoid some allocations, and whatnot.  Oh, and don’t forget to change the ulimit -n settings in limits.conf on Linux, so your process can actually accept a lot of connections!

As I was doing these optimizations, I would run a loadtest with a bunch of robots and profile the lobby.  I was at 500 robots at the end of Part 3, and I slowly raised the ceiling as I improved the code over the weeks:  569 robots…741 robots…789 robots, 833, 923, 942, 990, 997, 1008, 1076, 1122, 1158, 1199, 1330, 1372, 1399, 1404, 1445, 1503, 1614, 1635, 1653, 1658, 1659…

When I hit 1659 it was late one night, and so I stopped for the day.  When I resumed work and did the next couple of optimizations, I figured I’d get it to 1800 or something.  I always launched 20% or so more robots than I was hoping to support in a given test to account for internet and EC2 variation, and for plain old bugs in the clients that would sometimes manifest themselves, so this time I must have launched 2500 robots, because when I looked up from the profiles running in ssh terminals and over to my SpyParty client logged into the test lobby, I saw this:

This wasn’t supposed to happen yet.

Uh, I guess I was done optimizing?

I was actually kind of disappointed, to be honest.  I had all sorts of cool ideas for optimizations I was planning to do that I’d come up with while testing and profiling the code, and now, if I was going to follow my own plan and stop when I hit 2000 simultaneous robots, I would have to just take a bunch of notes for next time I optimized so I could pick up where I left off, and move on.  The good news is I’m pretty sure I can make the lobby almost twice as efficient if and when the time comes to do that!

Room at the Inn

If you look closely at that screenshot, you’ll see the thumb on the scrollbar for the player list is pretty small.  That’s because all 2010 players are in a single room, which is not going to work very well for a lobby full of real people.  In fact, the only reason there were 2010 players in that room was because at the time I’d limited the room size to 2010 because I didn’t want to bother teaching the robots how to use rooms.  There were actually a few hundred more robots knocking on the door but they couldn’t get in.  But, now that I’d hit my 2k target, it was time to fix that.

I immediately realized I had a problem.  Currently, when you connect to the lobby, it sends you a list of rooms, and you have to pick one to log in.  But, what if the rooms are full?  Oops, you couldn’t log in.  So, as soon as I set the room size down to something more reasonable, like 100, then the first 100 robots got in and the rest just sat there failing to join.

It seemed like there were a number of solutions to this problem, including allowing players to create new rooms before logging in, but in the end I went with the simplest and most robust solution, which is to have the lobby create a new empty room if all the current rooms are full.  The initial room is always called Headquarters, so I named these new dynamic rooms Headquarters 2 and onward.  Very creative, I know.  Somebody suggested using spy movie titles for these room names, but I figured that wouldn’t scale very well, ignoring the potential copyright issues.  If the lobby ever finds one of these dynamic rooms empty, it kills it, unless all the other rooms are full.  I also have the lobby automatically put you in a now-guaranteed-to-exist-non-full-room if you log in and try to join a full room, even if it wasn’t full when you clicked on it, so this eliminated a login race condition too, which is always a good sign.

This last bit also made it so I didn’t need to make the loadtesting robots know very much about rooms:  they always try to join Headquarters and if they don’t end up there, oh well.  As they join, they kind of spill over into the latest dynamic room until it fills up, and then they continue to the next, kind of like filling up an ice tray with water from one end.  I should probably make them test the actual room features by creating and changing rooms and whatnot, but the single giant 2010 player room was a way more intense loadtest than having 20 rooms with 100 players in each due to the chat broadcasting.

I don’t know if 100 is the right limit for room populations.  100 would still be way too many people to have in a single reasonable conversation, but I didn’t want to put too low of a limit on the size before I have tested things with humans instead of just robots.

The Client

There’s this annoying thing that happens when you’re testing computer code, and it’s that you encounter problems and bugs not only in the code you’re trying to test, but also in your test code.  This was no different.  I was constantly fixing various bugs in the robots that would keep them from all connecting correctly, and I even made sure some of the optimizations helped the client side so I could run more robots on a given EC2 server.  Plus, just making sure the robots keep trying to connect and login was important, because if there was a timeout due to an initial burst, you want them to try again automatically after it dies down, rather than just sitting there not doing anything.

As I said in Part 1, I started out running about 50 robots on each m1.small EC2 instance.  That didn’t scale, for some reason I’m still trying to figure out.  That worked okay with a low number of instances, but as I increased the number of instances, I had to lower the number of robots on each instance, eventually to around 20 per m1.small.  An AWS account starts with only being able to start 20 instances, so I did a total of two instance limit requests to Amazon, first to 100 and then to 300.  It’s scary to have 300 instances running…even though m1.small instances are only 6¢ an hour each, that’s still $18 an hour when there are 300 of them running, and Amazon rounds up to the hour, so if you miss shutting them down by a minute you just lost a large pizza!  It looks like Google’s new Compute Engine thing is about twice as expensive for their somewhat similar low end machine (ignoring performance differences), but charges in 1 minute increments after the first 10, which might be cheaper for this very transient use-case.

I seem to remember reading somewhere that Amazon allocates instances for the same account to the same physical machine if possible, which might explain this scaling problem, since it means I was probably maxing out a given piece of server hardware with too many instances bursting at the same time.  It’s hard to tell if this is the case, and I need to do more testing before saying for sure.  A commenter said there might be a packets-per-second limitation in EC2, as well, but I haven’t verified that.  Once I’ve tried a few different things, I’ll do a long technical post on chrishecker.com about EC2, linode, and my dedicated host machine, comparing the different results I got.

Finally, I had to do some optimization on the SpyParty game client when the numbers started getting high.  I went a little nuts with the chat system early on and it has completion on all commands, room names, and player names, but the code that builds the completion tree was calling the memory allocator 35k times per update when the numbers of players got high, so I had to remove some of the stupid in that code as well.

The New Server

With all that done, and 2010 robots running on the old server, I haggled with my hosting provider and started renting a newer and much faster server.  I use SoftLayer for dedicated hosting, and have for years.2 My old server was a Pentium 4 with a single hyperthreaded core, 1GB ram, and a 100Mbps uplink, and the new server is a Xeon 3460 with four hyperthreaded cores, 4GB ram, and 1Gbps uplink, so it’s slightly more expensive but a lot faster.  That said, everybody seems to be using VPS hosts these days.  I talked to some other indie game developers, but I didn’t have time to do a full evaluation of the tradeoffs, so went with the devil I knew, so to speak.  It seems like VPS is going to be a bit slower but also a bit cheaper, but the big advantage of VPS to me is that you can move the virtual machine image to faster hardware and have it up and running again in minutes.  That’s a pretty great scaling sweetspot between having a single physical server and praying it doesn’t melt, and a scalable system that elastically uses cloud computing The Right Way™, but it’s also hundreds of times easier to get a VPS image working and then move it to a faster machine than it is to scale elastically.  So, I dunno, it’s definitely something worth looking into more during the year as I see how things are scaling.

The new server ate the robots for lunch:

The new server works pretty well.

For reference, 4850 simultaneous players is pretty far up the top 100 Steam games by player count, so I don’t think I have to worry about those numbers for a while.  Here’s atop’s view of things:

Well within parameters.

What’s Next?

So, that’s it for the lobbyserver loadtesting.  Now I need to move the website and registration system over to the new server, test them a bit, and start inviting everybody in in big batches.  Soon I’ll send out email to the beta testers to set up some scheduled human loadtests as well.  The robots will be jealous, left out in the cold, looking in at all the humans actually playing the game. 

Open Beta is fast approaching.

1. I’ll  release my changes at some point. [↩]
2. Well, they were servermatrix when I started, and then The Planet, and now SoftLayer. [↩]