Joystiq E3 Party Playtest Metrics and Pics
The Joystiq Post-E3 indie game party was great and it seemed like fun was had by everybody! Thanks a ton to Joystiq for putting it on! It was good to see a bunch of cool indie games in attendance, like Monaco, Retro City Rampage, Z0MB1ES, and Q.U.B.E.
Hopefully this tweet by party-goer Andrew Schmidt is representative of everybody who got a chance to play!
Rest assured, I’m working on it!
I’ve put some pics from the party below, but I also processed the game journals from the games played that night, and so first, here are some metrics!
There were 35 games played over about 3 hours.1
| Match Result | Frequency | Duration |
| Spy ran out of time | 9% (3) | 210.0 |
| Sniper shot civilian | 34% (12) | 173.6 |
| Missions completed successfully | 29% (10) | 183.1 |
| Sniper shot Spy | 29% (10) | 144.8 |
I’m happy with these match results for the time being, with Spies having a 63% to 37% advantage right now. If you recall from the PAX West 2010 results, the Sniper had an advantage due to the learning-while-standing-in-line phenomenon, so at PAX East, I instituted a new playtesting format, where both players would play a single game of Spy Practice first, which would let each of them learn the Spy controls and how to walk around and do missions, and then they would play connected for real. I did this because, while you can practice the Sniper role by simply watching games, the Spy role is much more physical, and you need to actually hold the controller to practice. I wanted to make sure a single round of Spy Practice would switch things over to giving the Spy the slight advantage, and this worked well. For the beta, I’m excited to watch how this balance changes as people play lots of games.
In the games where the Sniper shot the Spy, here is the last mission the Spy completed, presumably the one that gave him or her away:
| Last Mission before “Sniper shot Spy” | Frequency |
| Tranferred microfilm | 40% (4) |
| Statue swapped | 30% (3) |
| Remove microfilm from book | 20% (2) |
| none | 10% (1) |
With only 10 games ending in the correct assassination, it’s clearly not enough data to draw a lot of conclusions, so here’s the same data from PAX East, which I haven’t posted before. You can see the same general pattern of the Transfer Microfilm and the Swap Statue missions being the most dangerous for the Spy:
| Last Mission before “Sniper shot Spy” | Frequency |
| Remove microfilm from book | 19% (34) |
| Statue swapped | 17% (30) |
| Double Agent contacted | 13% (24) |
| Transferred microfilm | 12% (22) |
| 30 seconds added to match | 10% (18) |
| Bugged ambassador while walking | 9% (16) |
| none | 8% (14) |
| Hide microfilm in book | 6% (10) |
| Bugged ambassador in conversation | 4% (8) |
| Fake Banana Bread Uttered | 1% (2) |
The “none” entry means the Spy got shot before accomplishing any missions, which probably means they were walking funny.
Here are the characters choosen or not, and shot or not:
| Spy Choosen | Character Shot | ||||||||||||||||||||||||
| (Top 3 and Bottom 3) | |||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
After processing this data, it occured to me that I had the code randomly pick the number of characters requested by the level from the available pool first, before the Spy could choose which character he or she wanted to be. This disguised tendencies to pick a given character (since that person might not be pickable this game), so yesterday I reworked the way the characters are handled during game initialization. Now the Spy can choose from all the characters during casting, and then the level code fills in the remaining number of required characters. This means any bias in who people like to pick as Spies will be present in all games, even on the smaller maps with fewer characters. This change should make the beta metrics more meaningful, and it also feels better, like the Spy is casting the party, rather than the game doing it.
The Action Test results are in line with what I want, meaning most people are either ignoring it, or hitting the “okay” zone:
| Action Test Result | Frequency |
| okay | 41% (63) |
| ignored | 32% (50) |
| good | 21% (33) |
| bad | 5% (8) |
| cancelled | 0% (0) |
I also now gather the Sniper’s laser position and direction over time, but I haven’t figured out what to do with that data yet. I need to correlate it with the timings on the mission stuff, and try to make a judgement of when the Sniper figures out who the Spy is, and how long that takes, etc. Here’s an image of the raw data from one of the games:
Finally, here are some pictures I snapped at the event. It was really dark in there, but I hate using a flash, so some are kind of blurry.2
Also, a huge thanks to my friend Lisa for helping out. She hadn’t even held a 360 controller before I trained her a couple days before, and she picked it up quickly and even shot me a couple times in practice! Speaking of shooting me, you might notice Alex Baard showing up yet again. I couldn’t let him play any of the newbs, so I played him as we were shutting down. He beat me twice! I need to finish up this beta server stuff so I can get back to playing my own game, I’m getting rusty!
why not use the laser positions to make a sort of 3d heat map ?
should give some insight on what the snipers focus on (for example, the bookcase or maby more on the general)
this might also give away positions that snipers find, that give the best overview for observation ?
its just a random idea ^_^
Looking forward too the beta (but not now… 2 weeks of exams first..)
good luck
Yeah, but a lot of Sniper behavior is character-relative, meaning tracking individuals (whether correctly or incorrectly), and it’s going to be hard to visualize that. Plus, as Snipers get better, they tend to just park the laser on the window frame or above or below the level.
Mmm… data. I swear I was a statistician in a previous life.
I love the concentration displayed in the player’s faces from the event. Normally gameplay shots, (such as e3), show more of a relaxed gameplay. This shows that the player is pouring every bit of attention they can muster into the game. What would be cruel is to have one of the scary maze game reactions.
Freaking consumer grade cameras. Megapixels mean absolutely nothing without a good lens/hardware. I can blow up a picture to 500mp, but it doesn’t make it a better picture. Also, don’t get me started on digital zoom.
As a side note, if you’re looking for some form of auditory entertainment while programming, (I listen to podcasts frequently while doing web dev), a great podcast to listen to is called We’re Alive. It’s reminiscent of the radio dramas of yesterday, but with a modern feel. The plot revolves around the zombie apocalypse, and what it would be like long term.
—
A quick question about the Sniper shoots Spy data. Were there any missions that the spy was never shot after?
I love that it strips even fake html tags. the first line was supposed to have </hyperbole> following it.
Flash is quite horrendous in my experience as well, it completely destroys the atmosphere and the colors.
Some nice pictures otherwise.
When the sniper shot a civilian is there any way to separate the sniper picking the wrong person versus the sniper running out of time and panicking/guessing?
I’m guessing, (based on the amount of data checker has been journaling), that he kept tract of time of completion. However, he has “duration” for the latest journaling, but not from PAX East.
Yeah, I’d have to make some guess based on when the shot happens relative to the clock, but that seems doable. A lot of people wait to shoot, though, just to make sure, so it wouldn’t be totally accurate.
Try visualizing just the sniper *shot* trajectories (color coded on success/fail and how long from time running out)
As to megapixels vs light resolution, larger pixel sensors would integrate more light per pixel, but of course you can get a similar effect by just downsampling the photograph yourself, you just need a camera that lets you set the shutter speed manually or that lets you say “severely underlight it” so it uses a fast enough shutter time. If you can get RAWs this might even actually produce useful results, haha.
Yeah, I’ve thought about downsampling (it’d have to be additive, not filtered) but these cheap cameras apply so many “noise reduction” algorithms in low light situations that I’m not sure it would produce anything useful. Plus, getting raw out is usually impossible unless you have a hacked bios, which I don’t think exists for my camera. Etc. etc. Sad.
Not so! Canon has a hacked firmware!
CHDK.
http://chdk.wikia.com/wiki/CHDK
Woops. got to read closer. “I don’t think exists for MY camera”
Yeah, Canons have awesome firmware hacks. I use a Panasonic Lumix for the awesome 12x optical zoom and 30fps 720p, but no firmware hacks that I know of. :(
Late to the party – but some of the Canons (S95 in particular, which I’ve just bought) have a ‘candlelight’ mode which does exactly what you mentioned in the footnote – downsample to 2MP and combine pixels for better low-light performance :-)
I’m thinking about interesting ways to visualize the sniper data you’re collecting. I guess my initial thought is to farm it out to one of the local university visual design students or something. (Uh, they do have a project-based visual design program, no?) Failing that, I would start by highlighting the info you’re already collecting. Alternatively, maybe take a page from Valve’s TF2 heat maps, so: successful shot, failed shot, time spent at each view, etc.?
My crazy idea originally was to feed it all into a flash app that would draw the time lapse of the events over the level, but then I realized I just need to make a spectator mode and record the whole thing! So, I’ll use the spectator recordings for any qualitative info, and try to limit myself to using the journals for datamining types of things.
I think you will come to find the collection of sniper scope data invaulable whe it comes to level design, and arranging objectives. Case in point: the 2 most difficult objectives for spy to complete are those that are closest to the window, with relatively little obscuring or distracting the sniper.
Additionally there is very little use for the area by the angled window.
Since I am not in the beta, yet;), I can only conjecture that there are minimal spy objectoves that require the sniper to utilize that particular view.
Sorry not the angled window, but the windowed corner of the room.
The windowed side is the best one for getting a view of both bookshelves, though, right down the wall, so it’s actually pretty heavilty utilized, at least anecdotally speaking. But you’re right, I should definitely check that assumption with the data. I can overlay all the laser info to make a Sniper heat map, which is definitely a cool idea.