How to Report Bugs the SpyParty Way
I just wrote this post for the Bugs subforum on the Early-Access Beta site. I figured you all would be interested in reading it, since you’re all going to be invited into the beta soon.1
"he probably would"
Finding and reporting bugs is a learnable skill, and you don’t have to be a programmer or even a particularly technical person to do it right, you just have to be methodical and thoughtful. Reporting bugs the Right Way means I can spend more time making SpyParty awesome and less time trying to get a bug to “repro” locally.
What is a “Repro”?
The goal of a bug report is to describe the steps to reproduce the buggy behavior (whether it’s a crash that deletes your entire hard disk or it’s a word that’s not pluralized correctly in the user interface somewhere) well enough that I can step through them on my machine and see the behavior first-hand. A bug that doesn’t repro is very hard to fix, whereas a bug with a “good repro” is usually pretty trivial to fix. It can be the difference between 10 minutes to fix and 10 days to fix, literally. So, when posting a bug report, the key is to do some work on your end to be methodical and give steps to repro the bug as simply and as clearly as you can. It’s basically a recipe to make the buggy behavior happen for you (and me!).
This does not mean you shouldn’t report bugs that don’t always happen the same way or happen infrequently, or that you can’t get a good repro for, but it does mean those bugs will be much harder for me to fix. In other words, it’s more important to report the bug poorly than it is to not report it at all, but a good repro makes a huge difference.
Bug Reporting Steps
I’ll be updating these steps as the beta proceeds, and eventually I’ll switch this part of the forum over to a real bug database, but for now, threads in this forum will have to do. The basic rule is one thread per distinct bug.
- Try to reproduce the buggy behavior on your machine. If you crashed, write down what you were doing right before the crash before you forget it, and then relaunch the game and try to get it to crash again. If you can’t repro the bug, you should still report it with as much description as you can, but as I say above, it’s way easier to fix bugs with good solid repros. The shorter the recipe for the repro, the better. As Einstein said, you want the simplest possible repro, but no simpler. A bug that repros in Practice Mode is easier to debug than one where I have to connect over the network, for example.
- If screenshots look like they’ll help to repro or describe the bug, feel free to take them (you can do it by hitting F5 when the game is running, or use FRAPS or your favorite screenshot tool). If you hit F5, the screenshot is put in your AppData/Local/SpyParty/screenshots directory, which you can get to by going to the Start menu and typing “%LOCALAPPDATA%\SpyParty\screenshots” into the little edit box, and hitting return. I’ve enabled attachments in the forums, so please attach the images to your bug report (as opposed to using a 3rd party image service, which might delete the images after a while).
- Make sure you have the latest SpyParty build. The easiest way to do this is to launch the game and hit “Play” on the main menu, which will try to take you to the lobby. The lobby will update your build if you’re out-of-date. If you update, try your repro steps again to make sure they still work.
- Search this subforum for similar bugs. If you find the same bug, then read the thread on that bug, and see if there is a workaround, or if I think I’ve fixed it in an update, etc. If I think I’ve fixed it in that thread I will mark the subject with ” – FIXED”, and I really want to know if you have the same bug again! This is called a “regression” and it means I’m a bad programmer!
- If you find the exact bug, feel free to add more information at the end of the thread. I will try to add a “vote for posts” feature so you can simply say “me too”, but until then, feel free to add a post if there aren’t already a bunch of others doing the same, or if you have additional information or slight modifications or a simpler repro that might help debug the problem.
- If you don’t find an exact match in the current bugs, but do find bugs that seem related, keep the links to those threads so you can insert them into your bug report. A real bug database allows you to reference other bugs, so this is our ghetto version of that feature.
- Create a new thread in the bugs forum if your bug is a new one. Use a descriptive thread title. “Game buggy” is a bad one. “Main Menu text disappears after window maximized” is a good one. I might change the title after we’ve discussed the bug a bit, but starting off with something descriptive will help.
- At the top of the post, describe the end state of the bug, meaning what’s wrong, like “the game crashes”, “the statues are floating in mid-air”, or whatever. Describe what you expected to happen, and what happened instead. Attach an image showing the bug, if appropriate.
- Then put your repro recipe in the post, written as clearly and as simply as possible. You can use the bbcode list tags, or just write it out, either works. Attach any more images you have to show how to repro the bug, but you don’t have to go crazy (saying “hit Play on the main menu” is fine, you don’t need to screenshot the main menu, take it into Photoshop and add an arrow with “click here” text unless the exact place to click matters to repro the bug, for example).
- SpyParty writes out a log file every time it runs. You can find these log files in the “logs” directory (“%LOCALAPPDATA%\SpyParty\logs”). The file name for the log files has the date and time of the playtest session start (e.g. SpyParty-20120412-11-54-49.log). Attach the log file that goes with you reproing the bug with the steps you’ve posted. Similarly, if your game crashed but said it could not upload the crash dump, then attach the correctly dated dmp file from the logs directory. Otherwise, SpyPartyHelper uploaded the dmp file to my server so I can access it, but mention this in the bug report.
- If there’s private information you need to send with the bug report, mention this in the report itself, and then send mail to support at spyparty.com.
- If you want to attach a really big file, say over 2MB, please mention this and I’ll figure out a place for you to put it, please don’t mail giant files to support at spyparty.com.
That’s it for now! Feel free to reply to this post with suggestions of how to improve the bug reporting process and I’ll edit this post.
Eventually I will add a way to report bugs from inside the game that will take a screenshot and everything automatically. Won’t that be nice?
- for a suitable definition of “soon” [↩]
Checker, you should use http://www.mantisbt.org/
Yeah, I haven’t even really started researching which one to use yet, too swamped with other stuff. I figure the subforum will work fine in the near term.
“Never underestimate the power of stupid people in large groups”
While you probably won’t be inviting large groups initially… especially when it’s easy to submit bug reports, you’re probably going to get many duplicates of every bug. I hope I’m wrong though!
@$#%&@*((*&%$!%&*()
I agree with quirk. When TeamSpeak 3 finally went into open beta their bug report forum exploded with bugs. Most of the reports were users lack of knowledge with the program and or their system. Which I think is unfair to blame the user for not being tech savvy. Not to mention all the duplicate posts.
You mentioned about crowd sourcing to a db and I think that’s a good choice. Let a few users sift through all the reports and convert them to the db for you. Maybe try and get a better How To and such. It seemed to do a good job for them.
Keep up the good work!
I jumped of my chair when I hit refresh and saw a new post.. Looking forward to help you and bring some bug reports to that reclusive Early-Access Beta forum.
http://i.imgur.com/0qVzV.jpg
:D, how did you get my picture?!
I am the boss of the internet.
New Spyparty post day is always awesome
Haha, that’s almost exactly how I looked when I saw this post, except that I looked more like this: http://thumbs.dreamstime.com/thumblarge_291/1216600699r47Z25.jpg .
Bring on the bugs! I’ll go through ’em like a Fear Factor contestant.
Not to influence your choices during the next invite wave, but I graduated Oxford with a first-class degree in Bug-hunting (not to be confused with bug-chasing). I’ve logged about 4.2 million bugs across thousands of titles, a feat that has earned me the title of Big Billy the Brainy Bug Baron among the inner circles of game development. I’m kind of a big deal.
Jack high… Royal Flush hmm, you got me ;)
Let’s hunt some bugs! Like this:
http://www.youtube.com/watch?v=AKlkD-D20OI
The most important rule of reporting bugs: Never say that something “doesn’t work”, or “is broken”. After describing what you did, say what you expected to happen, and what happened instead.
Haha, I have a picture for that too: http://rlv.zcache.com/its_not_a_bug_its_a_feature_card-p137081274885571207z85p0_400.jpg
Good point, I will add that!
Updated! You get bold text, too.
Aren’t twelve-step programs supposed to help us break our addiction to SpyParty, not make us [i]more[/i] addicted?
Would using a tracker be a bit easyer to manage then a forum?
Allows some for some good prioritizing and more.
Using a forum for bug tracking sounds like suicide D:
(Please use a tracker, maby some othe devs that are used to working with them could help filter the duplicated reports?)
Oh and not to mention, its a perfect way for making a TODO list, but just adding things to be done has high priority stories (bugs, whatever the software calls it and supports)
I will switch over to a bugdb when the forum becomes unmanageable, and then maybe I’ll crowd-source converting all the bug threads into db entries. :) For right now, I can’t bear the thought of installing and/or managing (or even logging on to) yet another server technology.
How verbose are the logs? Verbose enough to pose a potential cheating problem?
There are so many ways to cheat in SpyParty right now, it’s not even funny. I’m just hoping I can ignore them because the first few thousand people will be cool about it before having to deal, because it’s a giant pain in the ass and just takes time away from making the game cooler. Maybe I’m being naive, but that’s my “plan” until something blows up.
Was there something in particular you were thinking about? They’re verbose, but mostly about game state. No passwords or anything like that, but there are IP addresses in them.
I was thinking along the lines of “playing animation x”, where a sniper could figure out who the spy is based purely on the text and timing of log messages. You’re probably safe in assuming that early beta players won’t be inclined to cheat, but it’s definitely something to consider closer to shipping.
Yeah, that’s not in there (would be too much data), but that information comes across the wire, so if they’re snooping memory they can find it. I have a couple plans for this, but yeah, hopefully I won’t have to implement them in the near term.
Hey Chris,
I’m a Software Engineer on a large commercial C++ program, and we also struggle with how to deal with crash reports and bug submissions that are difficult to reproduce.
I’ve been driving a project to have internal clients use our product on a virtual machine running under VMWare Workstation to record the screen and internal state of the program. VMWare provides a Visual Studio plugin that lets you hook into the recording as though it were a live process and do (most) of the debugging you could normally do, like step through execution and examine memory locations. You can also do some exotic things under the replay that you can’t do in a live process, like notice a bad value in a particular location and play the recording backwards to discover who wrote it there.
The visual studio plugin is kinda buggy, and VMWare actually cut the feature from their latest release of Workstation, but it’s been golden for us because it means that any time someone submits us a bug report with a replay attached we’re 100% able to reproduce it and have full debugger level detail into exactly what was going on at any time during the recording. Depending on how much time you spend chasing bugs that you can’t reproduce it could be worth all the extra hassle of dealing with the recording and replay environments.
I’d be happy to talk to you about it in more detail if you’re interested.
That does sound pretty useful, but it also sounds like it would be a fair bit of extra setup. And couldn’t it introduce artifacts since VMs have virtualized graphics drivers?
It does have a lot of extra setup, and there’s a good chance that it’s not worth it for SpyParty.
In our situation we’re running boring old business software that doesn’t need high-end graphics drivers, and it’s huge enough program that we probably benefit unusually much from tools that help you track down unexpected side effects from distant unrelated chunks of code.
But like Chris said, “It can be the difference between 10 minutes to fix and 10 days to fix”, and for some problems I’ve gone chasing after it’s been hugely useful once you pay the setup hassle costs.
Thanks for the pointer. I’m going to hope I don’t need that big of a gun. I expect more problems from network issues than any bugs that happen on a single machine, so I’m going to do replays and whatnot (which I’ll use for spectation as well), and hopefully that will help a bunch.
In my (albeit limited) experience writing network code… the debugger becomes next to useless because half the problems will disappear the moment you start stepping through code. So in that context, it definitely doesn’t sound as useful as in-game replays.
so id say about 85-90 percent of the time i shoot someone i suspect as the sniper my game just stops responding and i am forced to close and restart. wouldnt be bad if it were once in awhile but its literally almost every single match and at many times when i am shot i get the same behavior and from time to time performing tasks as a spy causes it but its mainly when the sniper shoots me or i shoot someone. Like i mainly hear the shot noises just before it does this and of coourse the screaming but thats it, itll literally stay there and freeze basically and im once again forced to close the game and restart it.
That should definitely not be happening! Can you go to the Bugs forum and report the bug? There’s a HOWTO report bugs post at the top which’ll give you some tips and tell you where to find your log files for posting. The forum is linked from your beta homepage where you downloaded the game (there’s a link on spyparty.com too).