Feedback Column at MMORPG.com
I started to write this somewhat wonky piece for this blog, and realized that I had a column due at MMORPG. That happens to me a lot, lately. I got the kernel of the piece – that bit of research – from my friend and former minion Jeremy, who also wrote a blog post about it.
The article ended up being a little different from my intended writeup, because the audience there is more general than the seven of you, who read, presumably, because you care about community stuff.
I’m working on a couple of projects now where we’re struggling with the best way to handle user feedback and prioritize it correctly. I’m always insanely nervous at this point, because I know that what I lay down as policy now will live for freaking EVER.
And dear god, I wish there was some research to rely on. Right now, my experience tells me that answering posts with links to a knowledge base is the fastest way to train people to use the KB. Great. Duh. The real sticky bit is how do I train people to use a bug reporting tool. Personal response is my most potent weapon, and as with all potent weapons, I’ve become overly reliant on it as a technique.
But I don’t want to personally respond to the bugs I get through the tool, because personal response is simply not sustainable with a game that has any level of success. If I respond now while it’s possible, it will train them to use the tool, and I will totally fail to manage their post-launch expectations. What is a sufficient reward for using an automated tool? It’s not an automated response, that’s for damn sure. I’d like to see a study comparing usage rates of bug tools with different kinds of “rewards,” so I can set a policy and move on with my life.
Anyway. This has been an update from the Bureau of “This Blog Isn’t Really Abandoned.”
Wouldn’t a good form of feedback be somewhere, a list of bugs that have been reported, how often they were reported, their status, etc.? Sort of an open version of the internal QA database. But not the actual QA database, of course.
As only a player, I’d like to know that at least on some level bugs are being tallied and the public portion shames the developers to address them at some point. The trick being that the developers (or, I guess in this case, the community managers) get to pick and choose what goes up on the Board Of Shame, so they’re likely to be something realistic that might actually get fixed someday. You know, if the community team is being all savvy about it and all.
It could be a whole kabuki show or something that’s actually taken seriously, the actual level of “realism” would be up to the individual implementors and maintainers.
I haven’t read the article yet. Lemme check that and I’ll come back and report just how stupid I am.
Yea, sorry, dumb suggestion. Just on the level of the amount of work; to get approval, to implement, to upkeep.
And it doesn’t answer your question, that of research. It’s just an idea with nothing to back it up.
Maybe file it with the rest of the Pie In The Sky ideas coming from players. :/
Submitting bug reports is a tough one because most users are past the bug by the time they can submit it. It then becomes other players’ problem.
I would love a bug reporting system that gave tangible rewards, but there are a lot of pitfalls. Bug find of the week or a monthly drawing might be doable?
Can the hybrid chick speak up with some references to old PR/Community relations/Markety tactics?
actually it was so long winded i just emailed it to you.
I think the problem is trying to both receive and analyze data at the same time and it can’t be done that way.
Personally, I think market research tools — hate em if you will — are the best — they do take the ‘personal touch’ out of it however they can offer the best feedback. If you get an area where tons and tons of players are either griping about — such as being underpowered — or playing in one area vs another, you can then do further investigative research as to WHY. Rather than trying to figure out the ‘Why” at the same time.. use the data to look for patterns then analyze the date, taking out, of course, outliers…
Data may be boring, but it doesn’t lie in and of itself. It has to have help! ha.
From someone who has submitted bug reports in many games (always using whatever tools the devs give me if possible), I can say that what I would most like back is merely an acknowledgment that it is a known bug, and where it is in the priority list (or that it’s a new bug and will be looked at). The response doesn’t have to be personal, it just has to relate to whatever I submitted. Canned responses never address the specific item I’m taking the time to report, but personal responses are overkill. Sample:
‘Thank you for your report. This is a known bug, and is scheduled to be addressed in patch xx.xx.xxxx’
Easy, simple, and gives me what I want. If it’s a new bug, maybe do a followup email once the bug is confirmed and it’s in the pipe.
Basically what Max said, but, being a QA person, I know that the level of detail he asks for is pretty much impossible.
Some sort of feedback on the bug status would be nice, though I know from experience that a lot of user-submitted ‘bugs’ go right into the trash bin due to the fact that the report either doesn’t have enough info, is just a rant about some aspect of the product, is a ridiculous feature request, or simply doesnt make any sense at all.
I think the best that could be hoped for from a user experience is a simple one-time status update when the bug report has been examined. Something as simple as Acknowledged as a new bug, Duplicate of an already existing bug, Can’t Reproduce, or By Design. This could even be an automated part of the bug reporting tool depending on how things are set up.
On Dungeon Runners we maintained a Pligg-based web site (http://www.pligg.com/, it’s a “roll-your-own” clone of digg) where players could make suggestions, report bugs, and then discuss them (as well as rate them in importance).
The idea was to make it easy for everyone, players and developers, to see what was important, what players cared about, that kind of thing. It’s sort of an automated way of digesting player feedback and sharing it back so they can see what we’re doing with it (one big part of that was modifying Pligg so that we could mark a requested item as DONE, to show that we were actually listening and acting on reports).
Still, it was a lot of work; a whole new forum to moderate, with different moderation rules (one of the big ones, not so important in normal forums, is to avoid duplicates: PLEASE PLEASE PLEASE comment or vote on the existing entry, so that we have good data, if there is one). I’d say it was valuable, and probably worth the extra work, too.
I read because I care about community stuff?
I just heard there was punch and pie.
“I would love a bug reporting system that gave tangible rewards, but there are a lot of pitfalls. Bug find of the week or a monthly drawing might be doable?”
Never understood why more companies don’t do this for beta testers. Giving out free months post launch to randomly selected beta testers who actually report bug seems like it would cost next to nothing and actually encourage user feedback.
Sadly more and more companies seem to be using beta as a marketing tool rather than Q&A. No really! Short and limited “open betas” with very little user feedback are great for games. Worked out great for AoC and WAR.
I do find it silly that these developers will give you a new title for clicking on yourself 100 times or jumping off a cliff but there is nothing shiney for reporting 5-10-25-100 unique bugs to Q&A.
I’ve been doing bug reports since I first started playing MMOs almost 10years ago. I do them in Live games just the same as betas. There is no need to incentivize me, and indeed I find it insulting when games offer “rewards” for bug reporting.
It’s dead simple. I play the game. If there are bugs, it’s in my interest that they are fixed. What more incentive do I need?
And maybe I am naive, but I have always believed that all bug reports are read and acted upon when appropriate. If it takes months to fix them, it’s because resources are limited.
Not from your industry at all, but in telecommunications software development space. We’ve had extreme success using the KB implementation customized for bug reports. Upon submission they receive an automated response about the bug reported. If a similar bug has already been reported, the auto reply tells them “…aware of the issue and actively working on a fix”, “aware of issue and conducting further analysis”, “aware and it’s being prioritized”, “aware and working as intended…blah and so on.
First some back ground, I have absolutely nothing to do with the game community other than being a player, I am however responsible for management training in the company I work for. One thing that I have always found with dealing with any type of reporting or surveys is that the best policy is RFG, rewards, feedback, & games.
I learned years ago when I contributed some code to an open source forum project the no one remembers today, one of the things we tried out was an item shop added to the forums, items were “purchased” with SP or shop points. These could be earned in various ways, simply posting earned you 1 point, if your post exceeded X number of characters you received an additional amount of points. These points could be added to or taken away from by the forum moderators if the system was being abused. Other users could also donate or transfer points or vote a post to receive additional points.
I digress. . .
The point was that by rewarding people who submit bugs or ideas you give them a feeling of accomplishment which also makes them feel like you (the company) recognize and appreciate their contributions.
You could set up a system where as players submit bugs they are awarded some kind of points once the bug is confirmed by the people who do that sort of thing, I am sure you already have people whose job it is to sort through the reports and lump similar bugs (say a typo in a quest gets 100 bug reports). The first person to submit the bug gets 5 points and everyone else gets 2 or whatever formula you find works best. You could create a micro-store where people can purchase these points for Realm Money or they can participate in testing events and submit bug reports to earn points, make the items cosmetic in nature, new hair, special dyes, you get the idea.
I ramble too much.
C
If you reward with points, you have to monitor who’s spamming reports to reach the next level.
If you do a random drawing, too many will believe it’s rigged for your friends.
If you reply with acknowledgement of the issue and it’s current status, you will give a feeling of them being connected to the outcome of the games overall playability…and over time, you can automate it fairly well. When you have outstanding bug #109 being reported for the 10,000th time, then you also gather data as to which bugs are being experienced most often by your player base.
Let your CRM system handle most of it…All you need is one weanie to check the categorizations every so often to be sure you don’t have the same bug being reported 100 times as a Graphic bug, 200 times as a World bug and 300 times as a Quest bug.
I think perhaps I was not clear, it happens sometimes when I try to pass my thoughts from an idea into a written description.
The system would be tied to normal bug reporting tools, such that bogus reports are quickly scrapped and thus no points given. You couldn’t spam useless reports simply to rack up points, there would be a delay in receiving points as the bug would have to be verified first, recreated and then added to the bug database. Only when the bug is confirmed and added would points be awarded, making it much harder to game the system.
I really liked this article, and had to piggyback off it to approach it from another perspective on my blog.
I think that a lot of the expectations of being heard from a player perspective are inevitably (and sometimes mistakenly) married to a notion that a developer can, should, and must take your feedback seriously. Many players can’t cope with the idea that they might be submitting feedback that will ultimately not be used or even be disproved with other forms of feedback, and proceed to lash out at the developers when their version of the results isn’t shown.
I agree wholeheartedly with combating these bad feelings with transparency. When developers don’t make their thought processes at least somewhat known, the head-scratching and nerd-raging ensues over things that players don’t understand.
Hook into the tools a way for a developer to “approve” a bug. Once approved, the player gets something. Maybe it’s just a funny in-game mail from a lore character. Maybe it’s an email with a funny picture (LOLore!) Maybe it’s an XP bonus potion, or a flask, or maybe you get a cool title at 25 or something.
Or maybe it’s automatic based on the resolution of the bug; fixed or dupe is better than not repro or by design.
But it can’t just be automatic on submission, because otherwise you get flooded with crap bugs.
I am certain that there should be some research in other customer service areas (e.g. telco) that would be useful to help define est practices or find useful software to help – assuming there is a budget for that.
A few notes what I think can help the process of getting user bug reports, perhaps already obvious, but there always seem to be MMOs that fail on one or more:
* Make it easy to find the option to submit a bug report in the game. If it cannot be submitted from within the game people will likely not bother in many cases. And not everyone would instantly know or assume that /bug or /petition would be the way to do it (assuming that is the case).
* Allow some space for the player to explain the problem. Some games have so short text space that it hardly feels meaningful to try to write what the problem is. This is also relevant for the next point.
* Tell the player what information is collected automatically, if any. E.g. character name and other character data (snapshot perhaps), zone, location etc. If nothing is collected automatically, tell the player that. If the player knows what is collected they can make better use of the available space to explain the problem.
* Provide some examples of good/useful bug reports. Examples can be helpful in conditioning the players on how to submit reports.
I think feedback back to the player can be a two-step process, hopefully to a large extent possible to automate.
* Acknowledge receipt of the bug report
* Acknowledge (pending) review of the bug report
I do not think the latter need to say whether a fix will be implemented or not or exactly when it will be addressed.
But a response that indicates that someone has read it and either received or collected enough data for dev team to review it, or that further analysis and data collection is in progress. Even if the report may have been filtered out automatically as a common issue, the response should still have the flavour of have been read individually.
Ideally there should be a time given to the player after the initial recipt acknowledgement when to expect some status update, even if that is only “we are collecting more data on the issue before proceeding” or something to that effect.
I think explicit reward systems can be a bit tricky. Too good rewards and it people will try to abuse it for their benefit; useless/meaningless rewards may potentially project that it isn’t really important.
Some feedback loop back to the player (e.g. along the lines above) may be enough, since in essence it projects that the player’s action has made some kind of impact on developers and perhaps the game.
“* Make it easy to find the option to submit a bug report in the game. If it cannot be submitted from within the game people will likely not bother in many cases. And not everyone would instantly know or assume that /bug or /petition would be the way to do it (assuming that is the case).”
This is the big one, IMO. I got to try a recent Beta (name withheld to protect the guilty). There were, shall we say, major problems with trying to play the game. The only place to post them was the forums, and they looked like the Wild West to me. Sure, it felt like a lot of my issues would have been “me too” posts, but that was the only place to report them.
Problem is, there is nothing for actually diagnosing what was going on. Could it have been internet issues? Hardware? OS? Who knows. But either an in-game or on-crash reporting method could be a big help. Mozilla and Microsoft have crash reporting code that will send specific technical data to the company when something goes horribly wrong. The only thing a player can do is say “the game crashed”, and that’s only if they bother to write up a post.
I think the biggest thing a company can do to encourage testers is to make it easier for technical data to be sent with bug reports.
As others have said, a reward system giving a player free days in game or some game cash to spend in the store would be nice for reporting bugs.
It would be nice to get feedback on the bug, perhaps if only in the fashion that more open bug tracking does. If I report a bug, you look at it and determine it is the same as another reported bug, I’d love to see that you associated my bug report #68234 with existing bug report #8543, and for all of us to see when our bug is fixed in the patch (not fixed internally, but in a released patch).
It doesn’t need to be an entirely open system either. Associate the reports with the account and only that user can see the report (and reports deemed as related). This would also allow many users experiencing the same problems to communicate with each other and even assist in bug testing. If you let your users help you, you might be surprised at how many of them are willing to help you.
I’d review Second Life’s JIRA implementation — it’s a fairly transparent bug reporting and tracking system. I can’t speak to whether or not it’s worked for them, but it’s certainly an approach that most MMOs don’t try. See also Jon’s initial comment on this post.
An ideal bug reporting system should have an internal moderation system that rewards prolific good bug report submitters. Knowing that a report comes from a reliable source is an excellent way to speed the QA process.
I love the article at MMORPG.COM…
And I miss the golden days when you were a freaking goddess of the DAOC community and MMORPG’s were social and fun even when you were grinding. Until TOA of course
I think Sente @17 pretty much nails it. Ease of use is a huge factor in bug report tool uptake. In a game, when you find a bug, you’re often frustrated enough as it is. Making it difficult to report the bug can often be the tipping point that prevents a player from contributing at all.
One thing to keep in mind is that players who report bugs tend to be a specific type of player, and in a minority at that-most players will work around the bug and assume someone else is on it. You’ll never get the majority of players to regularly report bugs no matter what you do. So the key is to figure out how to cater to this minority.
Feedback is obviously a critical part of this process for these players. Rwards have been mentioned a lot, but I think that the nature of said rewards has been restricted too tightly to in-game rewards. What most of these bug reporting players want is the feeling of participating with the game development team, of actually contributing to the game in a concrete way. With this in mind, don’t dismiss the impact that public recognition will have on these players: “Hey, we’d really like to thank Poppydell for her hard work in identifying the Hobsnobber Exploit bug. She gave our team exactly what they needed to pin down the problem, and the fix will be going live in next week’s patch.”
It’s not all that’s needed by a long shot, but that feeling of participation and contribution is what many people play multiplayer games for in the first place.
This suggestion probably goes beyond the scope of the article, but when a player encounters a bug, and if that bug prevents the attainment of a reward prize, then the player should be rewarded with said prize if the player documents the bug. This means that in-game Customer Service must be empowered to evaluate the bug situation and authorize reward disbursal.
Let me give an example: Suppose three players band together to complete a series of quests in an instance dungeon (not really applicable to a football game, but a familiar reference to many); they’ve been challenged and fought to the best of their abilities through a lengthy series of puzzles, traps, beasts, et cetera to reach the final encounter; this encounter is, by design, extremely challenging for three people (the limit to the number of players allowed in the instance), and upon killing the final boss the encounter bugs and the loot is unobtainable. The rational option at this point, when the players call for a game master (GM), is for the GM to read the logs and determine that the players encountered a bug that prevents them from obtaining their reward, and that they had successfully completed the task otherwise. The GM should then encourage the players to submit a bug report with the promise of giving them the reward when they do so (i.e., player submits report -> player gets reward loot).
As a player, the most disheartening advice ever heard from Customer Service is that they’re sorry the players encountered the bug, but there is nothing they can do for them about it. (see what I did there … “I’m sorry, but ..” … which refers to an earlier blog here at Eating Bees). Bugs can be game breakers, and if players are to continue feeling good about the game, they need to be assured that bugs will get fixed quickly (so that they will not encounter them again on a subsequent attempt), and that the players haven’t had their noses shoved in the fact that they’ve just wasted several hours and lost their chance to obtain their goal reward.
In the above example, one bug annoyed three players. Those three players then told ten friends (or more) each about their negative experience in the game. One of two scenarios follow: the negative news about the bug (and subsequent inaction by CS) hits the forums and the drama llama is brought out; or the players quietly cancel and leave the game. It is hard to imagine how the game company could benefit from either of these situations when if, up front, the CS was empowered to make the situation right (without giving away the bank) and the fallout was avoided.
/ramble off
Agree with the issue that ease of reporting is huge. Also providing brief instructions in front of every report about what you want will mean that the quality of responses will improve (not everyone will read it every time, but it helps to remind / reassure people of the process).
City of Heroes / Villains offers the Bug Hunter badge to anyone who reports a critical bug along with steps to reproduce. To get it a lot of internal hoops have to be jumped though since the devs award it, but it is a highly sought-after badge.
Internally, I’d say that linking up your reporting database with your actual dev used database is huge. Using CoH/V as an example again, a developer recently said he hadn’t heard of a bug a lot of people had experienced and that people should not use the internal reporting tool if they wanted something handled quickly. He (and CMs) had to come back and do damage control – apparenly the QA team would work through people’s submissions from the in-game tool and would only escalate those bugs they found met necessary criteria. However, making it look like the devs don’t even see the player-submitted reports was a bad move.
Hmm… the software industry is decades old now. Why does it seem so much harder to implement bug reporting in gaming software than other software, especially with the large numbers of people actively using the software in question AND the ability to develop and use online tools to diagnose what exactly is going on? Most other software doesn’t have the unique ability of MMORPGs to actively monitor what exactly is happening.
Having said that, I like your piece. Simple acknowledgement of a bug report helps a lot, feedback to the reporter and customer base goes even further. An online database that allowed the bug reporter to track what was going on with his/her report would go a long way to satisfying gripes — as well as help software QA keep track of what they were working on and why.
“Oh wow, 9378 other people reported the same thing, great, maybe they’ll prioritize fixing it.”
“Hmm… I’m the only one to report that? With 10,000 people doing that quest daily? Maybe I did something wrong …”
“50 others reported the bug the same way I did but 4500 reported the opposite. That’s odd …”
For what it’s worth, I quit DAOC precisely because it seemed relatively obvious that the company wasn’t going to do anything about my carefully documented complaints (including detailed statistical analysis and logs). The devs seemed to be happy with the fundamental changes they had wrought, people who had little-to-no experience with the original mechanics were ambivalent or deliriously happy and the producers seemed to be ambivalent or deliriously happy that a large number of old-timers were quitting. Mythic never published any information on why they disagreed with the statistical analysis so many players produced or that they cared about anything other than pushing the patch as conceived on schedule regardless of what it did to the game. I tend to think there were probably valid reasons but it wasn’t evident to the player base from the externals — and THAT creates dissatisfaction.
Links to knowledge bases are fine if the knowledge base is updated. Ever.
I’m used to getting a robotic email acknowledgement that a bug report has been received, and that is just fine. They usually contain a disclaimer blah blah blah thanks, we may or may not do anything with this, blah blah blah. If I agree to test I agree to submit bug reports.
If the reporting mechanism is smart enough to already have my coordinates and allows me to fill in server, character information, etc, that is great. Don’t ask for step by step reports of what occurred and is it repeatable and verifiable blah blah blah because while I’m filling in this tome, my character is being attacked and killed and so much for my detailing.
“Personal response is my most potent weapon”
“But I don’t want to personally respond to the bugs I get through the tool, because personal response is simply not sustainable with a game that has any level of success.”
This is a pure guess but i’d have thought that the percentage of players who send in bug reports that are detailed, clear and precise descriptions of the bug would be quite rare in amongst vast piles of opaque craziness?
If so wouldn’t it be possible to identify those few and use the personal approach with them? Once identified and nurtured they could even be set proactive tasks to hunt down vague bugs reported by other players.
I agree with Andre that public recognition is a good reward for this.
Just to add – take a look at how Cryptic has its bug reporting system implemented in ChampO. It’s been good for me in seeing bugs that I come across that are already in the system.
I recall someone suggesting a kissing booth once.
Goe, wondering if you’re a grandmaster diaperchanger yet.
What UnSub said. I really liked the Cryptic bug tool. And it gave me indirect responses as well, since it worked similar to a ticket system. The tickets would get updated so I could see the status if I wanted to. Or not, if I prefered that.
Also, there’s still an MMO Examiner!
To be honest I rarely read articles on MMORPG’s site. I saw it in my RSS reader and read the whole thing. I for one rarely posts his opinion on others forums/blogs I thought your piece was just awesome and hit the nail on the head. I loved the bit about Customer Service and Positive feedback as I work in this field and I too do the same thing with Companies I have good dealings with. A lot of people just do not say good job anymore instead of focusing on the bad. Guess you made a new reader of your blog today
Keep up the great writing.
Take a look at Second Life’s approach. They give users limited access to JIRA, a complete bug database. This allows users to see their bugs and other people’s bugs. Later, as some of these bugs get addressed, you can tell users about the ones which were addressed in a positive manner (i.e., some kind of positive action was taken on them).