Mistakes in testing, IMO
This article is the view of testing an MMO with the "other side of the barricades", from the point of view of a gamer. I want to talk about how to do an alpha test, CBT and MBT more efficient. And why beta testers stop writing bug reports and move "in opposition" to the project.
In my life I participated in CBT and OBT in about 30 games. And the most important conclusion I made for myself – the game developers did not know how to dispose of those human resources in the form of several thousand eager to preklicati the full functionality of the gamers that they fall on his head. And that is why it was written this material. The gamer – developer sitting on the other side of the screen. From the tester , who do not understand how to organize the process of testing the players ' efforts.
The article is divided into four sections and written in a conversational style. I in order to dismantle the mistakes of developers in alpha test, CBT/OBT and the set of testers after the release. Those who want specifics without much foreplay and reflection, immediately flipping to the last paragraph of each section and conclusions.
In recent years, it has become a sign of good manners to invite players for the alpha test: World of Tanks (2009), WarThunder (2011), Lost Sector (2013), Armored Warfare (2014), etc. I would argue that this is a bad idea and at be held by professional QA engineers, not gamers.
Why? Because the main consequence of the presence of players on at – the loss of reputation for your company. It is obvious that this thesis may come as a surprise. Explanation below:
1. Players come before the others in the project in order to test it for bugs and to bring you all the bugs on a silver platter, and to get your portion of entertainment and fun first. Undoubtedly, if they find a critical bug that prevents them to play, they are about him otraportuyut. But in General, gamers want to play (and play!), and another glitch next test build will actually only cause irritation, not understanding. And since the negative people remember much more easily than the positive, then to release this part of the players will not remember the new features that you introduced, and all your mistakes. And will be the first army of critics your game network.
2. In practice this looks like this (for example, the reaction of the players in the next patch):
newbie: Yay, new patch! New features!!! It's great!!! I'll go likno developers on the forum.
beta tester: Oh, yeah... the New patch – new bugs. While I will not play. Wait until roll back, will correct and bring to mind. My nerves dearer to me. And while going to go play the competition.
alpha-tester: Well, if you re not fashioned to the whole world. Remember how the server went down when 0.4.5 rolled? Gonna go write in my LJ that developed screwed up again and did it all wrong, though we give them more on the PTA said that we need prevalencethe this map.
Of course, this is a hypothetical conversation, but I think the point is clear. I could bring a real dialogue players who came at different times in WoT, but it's against the rules of this resource. In particular, those items that relate to abuse and mats. Because I can't remember offhand any PTA person who would censor comments about developers.
Yes, the alpha testers and PTA/MBT-schnick much less than other categories of players. But they often occupy leadership positions in all clans/guilds/corporations (underline). At least at first. Respectively, as the most experienced players to form the opinion about the project in General within his clan.
Of course, if your project is “shot”, the first clan ”will be diluted” and General information noise the voices of the eternally dissatisfied the first testers will no longer be heard. But as they say, osadochek remains...
a Small clarification-summary: no, I'm not saying that the players-testers – evil. I'm saying that the players who came in at/CBT/OBT and QA engineers have a completely different purpose and therefore different reaction to the problems in the game. So the idea to invite players on at, when the game is a bundle of bugs – this is a very bad idea.
In theory (in my understanding), the main goal of CBT and OBT is testing the efforts of gamers dynamically updated functionality of the game for errors BEFORE release. And here we get the second goal – testing the reaction of CA to the concept of the game and the various features that introduce (or plan to introduce) developers. And as a side effect – minimization of costs for internal testing by employees of the developer.
In practice every CBT and OBT is converted primarily in the generator rather dubious infosum (as an example – Google blozhikah shkoloty, which, first heard about existence of Armored Warfare, giving the game to heaven and now pours mud, although the game even on release). From the point of view of efficiency test results is also very controversial. To be precise in the wording – the developers are doing everything to discourage the players-testers desire to do anything. And this is a catastrophic inability to work with an audience of several thousand people who do can give feedback, forced me to paint below the typical mistakes in the organization of the CBT/OBT and methods of their treatment (from the point of view of the player).
So there you go:
In General, in my personal opinion, most CBT/OBT held under the motto "test, don't know What". So let's say we have a hyperactive group of players-testers who day and night sitting in the game, find bugs and scribbling messages on the forum (for example, military-historical games is the "riveters" who defend the historic performance characteristics of your favorite techniques).
And here the main problem lies in the fact that neither the developers, nor the community managers do not know how to direct the energy of this target group in a constructive direction for the benefit of the project. The word “all”. Seriously. Where the recommended goal for testing? Where a rough plan of what the developers would like to see it tested by players? No. None of the above in practice no. So everyone is testing himself, in his own way – one doing a video on YouTube with penetration models, and the second sawing on the forum notices with summarized data on the penetration of cannons of various calibers from different sources.
The special beauty of the situation is that often the vacancies of the company hangs a "historical consultant", but in the game forum of professional historians with historia.ru or tsushima.ru pointing to inconsistencies, the developers send almost Mat, citing the fact that "in our 100% realistic the game historicism sacrificed in the balance." About wikis that make students on the basis of googletranslator articles do not say anything.
Separately frozen, when the developers patch the laptop do not write on test changes – from the series "with the goal of optimizing the balance was changed characteristics of some machines." Well, not knowing what changed, you can test innovations? Or to conduct regression testing? We red-hot pincers every time to get information from you, gentlemen developers? I would like to stress – we are talking about CBT/OBT when all performance characteristics in the process of balancing (and we – testers are well aware), and not post-release, when the AP nerf or a certain technology may cause a negative reaction.
Now let's imagine that despite all of the above, a certain John DOE (which does not even suspect that he really deals with black-box testing) intuitively guessed that it is necessary to test and wrote my first ever bug report on the forum!
And the answer is:
— never got any response;
— topic closed and moved to the archive without explanation;
— got the answer “known bug we are working on it” without even the elementary thanks;
What will be the reaction of Vareniki? – Right! Vasenka more in the life of a single bug report, not write.
Gamers (even testers) is, first and foremost, consumers. Never forget it. They are not diplomats and are not the spokespersons of corporations. So it is quite a normal working situation, when along with a bug-report to developers poured a bucket of cold water. Sometimes even with the mud (deservedly or not is another question). What am I? Yes, the fact that about a third of my friends on the PTA/MBT ceased to write bug reports, received a sharp rebuke.
The gaming forum under CBT/OBT I usually see two options for responding to legitimate criticism of players, backed up by a bug-report:
— punitive until the ban (came with the moderator.);
— guilty (come developer and started telling off the players that they are ungrateful bastards – it then does not sleep at night, wife and I have sex a week was not, and the players nose in, poke bugs, does not rest).
In the end, more some of the testers becomes hostile towards your project and throws its testing. This is the best. At worst, running to complain to the gaming portals and go to the competitors.
And then came the long-awaited OBT. Sitting in Tim-SPIK our hero John DOE, whose account 7 bug reports, and Dima Kozlov, who signed up the last day of the CBT (the real story) and didn't do anything. And the reward both for their participation in the PTA is the same – a vehicle, for example. Do you think Vasenka after this universal injustice, at least one bug report to write? Right thinking – no. And here is another group of skilled testers that have sunk into oblivion.
Overall, I think that the conclusions are so obvious. But when you explain it on the examples in the PM the next theme, then you run into a blank wall of misunderstanding (although outside of game maid is often quite adequate people, if you believe FB and VK).
1. If you want the players something really tested on the beta test – think of them as our QA staff: give them the test plan and create a forum infrastructure to digest the results of these tests. Explain What, When, How and Why it is worth to test. Ideally, with a newsletter to mail to everyone. In short, raise the standards of beta testing! You students for extras or testers needed!?
2. Provide your beta testers back and most importantly – timely (!) connection. Adequate feedback on bug reports – the key siciliaanse and long-term cooperation beta testers and GD.
3. Learn how to respond adequately to the criticism of the players and recognize their mistakes. The offended player is not only the place banned on your forum and 500 terabytes of your games distributed on the torrents (true-story). And not a single licensed copy of your game in a gift box on the shelf at his teammates.
4. Remember once and for all – not all participants in the CBT/OBT to award the same. Reward should be proportional to the contribution of the player in the draft! Otherwise, those who are really tested, you completely discourage the desire to do anything to end the life of your project.
After the release of nothing has changed: the players want new content, and one penny for the two mistresses and the little house on the Islands. So patches and Addons come out like hot cakes from under the Baker's hands, and the bugs continue to multiply like rabbits. The only difference is to replace the beta-testers come super-testers. And here begins the next act of the farce of our drama of the Absurd.
If someone does not understand – the criteria set in the super-testers. Remembering WoT, we were very amused when the application of our officers on tactics that in real life I work in IT and have a winrate 53-55%, rejected, and the son of our friend with articles 43% took. And all because unlike us he could play 12 hours a day. By the way, in those days, when I have already deleted my account in the tanks a year later, he found his first bug.
And not only tanks. For some reason on super-test everywhere prefer to recruit youth willing to play around the clock. I have nothing against CA – on someone do we have to fill experience and frags. But to offer them to test? People, are you serious? From what the tester would be more useful – from a teen that 16 hours is going to play in the style of "pysch-pysch, ololo – I am driver UFO!" or from hours of playing the adult who comes for the test to work.
On this I have almost everything and, as the classics said, "Shaherezada stop, going from permitted speech to ill thinking."
Actually I only have one clever idea – the gamification of game testing. If you want to see the players testing your game – make sure that they received pleasure from it! You're ranking the top players? – Add website top testers. Give out the medals and achievements for every sneeze? – Make some unique achievements for everyone who finds bugs and helps you to make the project better. You have on weekends x2 experience for nerds? – Enter higher rates for testers. Everyone will benefit from it.
P. S. I'm not saying that I "discovered the Truth". I just chewed quite obvious, clear to any who though time was engaged in organizational work. But as game developers from game to game, from release to release diligently continue to ride on the rake called “how to get gamers to stop writing out bug reports and to sabotage the testing of the game”, someone had to write this post.
Article based on information from habrahabr.ru
Foreword
In my life I participated in CBT and OBT in about 30 games. And the most important conclusion I made for myself – the game developers did not know how to dispose of those human resources in the form of several thousand eager to preklicati the full functionality of the gamers that they fall on his head. And that is why it was written this material. The gamer – developer sitting on the other side of the screen. From the tester , who do not understand how to organize the process of testing the players ' efforts.
The article is divided into four sections and written in a conversational style. I in order to dismantle the mistakes of developers in alpha test, CBT/OBT and the set of testers after the release. Those who want specifics without much foreplay and reflection, immediately flipping to the last paragraph of each section and conclusions.
I. alpha testing
In recent years, it has become a sign of good manners to invite players for the alpha test: World of Tanks (2009), WarThunder (2011), Lost Sector (2013), Armored Warfare (2014), etc. I would argue that this is a bad idea and at be held by professional QA engineers, not gamers.
Why? Because the main consequence of the presence of players on at – the loss of reputation for your company. It is obvious that this thesis may come as a surprise. Explanation below:
1. Players come before the others in the project in order to test it for bugs and to bring you all the bugs on a silver platter, and to get your portion of entertainment and fun first. Undoubtedly, if they find a critical bug that prevents them to play, they are about him otraportuyut. But in General, gamers want to play (and play!), and another glitch next test build will actually only cause irritation, not understanding. And since the negative people remember much more easily than the positive, then to release this part of the players will not remember the new features that you introduced, and all your mistakes. And will be the first army of critics your game network.
2. In practice this looks like this (for example, the reaction of the players in the next patch):
newbie: Yay, new patch! New features!!! It's great!!! I'll go likno developers on the forum.
beta tester: Oh, yeah... the New patch – new bugs. While I will not play. Wait until roll back, will correct and bring to mind. My nerves dearer to me. And while going to go play the competition.
alpha-tester: Well, if you re not fashioned to the whole world. Remember how the server went down when 0.4.5 rolled? Gonna go write in my LJ that developed screwed up again and did it all wrong, though we give them more on the PTA said that we need prevalencethe this map.
Of course, this is a hypothetical conversation, but I think the point is clear. I could bring a real dialogue players who came at different times in WoT, but it's against the rules of this resource. In particular, those items that relate to abuse and mats. Because I can't remember offhand any PTA person who would censor comments about developers.
Yes, the alpha testers and PTA/MBT-schnick much less than other categories of players. But they often occupy leadership positions in all clans/guilds/corporations (underline). At least at first. Respectively, as the most experienced players to form the opinion about the project in General within his clan.
Of course, if your project is “shot”, the first clan ”will be diluted” and General information noise the voices of the eternally dissatisfied the first testers will no longer be heard. But as they say, osadochek remains...
a Small clarification-summary: no, I'm not saying that the players-testers – evil. I'm saying that the players who came in at/CBT/OBT and QA engineers have a completely different purpose and therefore different reaction to the problems in the game. So the idea to invite players on at, when the game is a bundle of bugs – this is a very bad idea.
II. PTA&MBT
In theory (in my understanding), the main goal of CBT and OBT is testing the efforts of gamers dynamically updated functionality of the game for errors BEFORE release. And here we get the second goal – testing the reaction of CA to the concept of the game and the various features that introduce (or plan to introduce) developers. And as a side effect – minimization of costs for internal testing by employees of the developer.
In practice every CBT and OBT is converted primarily in the generator rather dubious infosum (as an example – Google blozhikah shkoloty, which, first heard about existence of Armored Warfare, giving the game to heaven and now pours mud, although the game even on release). From the point of view of efficiency test results is also very controversial. To be precise in the wording – the developers are doing everything to discourage the players-testers desire to do anything. And this is a catastrophic inability to work with an audience of several thousand people who do can give feedback, forced me to paint below the typical mistakes in the organization of the CBT/OBT and methods of their treatment (from the point of view of the player).
So there you go:
1. The lack of information on tests
In General, in my personal opinion, most CBT/OBT held under the motto "test, don't know What". So let's say we have a hyperactive group of players-testers who day and night sitting in the game, find bugs and scribbling messages on the forum (for example, military-historical games is the "riveters" who defend the historic performance characteristics of your favorite techniques).
And here the main problem lies in the fact that neither the developers, nor the community managers do not know how to direct the energy of this target group in a constructive direction for the benefit of the project. The word “all”. Seriously. Where the recommended goal for testing? Where a rough plan of what the developers would like to see it tested by players? No. None of the above in practice no. So everyone is testing himself, in his own way – one doing a video on YouTube with penetration models, and the second sawing on the forum notices with summarized data on the penetration of cannons of various calibers from different sources.
The special beauty of the situation is that often the vacancies of the company hangs a "historical consultant", but in the game forum of professional historians with historia.ru or tsushima.ru pointing to inconsistencies, the developers send almost Mat, citing the fact that "in our 100% realistic the game historicism sacrificed in the balance." About wikis that make students on the basis of googletranslator articles do not say anything.
Separately frozen, when the developers patch the laptop do not write on test changes – from the series "with the goal of optimizing the balance was changed characteristics of some machines." Well, not knowing what changed, you can test innovations? Or to conduct regression testing? We red-hot pincers every time to get information from you, gentlemen developers? I would like to stress – we are talking about CBT/OBT when all performance characteristics in the process of balancing (and we – testers are well aware), and not post-release, when the AP nerf or a certain technology may cause a negative reaction.
2. The lack of adequate feedback
Now let's imagine that despite all of the above, a certain John DOE (which does not even suspect that he really deals with black-box testing) intuitively guessed that it is necessary to test and wrote my first ever bug report on the forum!
And the answer is:
— never got any response;
— topic closed and moved to the archive without explanation;
— got the answer “known bug we are working on it” without even the elementary thanks;
What will be the reaction of Vareniki? – Right! Vasenka more in the life of a single bug report, not write.
3. Negative reactions
Gamers (even testers) is, first and foremost, consumers. Never forget it. They are not diplomats and are not the spokespersons of corporations. So it is quite a normal working situation, when along with a bug-report to developers poured a bucket of cold water. Sometimes even with the mud (deservedly or not is another question). What am I? Yes, the fact that about a third of my friends on the PTA/MBT ceased to write bug reports, received a sharp rebuke.
The gaming forum under CBT/OBT I usually see two options for responding to legitimate criticism of players, backed up by a bug-report:
— punitive until the ban (came with the moderator.);
— guilty (come developer and started telling off the players that they are ungrateful bastards – it then does not sleep at night, wife and I have sex a week was not, and the players nose in, poke bugs, does not rest).
In the end, more some of the testers becomes hostile towards your project and throws its testing. This is the best. At worst, running to complain to the gaming portals and go to the competitors.
4. Dubious reward
And then came the long-awaited OBT. Sitting in Tim-SPIK our hero John DOE, whose account 7 bug reports, and Dima Kozlov, who signed up the last day of the CBT (the real story) and didn't do anything. And the reward both for their participation in the PTA is the same – a vehicle, for example. Do you think Vasenka after this universal injustice, at least one bug report to write? Right thinking – no. And here is another group of skilled testers that have sunk into oblivion.
Overall, I think that the conclusions are so obvious. But when you explain it on the examples in the PM the next theme, then you run into a blank wall of misunderstanding (although outside of game maid is often quite adequate people, if you believe FB and VK).
So brief
1. If you want the players something really tested on the beta test – think of them as our QA staff: give them the test plan and create a forum infrastructure to digest the results of these tests. Explain What, When, How and Why it is worth to test. Ideally, with a newsletter to mail to everyone. In short, raise the standards of beta testing! You students for extras or testers needed!?
2. Provide your beta testers back and most importantly – timely (!) connection. Adequate feedback on bug reports – the key siciliaanse and long-term cooperation beta testers and GD.
3. Learn how to respond adequately to the criticism of the players and recognize their mistakes. The offended player is not only the place banned on your forum and 500 terabytes of your games distributed on the torrents (true-story). And not a single licensed copy of your game in a gift box on the shelf at his teammates.
4. Remember once and for all – not all participants in the CBT/OBT to award the same. Reward should be proportional to the contribution of the player in the draft! Otherwise, those who are really tested, you completely discourage the desire to do anything to end the life of your project.
III. Post-release
After the release of nothing has changed: the players want new content, and one penny for the two mistresses and the little house on the Islands. So patches and Addons come out like hot cakes from under the Baker's hands, and the bugs continue to multiply like rabbits. The only difference is to replace the beta-testers come super-testers. And here begins the next act of the farce of our drama of the Absurd.
If someone does not understand – the criteria set in the super-testers. Remembering WoT, we were very amused when the application of our officers on tactics that in real life I work in IT and have a winrate 53-55%, rejected, and the son of our friend with articles 43% took. And all because unlike us he could play 12 hours a day. By the way, in those days, when I have already deleted my account in the tanks a year later, he found his first bug.
And not only tanks. For some reason on super-test everywhere prefer to recruit youth willing to play around the clock. I have nothing against CA – on someone do we have to fill experience and frags. But to offer them to test? People, are you serious? From what the tester would be more useful – from a teen that 16 hours is going to play in the style of "pysch-pysch, ololo – I am driver UFO!" or from hours of playing the adult who comes for the test to work.
On this I have almost everything and, as the classics said, "Shaherezada stop, going from permitted speech to ill thinking."
IV. Clever ideas
Actually I only have one clever idea – the gamification of game testing. If you want to see the players testing your game – make sure that they received pleasure from it! You're ranking the top players? – Add website top testers. Give out the medals and achievements for every sneeze? – Make some unique achievements for everyone who finds bugs and helps you to make the project better. You have on weekends x2 experience for nerds? – Enter higher rates for testers. Everyone will benefit from it.
P. S. I'm not saying that I "discovered the Truth". I just chewed quite obvious, clear to any who though time was engaged in organizational work. But as game developers from game to game, from release to release diligently continue to ride on the rake called “how to get gamers to stop writing out bug reports and to sabotage the testing of the game”, someone had to write this post.
Комментарии
Отправить комментарий