Ddrak

Actually I was just using "raiding" as an example. Wanna test quests then get a quester. Test transsexual fae sickening girly content, get Winter. etc.


By "whole world", I meant the new zones. I figured that was taken for granted given the subject matter but I'll try to be more precise in future. The new zones are definitely not properly load tested in public betas simply because there's no hard guarantee that you can keep the load at a certain level while something else happens elsewhere which may be affected by it. That's really my whole point - it's not deterministic testing because you can't repeat the conditions when you think you've fixed something.

Why do you feel games don't need to be QA'd to the same level as an OS or accounting software? I tend to spend far more on games than any other class of software, and games tend to have massive development budgets so I expect them to be properly QA'd. Settling for mediocrity just isn't a good thing. I know it's how the industry tests games (I'm a software dev so I see it all the time) and it sucks. The fact people like you justify it as "that's how everyone else does it" makes it worse. If you don't demand quality then you deserve what you get.

The math is actually pretty simple for the cost of bugs getting into the live game. At a lower bound you have the cost of patching a bug found after release and pushing the patches out, with appropriate regression testing. Usually that's a man-week or two per bug. On top of that you have the reduction in image (saying people bitch about something else is a fallacy, surely I don't have to explain why) which is less measurable but significant. Lastly you have the number of people who quit and play something else - also not easily measurable but definitely there and statistical data can give you a good idea. Now add the minimal cost of 50ish people for 3-4 weeks and I guarantee you'll end up on top. Remember, this is a game pulling in 40-60% profit margins from subscriptions. They could charge *nothing* for the expansion and still make a profit.

Using an uncontrollable public beta to find bugs is just bad software development practice. I don't mind them using it for the "buzz" but to call it "beta test" is a pretty bad joke. Unless you can recreate scenarios to validate fixes then it has no business being a "test" at all.

Dd

__________________

clint

If you think the ramifications of a serious bug in a computer game is at the same level as in an OS or a major accounting package then maybe you play too many games. ;)

__________________

Ddrak

The ramifications are exactly the same. Loss of profit for the company producing the product. If you think they are different then you misunderstand what companies publish software for. ;)

Dd

__________________

RadarX

I agree with you, which is why they pick certain groups and people to test it. Tradeskill stuff needs to be tested, and that's why people like Calthine and Niami are there. Raid content needs to be tested and that's why the guilds that have been listed are there. A "grouper" or a "quester' are a little more vague to define than a raider or crafter so I can't see a problem with bringing in random people. I don't really know their selection process to be honest.

I can't compare it to OS or Accounting software because I don't see it as operational critical. If our accounting system fubars (which it is right now) our entire purchasing system grinds to a halt. You are comparing software critical to running a business, to software designed to entertain. If it were the same, they'd adopt identical testing methods. I knew you were a software dev when I read your post. You are looking at it from that perspective, not a production perspective. They don't like bugs, but at some point the software has to hit the market.

Your next paragraph I don't even know how to respond. Do you actually design MMO games, because that is a lot of speculative numbers considering SOE doesn't publish any of that. Charging nothing for the expansion is actually a good idea. Works in Asia.

You are assuming the public beta is all they are doing. You ever done something you want people to look over to see if they can find broken stuff? I'm not saying they aren't running the beta in order find bugs because I honestly don't know that. I'm just telling you there are some very dedicated people in there testing.

Ddrak

Actually, it's just my pet bitch about the whole industry and I think the games industry is the worst of the bunch. Testing as a whole sucks and tends to get squeezed out by managers who see the code running and testers as just a bunch of people who impede the process of getting it out the door.

No, I don't design MMOGs, or code them (two very different jobs). Just been reading a bunch of software process stuff lately that uses plenty of those sorts of things to hammer the importance of testing. I think the general rule was something like a bug found after release costs about ten times as much to fix as one found before.

The worst part is that users buy into "it doesn't matter if it only breaks a little bit" mentality. If I shell out $200 a year for an online game and only $200 every 5 years for an OS then I definitely want to see a more polished product for the money spent.

Dd

__________________

clint

If you think the only issue is loss of profit for the production company that explains why you are completely missing the point. Bugs in the OS and accounting software (2 out of many examples) would also have huge implications for the users and in some cases society in general (hey let's compare a bugged Tarinax to a major security flaw in ATMs).

But if you want the same level of QA for a video game as in a NASA launch, for example, be prepared to pay a lot more than you do now.

The difference is that every computer has to have an OS to run. First that means there are a lot more users to piss off if there is a bug and second it means the developer of the OS has a much larger budget for QA.

__________________

Ddrak

Society doesn't pay for testers. It comes out of the production company's budget, which is why the profit margins for that company are the only point in question. You're not seriously suggesting companies care about users or society in general unless there's money involved, are you? That would be an infringement of their responsibility to the shareholders.

I don't, and never suggested they should. Please stick to the argument at hand and avoid bringing in your own strawmen. What I am stating is that ad hoc testing isn't particularly good at finding bugs and that there are far better ways to do it that (i) don't cost much money and (ii) give better results (ie profits) for that money.

That's a weak excuse for accepting bugs from game developers. The fact remains that you pay more for EQ2 than you do for your OS, but are prepared to accept lower quality for your money from EQ2. Your only explaination is something to do with other people?

Dd

__________________

Shaggis

Basically as long as the users are willing to accept it, they will continue with this level of quality. The lack of quality process is very apparent. And quality does not have as much to do with picking the right beta testers as it does having a well thought out quality software development process that involves everyone from product management to software engineers to testers. If you wait until beta to find the showstopping defects you have already failed. I have NEVER worked on a product where we relied on beta testers to find functional bugs for us. They are usually best suited for usability and fringe configuration testing. Ideally nothing more than refinements and minor tweaks should be made after beta. I thought about applying to a testing job at a game company, but I decided that I would loathe the job after what experience I do have on how much they value the quality process.

This quote was the nail in the coffin for SOE in my eyes:

This is an MMORPG that has requirements to support thousands of users and you are testing it with a few dozen people? Don't shoot the messenger here, he's just a PR guy, but whoever fed him that line of crap should be strung up by the short and curlies.

Seriously have you ever experienced someone who is "hardcore" but still doesn't really understand the detailed mechanics of the game? Its a mindset not just how much you play or if you raid x number of nights a week. Some people are just damn good at execution and don't have to know as much about mechanics while others figure out the best way to counter certain abilities or mechanics challenges.
I'm not real sure whether I'm casual or hardcore, but still you should know better than that generalization. :P

I'm sure I could find lots of bugs on beta, but what would be my motivation to report them all? You don't pay me, so you are relying on my feeling of good will, my charitable contributions to the quality of the game because I play it or my desire to get into future betas. Some people might just forget to report a few bugs as a "pension plan" or solely be beta testing in order to get a jump on their competition. However I highly doubt I'm going to see a server wide "Emerald Order" message about a discovery.
I don't think contested content should be available in Beta, they should do that testing internally for obvious reasons.

IMO getting into Beta is nothing more than another carrot they are dangling to keep your interest in the game and to flush out the major server outage type of bugs since they do not scale test well internally. I'm pretty sure their first priority is to keep the servers up and somewhat playable. Whole zones can be non-functional and some raid content completely borked, but as long as 90% of the user base can log in and play they will go live with it.

Ddrak

What Shaggis said.

If what he said disagrees with what I said, then I was wrong. He put it much better than I could have.

Dd

__________________

Illuminator

Game developers don't always have a freehand to become completely sloppy with bugs. Read-only release media is a powerful incentive to get it right the first try, which is why I get more than a little peeved when Microsoft gets fucking giddy about putting hard drives in X-Box's to allow bug patches.

I know it was the case with old Nintendo cartridge games that you were not *allowed* to release with significant bugs, something related to the licensing.

__________________


Bow down before the one you serve. You're going to get what you deserve.