Leaving intentional bugs in code for testers to find

247

We don't do this at our firm, but one of my friends says that his project manager asked every developer to add intentional bugs just before the product goes to QA. This is how it works:

Just before the product goes to QA, the development team adds some intentional bugs at random places in the code. They properly back up the original, working code to make sure that those bugs aren't shipped with the end product.
Testers are also informed about this. So they will test hard, because they know there are bugs present and that not finding them might be considered as a sign of incompetence.
If a bug (intentional or otherwise) has been found, they will be reported for the development team to fix. The development team then adds another intentional bug in a related section of the code just before the product goes to the second-level QA. The project manager says a tester should think like a developer and he/she should expect new bugs in sections where changes were made.

Well, this is how it goes. They say that this approach has following advantages.

Testers will be always on their toes and they will test like crazy. That helps them to also find hidden (unintentional) bugs so developers can fix them.
Testers feed on bugs. Not finding any bugs will affect their morale. So giving them an easy one to find will help their morale.

If you ignore the scenario where one of these intentional bugs gets shipped with the final product, what are the other drawbacks we should consider before even thinking of adopting this approach?

Some clarifications:

They properly backup the original code in source control.
When a tester finds the intentional bug, the development team just ignores it. If tester finds out an unintentional (original) bug, the development team first checks whether it is caused by any of the intentional bugs. That is, the development team first tries to reproduce that on the original working code and tries to fix it if they can.
Just ignore the relationship issues between QA and development team. I specifically asked this question on Programmers, not on The Workplace. Consider that there is good rapport between QA and the development team, and they party together after working hours. The project manager is a nice, old gentleman who is always ready to support both teams (Godsend).

Krishnabhadra

Posted 2015-01-28T10:56:26.480

Reputation: 1 320

Comments are not for extended discussion; this conversation has been moved to chat.

– balpha – 2015-02-04T13:01:29.973

50"A test should think like a developer"... interesting. I would have thought it was obvious that a tester should not think like a developer but like a user. – Trilarion – 2015-02-04T14:05:11.253

9What happens if an intentionally introduced bug covers up another bug the testers could have found had that intentional bug not been introduced? For example, suppose a chunk of code has a fencepost problem and that the development team is unaware of this bug. A programmer decides to insert an intentional fencepost error at that spot. Now the code has a double fencepost error. Suppose the testers detect the error, but don't see that it's a double fencepost error. Congrats! The testers found an introduced bug. The original code will be restored to contain the original fencepost error. Oops! – David Hammen – 2015-02-04T18:40:34.810

12I'm a QE. I'd rather find real bugs, thank you. I'd quit this company like it's on fire. Nobody gets to (intentionally) waste my time. – ArjunShankar – 2015-02-05T12:32:17.530

5"We don't do this at our firm, but one of my friends says that his CTO asked every product manager to add extra features at the start of every feature development cycle..." – Marco – 2015-02-05T17:41:10.710

"... also find hidden (unintentional) bugs so developers can fix them". Interesting! Mobilizing a full scrub down to ferret out real bugs! Bold idea. – jxramos – 2015-02-06T02:07:49.523

This is called "bug seeding". By finding out how many % (a) of the seeded bugs have been found, and knowing the number of other bugs (b) that have been found, you could extrapolate to know the total number of bugs in the application. (b/a) At least, in theory. I'm sure there are a lot of ways in which you could do this wrong in practice. (like, the artificially inserted bugs are probably nowhere near comparable to real bugs, so statistical extrapolation is not appropriate) http://c2.com/cgi/wiki?DefectSeeding

– Erwin Bolwidt – 2015-02-06T02:57:13.390

There's also Defect Seeding mentioned by Steve McConnell - same idea as the one mentioned by @ErwinBolwidt but from someone rather well known in the industry.

– None – 2015-02-06T03:08:37.460

2I suspect adding intentional bugs creates risk. What if an intentional bug actually fixes something unintended? The positive side effect is not reported, the code is removed and a real bug gets through QA. By their nature, these last minute "intentional bugs" will be ill considered, if not, the bugs are wasting too much developer time. – Jodrell – 2015-02-06T11:56:23.403

1How does this affect shipping things? It seems like for every project/release you'd have to allocate almost twice or three times the amount of QA necessary just for this. Not saying it's necessarily a bad thing but it seems for what you are trying to achieve it isn't really worth all the resources that go into it. – aug – 2016-01-13T02:24:02.030

Answers

440

This sounds absolutely nutty. It is expending a great deal of effort for very questionable benefit, and the practice seems based on some faulty premises:

That QA won't work hard unless they know they are being tested every day (which cannot be good for morale)
That there are not enough unintentionally introduced bugs in the software for QA to find
That QA's job is to find bugs - it isn't; it is to ensure the software is production quality
That this kind of battle of wits between development and QA is in some way healthy for the company - it isn't; all employees should be working together against the company's competitors instead of each other.

It's a terrible idea and the project manager in question is a jerk/idiot who understands nothing about people and motivation. And it's bad for business.

To expand on my description of "QA's job:" QA definitely should be finding bugs - both in the code and in their test suites - as an artifact of doing their jobs, but the role shouldn't be defined as "you have to find bugs." It should be "you have to keep the test suites up-to-date to account for new features and ensure all high coverage of testing. If this does not result in finding bugs, then the testing procedures are not sufficiently sophisticated for the product.

James McLeod

Posted 2015-01-28T10:56:26.480

Reputation: 6 542

This sounds absolutely nutty That is my first thought in real words. But they say they have very good success with this approach. They have built up a rating system based on the results of these approach, with a significant benefit for guys who top it. And they say there is no battle(bad blood) between developers and testers because of this. – Krishnabhadra – 2015-01-28T11:25:09.897

17That QA's job is to find bugs - it isn't; it is to ensure the software is production quality This requires some clarification. Isolating and fixing bug is one important process in shipping production quality software. – Krishnabhadra – 2015-01-28T11:26:45.797

If QA runs through a test suite that demonsrates the software works as expected and find no bugs, they have still done their job successfully. As for the company having syccess with this approach - compared to what? – James McLeod – 2015-01-28T11:30:55.317

+1: "all employees should be working together against the company's competitors instead of each other." Sadly an awful lot of management doesn't get this idea. – Julia Hayward – 2015-01-28T11:56:30.757

21Actually, in many companies, QA's job is to find bugs, and if there were new features added to a product, and QA just then runs a test suite which does not show up any bugs, I personally won't trust that test suite and supect it to be imcomplete. – Doc Brown – 2015-01-28T12:08:48.213

5... and I find it pretty reasonable to insert some bugs to test if the test suite is really working. As long as that is done in cooperation with QA, I don't think it is a negative thing. So I give you a -1 not because I do not agree with lots of what you wrote, but because I think your perspective too single edged. – Doc Brown – 2015-01-28T12:38:35.017

8I agree, except for the last point. Having an adversarial sort of approach between QA and development (and business) is largely inevitable. Each group has their own desires and expertise. As a company, these need to balance out to work well. In my experience, "playing nice" just leads to the groups not pushing for their agenda, leading to stagnation or imbalance. The best companies I've seen have been ones where development, QA, and the business side push for their needs, but acts as a check on the others, leading to compromise on the best balance for the company. – Telastyn – 2015-01-28T12:44:16.420

3@DocBrown - that's a fair point, and I shall update my answer this evening to show a little more nuance. I agree that QA needs to find bugs, but I see that as a side effect of doing their job, not their main goal. – James McLeod – 2015-01-28T12:59:34.790

39I would add another point: An intentional bug could hide a true one that would have appeared if the intentional bug didn't stop the process (by throwing an exception for example) before. – nkoniishvt – 2015-01-28T17:58:32.420

1"It's a terrible idea and the project manager in question is a jerk/idiot who understands nothing about people and motivation." +100 for this. – David Moles – 2015-01-28T19:29:10.420

4@DocBrown Just because a test suite finds bugs doesn't mean it's trustworthy or complete. And just because a test suite finds bugs intentionally introduced by developers doesn't mean it's going to find the unintentional ones. – David Moles – 2015-01-28T19:33:22.623

27If I were a QA guy and found out I was wasting my time researching and documenting purposefully introduced bullshit bugs, I would be finding a new job. – Kik – 2015-01-28T20:01:01.227

1@DavidMoles: you should have read my comment more carefully. I did not write "a test suite which finds bugs is trustworthy", only "a test suite which does not find any bugs is not very trustworthy", and in my comment above, I was not referring to bugs introduced intentionally by the devs (though I think if a test suite does not even find those bugs, it is really not trustworthy). – Doc Brown – 2015-01-28T21:57:12.083

1@Kik: so as a dev you find it ok when your work is double checked by a tester, but as a tester you think it is not ok when your work is double checked by someone else? – Doc Brown – 2015-01-28T22:00:44.067

2QA shouldn't be doing their work in a vacuum. If new features are added then they should be talking with software on the best way to test the feature. After incorporating the updated tests/procedures they should get concurrence from software that their tests/procedures adequately test the new feature. They should dry run those procedures with interim builds of the software. So by the time a formal QA test is run, the QA test procedures/test scripts should already be known to work. – Dunk – 2015-01-28T22:20:52.430

@DocBrown You should have read my comment more carefully. I didn't make any claims as to what you wrote. – David Moles – 2015-01-28T22:23:33.547

1@DocBrown, "...but as a tester you think it is not ok when your work is double checked by someone else?" He did not say that. And also, the customers check the tester's work, right? – Tracy Cramer – 2015-01-28T22:58:43.713

1@Telastyn It's quite easy to work for a common goal - the customer. True, competition can sometimes improve productivity and relations (in very specific teams), but it can very easily get out of hand. A big part of delegating tasks is mutual trust - you might be breaking that. And if QA spends 20% of their time handling bugs "on purpose", that's 20% they put in vanity metrics instead of customer value. – Luaan – 2015-02-06T08:21:03.673

1Additionally, I suspect adding intentional bugs creates risk. What if an intentional bug actually fixes something unintended? – Jodrell – 2015-02-06T11:55:32.573

When you say "If this does not result in finding bugs," surely you mean "if this does not result in finding bugs, when they do exist?" – Amani Kilumanga – 2016-01-13T00:52:01.780

Yes, I surely do. – James McLeod – 2016-01-13T01:24:41.443

198

Well, based on what I've learned:

It's not a school nor job interview;
The testers are not children;
It's not a game;
It wastes company's money.

The QA are not there only to find bugs but also to worry about how intuitive the system is, what is the learning curve for the user, usability, and accessibility in general. For example: "Is the system ugly?", "Is the user color blind and stuff is red and green?" They should complain too.

The minimal requirements for a system to pass QA is usually described in a user story for that particular feature or in how magical the PO wanted the system to be in his head.

tl;dr

It's not only bugs, testers should grow out of this narrow view.

SparK

Posted 2015-01-28T10:56:26.480

Reputation: 1 717

23+1 Strongly agree with all 4 points, especially the first. The competitive approach that so many newer devs bring often reflects their previous 15 years of schooling - an extremely competitive environment - unlike the workplace where co-operation would be a better approach. – Michael Durrant – 2015-01-28T12:51:21.270

1Much prefer this answer to the top answer. – Pharap – 2015-02-02T00:24:10.297

1"The QA are not there only to find bugs but also [...]" - I just want to say that in many places, the terms software testing and quality assurance are used interchangeably. Yes, that's bad. Where I used to work, we had an employ who used state -- at every QA department meeting -- what we do here isn't quality assurance but quality control. (She meant this as a criticism of our QA department.) – Mario – 2015-04-17T13:41:49.677

Bad idea.

From the tester's point of view: "So they will test hard, because they know there are bugs present and not finding them might be considered as their incompetence." Basically the devs are booby-trapping the code. Few people like doing work which is ultimately pointless (because the bugs are known in advance) but which still affect how they are perceived. If there are tangible punishments for not finding the booby-traps, more so. And do you know testers thrive on finding bugs? That sounds like a toxic confrontational environment; a QA should be happy if the code they are examining is high quality. Although if they are paid by the bug... http://thedailywtf.com/articles/The-Defect-Black-Market

From the dev's point of view: the QAs are being incentivised to find the bugs you know are there. That may well increase the likelihood of real bugs going out of the door; the QAs are spending at least some of their time looking for the sort of bug that is easy to plant, not really subtle ones. Plus there is a small chance that a booby-trap may make it out of the door.

Julia Hayward

Posted 2015-01-28T10:56:26.480

Reputation: 2 595

If they pay per bug, then this

– BЈовић – 2015-01-28T12:22:06.823

+1 It reminded me of that DailyWTF post too! – Andres F. – 2015-01-28T12:49:09.053

12"incentivized to find the bugs you know are there" Excellent point. If an organization is doing this, it likely means that someone is breathing down the QA folks necks to make sure they find the planted bugs, so that will be their top priority. What if they get together and figure out, say, "Hey the planted bugs are almost always that it fails to save one field on an edit screen with a bunch of data" (or whatever). Then they'll spend an inordinate amount of time looking for that one type of bug, and increase the chance that they'll miss other types of bugs. – Jay – 2015-01-28T18:23:05.827

The first thing that jumped to my mind was Wally's going to code someone else a minivan this afternoon

– Dan Neely – 2015-01-30T22:13:59.633

10>real bugs going out of the door. I used to do Big Testing. You start with the thesis that (non-trivial) code always has bugs. QA are the heroes who find them before the customer does. The bugs are always there. If you introduce artificial bugs you are wasting time you could be spending finding the real bugs; time for testing is limited, you are reducing quality by adding unnecessary work. – RedSonja – 2015-02-02T08:14:10.283

I agree totally with the answers above as to why this is bad for motivation and just generally awful people management. However, there are probably sound technical reasons for not doing this as well:

Just before the product goes to QA, the dev team adds some intentional bugs at random places in the code. They properly back up the original, working code to make sure that those bugs aren't shipped with the end product.

Based on the first statement, you never actually test your intended production code in these two passes.
I would imagine you vastly increase the likelihood of accidentally including an 'intentional' bug into your released production code when trying to rush through a change for a customer. Might cause a few red cheeks at some point.
I would imagine that this just trains your testers to think like your developers (i.e. how would Tom add a bug here) which probably makes them less likely to find the bugs that Tom hasn't thought about.

Paddy

Posted 2015-01-28T10:56:26.480

Reputation: 1 401

40+1 for you never actually test your intended production code in these two passes. How you can even think about releasing without testing the production code is beyond me; if you're testing again without the intentional bugs then you're repeating your effort and wasting the initial effort. – adamdc78 – 2015-01-30T00:21:19.577

Edit

I want to be clear that this answer is only talking about the concept of testing your QA process, and I'm not defending the specific methodology portrayed in the question.

End Edit

There is a valid reason to check if your testing/checking is actually working. Let me give you an example from manufacturing, but the principle is the same.

It's typical when feeding material through a machine that the feeder might not push the material through far enough. This is called a "short feed" and to prevent this we might install a "short feed sensor" (typically a through-beam type sensor that's blocked by the material). This sensor detects the end of the material when it reaches the full feed length. At a certain point in the machine cycle we check that the sensor is blocked, and stop the machine if the check fails.

Now you have to think about how the test itself can fail. For instance, some dirt or other debris can block the sensor and it will always report "OK" and never stop the machine. Also, the nature of the sensor is that the receiver turns on when the beam hits it, so depending on the type of sensor you have installed, electrically you get an "ON" input when the sensor is not blocked. That means if the cable got cut or power was lost to that sensor, or the input failed, the logic of your program would read "OFF" and that would mean "blocked" or "OK".

To catch these failure modes of the test, we typically insert a second check to make sure the sensor is actually unblocked during a second part of the cycle. In this way we check that the test is actually still operating (as best we can).

Similarly there are many ways that a QA department can fail. Perhaps the automated tests didn't run and the report is looking at an old copy of the test data. Perhaps someone isn't doing their job right. Testing the QA department is a reasonable thing to do.

Obviously the draw-back is that a "test bug" could make it through the QA department and into the finished product. In the manufacturing industry there are sometimes cases where a known bad part, sometimes called a "Red Rabbit", is inserted into the process (typically by someone from QA) and they watch that part go through the process and measure how long it takes to find the part and remove it. Normally this part is painted bright red (or orange) so it can easily be tracked. Since someone is watching the part go through process during this test, the chance of it making it into the final product is practically nil. There are, of course, apocryphal stories of someone throwing a known bad part into the process to "see if the system can find it", and of course having to quarantine all the final parts produced that day and manually sorting them, but that's just a case of not performing your test with due diligence.

Scott Whitlock

Posted 2015-01-28T10:56:26.480

Reputation: 19 232

Comments are not for extended discussion; this conversation has been moved to chat.

– Yannis – 2015-01-29T19:02:18.423

Hello all. The discussion was getting a bit too long for comments. As you can see from my earlier (automated) comment, I've moved all comments to a dedicated chat room. If you want to continue discussing the answer, please do it in that chat room, and not here. Thanks.

– Yannis – 2015-01-29T19:07:43.667

3So described approach could be used to test QA occasionally, not as a permanent process. – gerlos – 2015-02-03T11:40:54.763

Honestly, I'd call this behavior blatantly unethical and impractical. The PM is in need of some serious retraining, if not termination.

It demonstrates a fundamental lack of understanding of the concept of quality assurance. Testers should not think like developers: they should think like end users. The entire reason for having QA teams is that developers are inherently too close to the code; QA is supposed to maintain enough distance from the code that they can catch what the devs miss.
It wastes QA effort. Assuming that these bugs are not trivial -see below for when they are- it means that QA is spending time and resources investigating things that are already known, when they could be spending that effort seeking out what isn't known.
It wastes developer effort. For QA folks to catch these non-trivial bugs, developers must first write them. This requires yet further effort, spent not just coding the bugs, but also considering the software requirements and design.
It puts production at needless risk. It is only a matter of time before changes don't get merged in properly.
If it doesn't do the above, then it is pointless. If all of the known bugs are trivial, then they won't catch substandard workers: they will only catch people who are doing no work at all. There are better ways to do that.
It poisons the work environment. Your QA testers are professionals. They should be trusted to be professional until there is actual reason to suspect otherwise. When there is reason to suspect otherwise, there should be a proper investigation instead of these mind games. Anything else kills morale.

Seriously. Even if the PM's paranoia turns out to be well-founded in this specific case, this is not someone who has any business managing testers.

The Spooniest

Posted 2015-01-28T10:56:26.480

Reputation: 1 790

Personally, I feel uncomfortable with this approach.

The main thing that concerns me is the practicality of inserting intentional bugs. This seems to me to be difficult to do in any way that is predictable.

Any code changes (intentional or otherwise) risk having side-effects. These side-effects may well be revealed during testing but it may not be obvious (even to the developer who planted the bug) what the root cause is. It doesn't feel "safe", if you know what I mean (I'm speaking from my gut here).

Also, the tester will waste a lot of time testing code that isn't actually going to be released. Once the intentional bugs are removed, a complete re-test should be done anyway, in my opinion. That's the whole point of testing. Something changes, anything, and you re-test everything. Ok I know that doesn't ever happen in practice, but that's what regression testing is all about.

So, overall, not convinced.

On the other hand, we tend to let the customers verify the work of the QA teams, which is possibly not ideal. It is a very powerful feedback loop though.

Roger Rowland

Posted 2015-01-28T10:56:26.480

Reputation: 370

1I like the idea of feedback loop power! – jxramos – 2015-02-06T02:19:27.230

It's a bad idea for all the reasons already given, but bug seeding is a useful tool for a different purpose. You can use it to get a rough metric of how effective the QA process is.

In its simplest case, let's say you seed 100 bugs and they're representative of the full span of real bugs (I know, unlikely, but I'm simplifying). You do not tell QA you're doing this to avoid spoiling the experiment. At the end of the QA process let's say they found 60 of the 100 seeded bugs (and other real bugs). Now you know QA is finding 60% of the bugs.

You can extend this further by counting the number of real bugs QA found and apply the fake bug ratio. In our example, if QA found 200 real bugs then you can conclude they only found 60% of them, so 133 remain.

Of course, this is just a broad estimate with huge error bars. Writing realistic, representative bugs is hard. The bugs you write are likely to be easier for QA to find because developers are trained to not write bugs. It may be better to simulate a class of bugs such as off-by-one errors, Unicode mistakes, buffer overflows, and so on.

This should be applied to the entire QA process which would include developer unit testing, continuous integration, and, if available, a dedicated QA team.

This is a metric, and should not be hijacked as a management motivational tool.

Schwern

Posted 2015-01-28T10:56:26.480

Reputation: 369

This would be the only way that any meaningful data could be collected. But the amount of time and effort that would be required to determine the proper test cases to get meaningful results would bust any budget and schedule. And even if you were given the budget and schedule you would then have to get over the hurdle of ensuring you have people qualified to understand statistics and software well enough to be able to identify the proper subset of tests. I don't think you'll get all of that in one project. So in real life the best this method can do is get erroneous, if not misleading numbers. – Dunk – 2015-01-29T16:20:37.507

1SQL injection is a good one to do this for, as you can just pick n sql statements at random to "break" – Ian – 2015-01-29T18:19:36.727

1A big problem is that intentional bugs will tend to be very different from issues you would get naturally - you might simply be training your QA to think like the programmers. That's pretty much destroying the whole point of QA - to have a POV closer to the customer than to the code. A huge part of QA is being a sanity check against things that devs think intuitive (either for ignorance of the ignorance of users, or for proximity to code, time spent with UI etc.). Intentional bugs are not a well distributed sample. – Luaan – 2015-02-06T08:28:25.757

Bad idea.

This is the sort of logical, binary approach that developers often bring, but it is demotivating for the QEs. It simply demonstrates a lack of trust. QEs often get placed in these situations without much input from them, and it assumed that they are OK with it, and it is not their place to suggest otherwise.

This kind of thinking combines to QEs being manual testers only and not being motivated to understand the actual code under test.

I am a senior QE and this is a familiar issue in most organizations I have worked in.

Michael Durrant

Posted 2015-01-28T10:56:26.480

Reputation: 9 313

7My wife did QA for 8 years, and just left for dev -- primarily because of trust issues like this. It's just insulting to the tester. – insta – 2015-01-28T15:23:04.083

I'd say bad idea.

One: Programmers are going to spend time putting deliberate bugs in the code, and some effort to save the good version. While the testers should presumably be testing everything, including the features with the planted bug, when they find one they will presumably have to go back and re-run that test to verify that this was indeed a bug (and not that the tester got confused in some way). At a minimum, testers are going to spend time writing up the planted bugs. Then the programmers have to spend time fixing the bug that they planted. This is a lot of effort that could be spent trying to write good code and write up real bugs.

Two: It sends a clear message to the testers that the programmers and/or management think they are not doing their jobs and must be treated as children. I can't imagine that this is good for morale. As a programmer, if I was given ambiguous or contradictory specs for a program and had to spend a bunch of time getting them clarified, and then after wasting hours or days my boss told me, "Oh, yeah, I deliberately put contradictory statements in the specs just to make sure you were really reading them", I think I would be really annoyed. If that happened regularly, that might well be enough to make me look for another job.

In real life, all but the most trivial code changes WILL have bugs. I have never had a problem with testers getting complacent because the first draft code they were given was so often 100% perfect. I've had to deal with lazy testers who don't do an adequate job, but they didn't get that way because the programmers were so perfect. The best testing person I ever worked with once told me that for a new software release, he set himself a personal goal to find 100 bugs. Okay, whether 100 is a realistic number depends on how big the product is and how extensive the changes are, but in our case, he almost always managed to meet that goal. Sometimes he had to stretch things, like calling a mis-spelled word in a message a "bug", but hey, it did have to be fixed.

Post script: If you do this, I bet that sooner or later the programmers are going to deliberately plant a bug, the testers don't find that particular one, and the programmers forget to put the good code back. So now a deliberately planted bug gets shipped to the customer.

Jay

Posted 2015-01-28T10:56:26.480

Reputation: 2 324

That point about the spec in "Two" is an excellent analogy. – Kyralessa – 2015-04-17T18:06:54.760

I don't really think this a bad idea. There's just a lot of things that I would speculate work better:

Make QA accountable for the quality any way you can. For example by making support their responsibility also. This will increase their motivation to make sure the products shipped have higher quality. It always takes less effort to discover an inadequacy (bug, obviously missing feature, counter-intuitive behavior) yourself then to try to understand what your upset user is trying to explain. And putting some of that responsibility even on developers might increase their motivation to help the QA do their job the best they can.
Have multiple QA teams, which can the compete. You need to find a sensible metric of course. Definitely not just the number of issues. Factoring in the severity of the defect or the business value (as determined by stakeholders) of proposed enhancements should help.

It's hard to tell whether QA is "good enough". It's easier and possibly even better in the long run to find ways for QA to be "ever improving".

Still, there's one issue to be aware of if you introduce intentional bugs: How do you know the "correct" code ever actually was correct in the first place? After the 2nd QA you remove all intentional bugs that hadn't been discovered. There's no way of knowing that you aren't either just replacing them by code that is broken in a different way or that you are not enabling broken behavior that was unreachable before (exaggerated example: some dialog did not open because of an intentional bug, but the dialog itself is broken - you just don't find out because testers didn't get to see it).

back2dos

Posted 2015-01-28T10:56:26.480

Reputation: 26 081

4If you left that first sentence out I would have +1'd you because everything else is good:) It is simply a terrible idea, bad is an understatement. The easiest way to make QA accountable is by keeping track of the number of bugs that make it into the field. That alone will accomplish EVERYTHING the proposed method claims to be its benefits. – Dunk – 2015-01-28T15:56:10.437

@Dunk: Keeping track of that number won't make your team better automatically, much like keeping score in a sport doesn't make you the best athlete you could be. In fact athletes do train, i.e. perform artificial tasks to increase their performance in a controllable fashion, which is not unlike what is being proposed here. Unless you have an idea how to get people to improve that number, it's of little value. – back2dos – 2015-01-28T18:01:46.643

I don't claim that it will improve anything. I only claim that it will accomplish everything the "insert false errors" method will accomplish but without all the costs and wasted time. What it will do is give an indication if too many defects are passing through QA. If it is determined to be the case then either the process or the people need to be re-evaluated. The "false error" method doesn't provide any more information than that, but it actually provides less useful information. So your costs are higher for less gain using the "false error" method. Like I said, a terrible idea. – Dunk – 2015-01-28T18:35:42.170

@Dunk Then you didn't read the question properly. It suggests that this method increases morale and also thoroughness. Also the number of bugs that make it through QA doesn't reliably measure the effectiveness of the QA team. It is equally affected by how many bugs the developers introduce. If they start using TDD and there is a sudden decrease in defects in the release, what does that say about the testers? Nothing. – back2dos – 2015-01-29T08:01:10.523

@Dunk As opposed to that, the "false error" actually does give you more information, under the assumption that the difficulty of finding them doesn't fluctuate erratically (which can be arranged). Because you know how many artificial defects there are, you can tell exactly what percentage of them were caught in QA. So the extra information you get is how effective QA is in detecting artificial defects. And that number certainly correlates more with their overall effectiveness than the one you suggested. – back2dos – 2015-01-29T08:12:29.547

@2dos That is completely and totally false unless there was an extensive study made to ensure the sampling of injected errors was correlated to match the type of bugs that would be found by a fully working QA test system. That just isn't going to happen. So your claim might be nice in theory but will not reflect reality in the slightest. Thus, any numbers you come up with will be pure fiction. Just because you can gather numbers doesn't mean you can use them in any old way you want and have them reflect anything meaningful. – Dunk – 2015-01-29T16:03:19.837

@Dunk No that is not "completely and totally false". Try reasoning in categories other than extremes. I said the number correlates more than the one you proposed. If less bugs make it into production in release B than in release A, then it can meaning anything. For example that release B was easier to get right. Or, as I said, that the programming methodology has improved or what not. Or that users do not discover the features in B and therefore do not report defects they encounter. Almost none of the effects you propose measuring say anything about the QA. – back2dos – 2015-01-29T16:34:41.180

In real life, what would end up happening would be equivalent to somebody going to Minnesota in August and measuring the temperature for 10 straight days and then concluding that the average year-round temperature in Minnesota is 85 degrees. Without applying sophisticated techniques requiring wide ranging skills in software, statistics and the actual system itself all the information you gain will be like the Minnesota example. So your data will at best be wrong most of the time and actually misleading much of the time. No data is far better than misleading data. – Dunk – 2015-01-29T16:40:51.813

@Dunk You are simply contradicting yourself. You said the number you proposed is a sensible metric. It is far from it, as I have abundantly explained. Your analogy is much better suited for your claim than mine. Why I say the idea is not terrible is because it is merely the application of the general idea that if somebody improves during training, they will do better in the field. That's why people train. Of course you cannot accurately measure the effectiveness of QA. I didn't claim you can. But suggesting to therefore not even try is a perfect solution fallacy.

– back2dos – 2015-01-29T17:01:56.277

I certainly am not contradicting myself. The goal of QA is to ensure that a quality product in the eyes of the customer is delivered. It could be the most horrendous piece of garbage imaginable but if the users are happy with it and don't find bugs worth reporting then QA did its job. Users will report bugs they are concerned about. Those matter. No other QA statistic carries any meaningful weight when compared to that one statistic. You can fix a thousand bugs but if the users don't care about any of them then you added no value doing so. – Dunk – 2015-01-29T17:22:22.903

As for the perfect solution fallacy, that is what is being claimed by this system as the reason for needing it. They don't think what exists is perfect. I'm not claiming we need a perfect solution. A properly run QA system works very well but is far from perfect. Not only am I claiming that the proposed inject errors method is not perfect, I am claiming that it is useless at best and potentially harmful. If there was anything worthwhile then I would concede. The only possible way to make this method worth while would be as Schwem's answer described, which isn't practical in the real world. – Dunk – 2015-01-29T17:28:30.310

@Dunk I have pondered this a lot. We probably just have a very different culture. You seem to perceive work as a process that creates products to satisfy some metric. I think of it as a path to personal growth, a broadening as well as a deepening of skills. Circumstances change. What has been good enough yesterday, may be obsolete tomorrow. The habit of putting yourself to the test helps in transcending the volatile nature of things. Developing and following it, I think, is not a terrible idea, but one well worth the attempt and even failure. – back2dos – 2015-02-05T11:01:14.243

Yes, athlete's train. But the analogy to that would be to send your testers to a class. We don't tell athletes at the end of a game that half their touchdowns didn't count because they were part of a training exercise. If we did that regularly -- some parts of the game count and some didn't and we don't tell the players which until the end -- I suspect the players would waste a lot of effort figuring out what parts were real rather than just doing their best. And when you tell a player that his incredible long pass that he was so proud of didn't count, that would have to kill team morale. – Jay – 2016-01-13T17:27:06.177

@Jay Classes are for education, not training. Your analogy is also off in that what is being suggested is the exact opposite of not making it count when a tester finds a "fake" bug. If anything, the actual risk is that testers will (consciously or not) adapt their behavior to "score" on the fake bugs and become less effective at their task despite all indicators suggesting the opposite. Which is why I stressed that this mustn't be the only indicator. But if someone can get it to improve their self-evaluation, then great.

– back2dos – 2016-01-13T19:46:00.400

As others have said, the developers should not be purposefully adding bugs in the software, but it is a legitimate strategy for your test suite to add bugs into the software as part of the testing process.

It's called mutation testing. The idea is to use software to automate the creation of small changes in the source code (called mutants). The changes are designed to create different behavior, for example, we could change

if x < 10:
    print "X is small!"

into

# we flipped the inequality operator
if x > 10:
    print "X is small!"

and a good unit test should detect that the mutant code fragment no longer works as expected and kills the mutant. When the original code passes the test, and all mutants (that don't happen to be functionally equivalent) fail the test, then you know that your code and your tests are strong.

James Mishra

Posted 2015-01-28T10:56:26.480

Reputation: 511

I like the idea. Was it General Patton who said, "The more you sweat in peace, the less you bleed in war."

Putting intentional bugs "wastes time" of the testers. But that also makes them work harder, meaning that they will also do a better job of finding UNintentional bugs. (And you have a copy of the "original" so you don't have to live with what you've done.)

Finding more unintentional bugs will likely save you more grief in the long run that the cost of dealing with the intentional ones.

Plus, you can get an idea of how good your testers are, not a small benefit in itself.

Tom Au

Posted 2015-01-28T10:56:26.480

Reputation: 793

1I think that there are good parts to this. It's better to find a bug BEFORE it makes it into the wild, and I'd rather press my internal QA (That's what they are getting paid for after all right?) than respond to external attacks. Bug Hunting is a part, and as long as this kind of testing is handled properly, I don't see why it can't be a valuable part. – WernerCD – 2015-01-28T14:53:59.730

1The copy of the "original" may not be bug-free and is also by definition untested (because the code was changed to add bugs). – Roger Rowland – 2015-01-28T15:06:07.150

@RogerRowland: With all due respect, OTHER bugs in the original (not related to the changes) will probably be picked up by the tests. That's as good as one can hope for. – Tom Au – 2015-01-28T15:13:20.970

1In my experience, bugs are not isolated animals and they don't sit alone. Software is part of a system and bugs - intentional or not - affect the system. Unless of course we're talking about trivial pieces of software. – Roger Rowland – 2015-01-28T15:14:39.593

@RogerRowland:Interesting point. So you're saying that the "new" bugs could affect the system "over and above" their own weight. Then I guess one would have to do a cost-benefit analysis of "bugs found" versus (potential) new bugs added (not accounted for originally). – Tom Au – 2015-01-28T15:17:21.837

18There is zero proof that this method would find even 1 more additional bug beyond the intentionally inserted bugs. There is zero proof that this would make QA work harder to find bugs. They may try less hard. Also, since you've wasted an entire running of the acceptance test procedure testing cycle while testing intentionally broken code(Our full blown test takes 3 weeks), you now have to retest with the actual code that is going to be deployed because the broken version is not the same build, so its tests are pretty much useless for validation of the "real" build. – Dunk – 2015-01-28T15:21:51.733

@Dunk I agree. Tom, we may have an unusual situation here, we have a codebase coming up to 25 years old, masses of technical debt and as a result the viscosity is high and the chances of small change in one place having big effect in an unrelated area are significant. We have ~11 million lines of code in all, so it's not the sort of beast you'd want to deliberately antagonise with intentional bugs. – Roger Rowland – 2015-01-28T15:39:03.633

6I'm guessing that Patton meant that you should have rigorous training and field exercises during peace time. The analogy would be to have rigorous classes in IT school or post-degree training. I'm pretty sure Patton didn't mean that officers should be instructed to shoot at their own troops from behind to keep the troops on their toes! – Jay – 2015-01-30T17:01:12.900

@Jay: I'm not sure that Patton did this. But apparently some of the Soviet generals did this. – Tom Au – 2015-01-30T18:45:04.363

@TomAu Good old fear based management. Because it works so well, right? :) – Luaan – 2015-02-06T08:35:09.970

@Lusan: Better in some contexts than in others. You need to "pick your spots." – Tom Au – 2015-02-06T14:01:52.180

There is no basis for a reward or punishment on its own merit, but on the result of the behavior you're targeting. And sometimes there are unintended consequences. Is the goal to keep the QA team from slacking or to make some manager feel like he's actually contributing something without realizing he's just getting in the way.

Positive Outcome - The QA team works harder to find bugs. Who knows, maybe they see this as a challenge. It's a friendly game. Or they're just doing because they're being watched (Hawthorne Effect?).

Negative Outcome - They may not work harder and find the bug anyway. QA sees this as petty and adversarial. So now, they go into hyper-bug finding drive and return all sorts of nit-picky little problems. That font doesn't render properly when I take a screen shot and convert it to a pdf and view it at 500%.

No Impact - sound to me like this makes no difference, so why bother? You just risk wasting time and irritating people.

We could all agree that this won't work 90% of the time. That doesn't do much good to the other 10%. Test things for yourself. Are customers more satisfied with a release that has the intentional code bugs? Does it impact worker morale and productivity in other areas? Increase turnover? You tell us.

JeffO

Posted 2015-01-28T10:56:26.480

Reputation: 33 459

Definitely agree with this leading to nit-picky problems being reported. – Adam Johns – 2015-01-28T15:27:06.950

@AdamJohns - You never know for sure unless you try and test it. There are better ways, so this would almost be a last resort for me. – JeffO – 2015-01-28T16:56:21.377

Coming from a world where developers are expected to write and run the tests themselves, this "testing" "QA" silo you are referring to frightens and confuses me, so I'll try to answer from this perspective. As an aside, qualified QA engineers, from my perspective, (as is described well in @SparK's answer), should focus on the larger issues of making sure that the software fully satisfies the user stories and has overall "quality" (in regard to the domain the software is intended for), instead of hunting for bugs.

What drew me here is @JamesMcleod's mention of "defect injection" in the comments to the question. I actually think that having the developers think how they could inject bugs into the system is a great idea for targeting the concept of defense in depth. No single bug should ever be enough to bring down the entire system in an uncontrolled fashion (without clear actionable logging), cause any data corruption, or by itself expose a security vulnerability.

Having the developers of each component create intentional defects, handle those of other components and overall enter a more adversarial mindset about their software could possibly do much towards improving the robustness of the software. Even the immediate benefit could be significant - I'd require that during each such injection of a new kind of defect (that was hitherto untested), the developer would immediately cover it by a new test, which will be set with a flag that will allow the bug to live in the code base undisturbed for a short while, and then switched on before delivery (and the defect removed), to turn into a regular test that will make the test suite more comprehensive.

A related option is the use of feature flags to intentionally turn off features in particular components to examine how other components deal with that. I would also like to highly recommend reading the free book/article "Learning from First Responders: When Your Systems Have to Work" which describes such extensive testing of the software infrastructure to be used by Obama team for the 2012 election.

yoniLavi

Posted 2015-01-28T10:56:26.480

Reputation: 281

4Rather than having developers "inject" bugs in the code, their time would be far better served identifying how those bugs could have gotten in the system to begin with and then correct the code to ensure those bugs can't happen or are handled properly by the software. The goal of developing a project is not to test the QA system, it is to build a usable robust and working system that does what its users want it to do. – Dunk – 2015-01-29T16:33:56.480

As others have already said, it's not the job of QA to solely find bugs. I would go further and say it's not their job at all, technically. Developers should be responsible for keeping their own code bug-free. Test suites should be run before new code is ever even committed, and if the test-suites fail, then it should never be making it to QA in the first place. Introducing bugs intentionally means you definitely can't pass your test suites, so why is your code going to QA?

The job of QA is to validate the application against the user stories it implements. They should test the flow, the UI, etc. and ensure that the user can do everything the user should be able to do, in the most usable and accessible way as possible. While doing this, of course, they may stumble upon bugs, but that's a side-effect of what they do, not what they do. Remember QA means Quality Assurance, not Bug-free Assurance.

Chris Pratt

Posted 2015-01-28T10:56:26.480

Reputation: 3 155

This isn't necessarily as crazy as it sounds. It rather depends on your motivation. If you are looking for a stick to beat your test team with, well that would be crazy. On the other hand, one of the most difficult things in software development is to know how effective your testing approach is.

So if you structure it properly, you could use this technique to estimate how many unfound bugs remain in the product you are about to ship. So imagine that you've artificially seeded 100 bugs in your test build, and the testers find 50 of them. Then you can infer that there's a certain likelihood that if they also found 50 unseeded bugs, perhaps there are 50 left to find.

Of course, this is fraught with many problems. You could decide whether to ship based on these statistics, but in real life, you might find one very nasty issue, or a thousand minor irritations.

Still - knowledge is power, and without this technique, you have even less idea of the quality of your code base. If you can implement it respectfully and for the right reasons, I'd say "Why not?"

Dominic Cronin

Posted 2015-01-28T10:56:26.480

Reputation: 377

One thing nobody else has mentioned yet: mutation testing.

This is where an automated tool takes your source code and deliberately inserts bugs into it. (E.g., delete a randomly-chosen statement, change an AND to an OR, or whatever.) It then runs your full test suite, and checks whether the tests pass.

If all tests pass, then there are two possibilities:

The thing that was changed doesn't do anything. In other words, you have dead code.
The change introduced a bug which your test suite isn't catching. You need more tests.

Note that, unlike your proposal, everything I've described above is automated. You're not wasting developers' time inserting pointless bugs by hand. And you're not wasting testers' time finding known bugs. The only thing you're using up is machine time, which is much cheaper. (Machines don't get bored of doing the same test 20,000 times. Humans stop caring after a while!)

I would suggest that automated mutation testing is a far, far better approach than the manual scenario you're talking about.

Note that if you ask a developer to manually insert bugs, the kind of bug you get is probably not representative of the kind of accidental mistakes humans might make. (E.g., if you haven't realised there's a possible race condition, you're unlikely to insert a deliberate one either.) Whether an automated tool manages to be more objective remains to be seen, of course…

MathematicalOrchid

Posted 2015-01-28T10:56:26.480

Reputation: 401

Although it is a bad idea in general (the other answers perfectly explain why), there is a few special situation where intentionally injecting bugs into the production code in a controlled, temporary manner can make sense.

When you refactor the test code - and you should, test code deserves the same attention to detail as production code - you might want to know whether the test code is still finding the bugs it's supposed to find.

You can then intentionally break the production code in order to verify if the tests still work.

There are multiple levels on which this is possible:

A developer that has just refactored some unit test might break the production code to verify that the unit test still finds what it's supposed to find.
A tester that has just refactored some acceptance test might break the production code to verify that the acceptance test still verifies what it's supposed to verify.
If the interface is stable and robust enough (i.e. protocol based), the company might want to keep a suite of known faulty product versions and run tests against them in order to regression test the test.

Whether these things make sense depends. If I'm a developer and it takes me just a minute to inject a bug, test the unit test, remove the bug - then why not. But I should have my editor, my cycle and my version control system under such a good control that I would not accidentally commit / deliver / check-in / push the bug. Same goes for the tester and the acceptance test.

Whether it makes sense for an organization to keep suites of known faulty product versions and regression test the test depends. For an online shop I wouldn't. For automotive embedded, aerospace embedded, banking cards or pay TV cards I would.

How much effort this is strongly depends on how decoupled the tests are from the production code. The more decoupled the tests are from the production code, the less effort it is to do this, the more cohesive the tests are with the production code, the more effort.

The reason is simply this: When your tests and your production code are cohesive, changing the production code requires changing the tests often, and that would break the dependency between the tests and the faulty production samples. You would then have to also maintain the faulty production samples. In rare cases even that can be worth the effort, and mocking as well as smart usage of a version control system can significantly reduce the effort, but it requires far above skilled developers.

The concept of intentionally injecting faults in production code is called sabotage, the injected fault is called saboteur.

Christian Hujer

Posted 2015-01-28T10:56:26.480

Reputation: 1 713

A tester who is not taking the code-to-be-tested directly from the repository is doing it wrong. (1)

A developer who is checking in known-faulty code into the repository is doing it wrong. (2)

So at this stage, there is already no way for this scheme to work without one or both sides violating very basic premises of how development and testing ought to be done.

(1) Because you need to document which version you tested. A version tagged by a Git hash or a SVN revision number is something you can test, "the code Joe gave me" isn't.

(2) Because you just don't, outside of a test driver that is expecting failure.

This is an attempt at a shortest-possible, "elevator pitch" reason that should make immediate sense to devs, testers, and management alike.

DevSolar

Posted 2015-01-28T10:56:26.480

Reputation: 365

1This is a circular argument. You are saying "seeding bugs in a test build is wrong because developers shouldn't make a build with known faulty code". – Dominic Cronin – 2015-02-05T11:34:24.890

@DominicCronin: Nothing circular about it. Whatever gets commited to the repository should be the best possible quality. There's a whole bouquet of reasons there -- avoiding artificial change of code lines is one (w.r.t. "svn blame" and similar repository functions). The danger of "forgetting" to take the error out again. The problem that the testers basically could look up what has been "seeded" by looking at the repository change log. Many more reasons, with virtually no benefit to counterbalance. But I'm running out of space, and anyway the idea was to provide one, short reason. – DevSolar – 2015-02-05T11:43:50.190

@DominicCronin: Or, to put it differently -- there might be a case to be made for "seeding" a bug, but the line has to be drawn well before committing that to the repo. And on the other hand, while having "seeded" code for testing might have one or two things going for it, you should only ever test committed code. The two ideas -- each being already controversial on its own -- simply don't connect in any sensible way. – DevSolar – 2015-02-05T11:52:31.170

I recommend against deliberately injecting bugs into EVERY build you send to QA.

You could, from time to time, let's say once a year, do a covert "QA audit". Take a "tested and working" codebase, and as many small new features from your Todo list as possible. Implement them "a bit more sloppily" than you usually do. Think of the edge cases, write them down, but don't fix your code to take them into account. Send it off to QA.

If they find more non-working edge case bugs than you wrote down, it's certainly not your QA that needs supervision... ;-)

Alexander

Posted 2015-01-28T10:56:26.480

Reputation: 148

2this doesn't seem to offer anything substantial over points made and explained in prior 16 answers – gnat – 2015-01-30T12:48:34.473