Is Crowd Testing Really Safe? Here are 7 points an Enterprise must note before going for a Crowd Testing Platform

Crowd Testers

Crowd is the ‘in’ word in today’s highly interconnected equally crowded & world. From crowd funding of startups to crowd sourcing ideas, we have come a long from managing isolated silos of knowledge to harvesting knowledge from across the world, thanks to innovation in technology and emerging openness among corporations to experiment with people outside of their comfort zones

Of late, crowd testing has been an important area where the collective wisdom of the crowd has proven more effective & efficient than in-house testing of software apps. With more software components being exposed directly to clients, quality of software becomes paramount in determining customer loyalty & retention. Software testing today is the critical cog in the wheel of software development which can make or break businesses. Crowdtesting as a methodology that leverages the power of a vibrant independent testing community across demographics ranging from a few dozens to few thousand testers with one passion – Find & fix bugs

More than software components being more client facing, it’s the adoption of trends like cloud computing & BYOD (Bring Your Own Device) that has ensured that software is now consumed across mediums, channels, devices & I.T system configurations. In such a scenario, software has to be compatible across a matrix of configurations. In-house testing would fail in such a case where critical elements may be overseen due to lack of resources & time

With crowdtesting, the app is subjected to real world scenarios in advance which make refactoring a stress free experience. In a well organized crowdtesting environment, usually all bugs are spotted & logged well before app release resulting in a bug free app & a happy customer

Despite these winning points for crowdtesting, there are a few points that an enterprise software firm should consider before hiring a crowdtesting partner with a ready to deploy army of crowd testers

1.     Data sensitivity, authorization & confidentiality

How much of my data is sensitive & can cause severe breach of privacy?

How do I know if the tester isn’t a shady hacker?

How many levels of access does the crowdtesting platform have?

Do I have options to recall & reset the code to pre- test scenarios?

All these questions should be raised with the crowdtesting platform to understand the safety measures they undertake for glitch free delivery

2.     Is their testing process robust?

For enterprise products, the biggest challenge would be to draw a common ground between exploratory testing & inside out approach of crowdtesting. A good crowdtesting platform would rope in people who have extensive experience handling apps with multiple levels of complexity. Crowdtesting methodology is typically modeled on existing systems like Agile, and hence adopting it to a larger enterprise wont be tedious. Compare your internal testing methodology with the approach the crowdtesting platform provides and take a call

3.     Learn from the masters

‘Big bounty’ programs have always been a hit among ace testers. It’s a program where external bug reporters work as your internal security experts and get paid for every critical bug they find.

Researchers at University of California have categorically suggested that it’s worthwhile to hire external crowdtesters to do the job of bug hunting.

The study also suggests that major players like FireFox, HP, Google chrome & Facebook have been actively running crowdtesting programs & rewarding best performers  & that every software company must have a round the year reward program for crowdtesters

“Our vulnerability reward programs have been very successful in helping us fix more bugs and better protect our users, while also strengthening our relationships with security researchers,” – Google security research team

4.     How do I choose my testers from the crowd?

The backbone of any crowd platform is the strong community ecosystem. Watch out if the platform has an active forum where testers share knowledge & rate each others’ work. An active community is also an indication of transparency. Also ensure that the platform itself isn’t dictating community rules & allows the community to have a free hand in sharing content they like

Also, watch out if the platform has a mechanism to rank testers. In a highly competitive world, peer ranking is seen as a real & accurate indicator of one’s capabilities. You can ask the platform to choose crowdtesters based on your requirements too. The more transparent the platform is, the more assured you can be that your work would be done in a hassle free manner

5.     Leveraging based on risk & priority

We recommend crowdtesting for apps that have high Defect exposure factor (DEF). DEF is defined as

DEF = (No of releases per year * Average number of daily critical stakeholder visits.

These stakeholders could be your customers, your customer’s customers whose experience can make or break the app’s sale-ability

6.     Is their pricing flexible?

TaaS – Testing as a service.  Crowdtesting platforms typically operate in the cloud. Their flexibility helps them to be flexible with pricing as well. Go for a platform that has a hybrid pricing model of charging minimal price for accessing the platform, generating reports etc & a performance linked pricing that links payouts to no of bugs delivered

7.     Watch out for new trends

There are niche vendors who are carving a market for themselves in the crowdtesting market. Some venture into specialized testing methods like performance testing. We would see a scenario where crowdtesting would be synonymous with community testing.

Since crowdsourcing is getting crowded, crowdtesting platforms need to stand apart from competition. Watch out for crowdtesting platforms that have any of the following initiatives

  1. Crowdtesting contests or testing hackathons
  2. A vibrant idea sharing community
  3. Platforms that share strong & relevant content that makes you smart
  4. Emergence of social media – Give priority to firms that have active social media groups that act as a great storehouse of knowledge & content


Happy testing!

8 Reasons Why Enterprise Software Testing is best done with Crowd-sourcing Testers

Enterprise Crowd Testing

In a world swarmed with software products & apps, quality can make or break your customers’ user experience. In a fragmented & highly competitive market where customer attention span is minimal, even an insignificant bug can result in downward spiraling of app adoption & huge opportunity losses for the company. Effective & fool proof software testing plays an integral role in acting as a solid base that enable software products to be smart, agile & user friendly.

Experts claim that about 65% of software product companies fail because they weren’t well prepared in terms of quality assurance before the launch

But sadly, QA function in big software companies as of today is about doing too many things with minimal budgets & teams. For many CIOs, QA isn’t always ‘mission critical’ & QAs get step-motherly treatment in the eyes of the decision makers. QA leaders are always in the fire fighting mode & their efforts are spent more in delaying the inevitable (software crash) than actually hunting down & fixing critical bugs

With the evolution of QA from desktop based to cloud, there were assumptions that testing would be simplified. But reality was that these apps needed to be tested across platforms & devices & given the tight timelines & budgets, QA teams weren’t able to cope with this drastic increase in testing matrixes.

Enter crowd testing

“If you are testing software that all kinds of strangers are going to use, then why not use a bunch of strangers to test it.”

The cloud adoption in the past decade also saw the evolution of community driven multi-device & multi-platform testing platforms which had the best testers around the world coming together, collaborating & finding the best solution for your software bugs. Collaborative testing became in the ‘in thing’ with crowd sourced team pulled off the ‘impossible’ that included

  1. Maintaining app quality
  2. Delivering on time
  3. Working around the clock with tight budgets
  4. Broad testing coverage
  5. Transparency in communication
  6. Control & compliance

Instead of hiring 5 in-house testers, you can hire 50 part time testers through crowd sourcing who can produce 300% more work in the same time & with 70% less fatigue & 10 times greater coverage when it comes to devices & platforms

As a CIO who wishes to release bug free apps/products of the highest quality under tight control & budgets, here are 6 reasons why enterprise software testing should be done by a crowd sourced team

      1. Lack of product or app bias

A crowd sourced environment has people who have an open & fresh mindset to approaching apps. A typical in-house team would have people drafted from the product team who might be biased to report bugs because the product is ‘their baby’

Internal teams tend to get hesitant to report bugs during the initial phase of product launch. This may end up spoiling the user experience for the early adopters & the product growth may hit the downward curve thanks to poor word-of-mouth

Crowd testers also boast the knack of testing multiple products which may be similar in nature. This would ensure that the first bugs that an early adopter would encounter would most likely be spotted & fixed by a crowd tester

      2. The power of ramping up & breaking ‘entry barriers’

An in-house testing team simply doesn’t scale. Given tight budgets & lack of autonomy (in some cases), in house testing gets into the rigmarole of blame games & actions that only delay the inevitable event of product failure than solving actual problems.

A crowd sourced team scales up faster as there is already a ready set of knowledgeable testers who have a fairly automated process in approaching software testing. Many SMBs typically face challenges in accessing multiple devices & platforms given their tiny scale. Crowd sourcing helps solve this problem by providing a pool of testers who would be armed with every single hardware-software combination that would ensure testing is water-tight. So instead of testing across 5 devices, you would now be able to test it across 50 devices at fairly the same cost

      3.Control over delivery

It must be reiterated that crowd testing would still be functional under the boundaries drawn by the client. Though testers are free to play around with test cases as much as they want, the final deliverable is a Yes or a NO. You either accept the bug, pay for it or reject the bug & not pay. The client would anyway have the final say

      4. OPS – Organized, Professionalized & Standardized

Crowd testers are a community. They behave like on. They compete against each other to get the best work and are professionals in their own right. Since it’s a self regulating marketplace, the community ensures that the testers behave professionally and maintain highest standards of quality & professional integrity

As with every other software process, crowd testing processes become standardized over a period of time. What works for one client can be replicated with minimal changes for the next client.

      5. Real world conditions & real users – Clean lab setups

Another challenge most companies face is the lack of ‘Clean lab’. A clean lab is an ideal environment which mirrors what the client would be seeing & experiencing. Since its tough for in-house teams find it tough to mirror such a setup, they end up running behind customers ‘after’ the bug is logged. This may not go well with many serious enterprise customers who want bug free softwares at the first launch

Since the testers are customers themselves (in case of consumer apps), they would be in a better position to test & deliver results

      6. Complement, not compete with internal teams & processes

A crowd sourced team should be seen as a ‘testing team on call’ & not as someone who would threaten the in-house teams’ very existence. After all, a crowd sourced team specializes only in testing and cannot replace the wisdom of in-house teams with strong product background

     7. Quality & Quantity of improvements

Our experience says that crowd testing platforms have significantly improved both quality & quantity of improvements reported. A direct reason could be that a group of people, with different approaches would be able to dig & recommend more improvements than a small team of in-house testers

     8. Tapping a wide knowledge bank

Ultimately, communities are known for their wide reach of people & an ever growing appetite for knowledge consumption. Such knowledge should be tapped by every enterprise to get the best out of their testing effort & also ensure that their own testing teams become smart, agile & well prepared to face future issues


Happy testing!

Advice for Software Testers from Bug Validation – By Krishnaveni

Advice for  Software Testers from Bug Validation – Krishnaveni


I recently got an opportunity to validate two contests on

This experience was quite refreshing, since it was like a refresher’s course for me to re-learn and re-visit things that I had learnt when I started out as newbie in Software Testing.

Bug Report is a crucial aspect of software testing. For bugs to be of value, effective bug reporting is a must.

Here are the lessons I learnt yet again.


Scope of Testing

First and foremost consider the scope of testing mentioned. Any bugs not coming under the given scope will be deemed invalid.


Bug Logging Guidelines

If the client or the contest owner has stated specific guidelines to log a bug, make sure to go through it before logging any of the bugs. If the tester fails to read them, chances are high that the bugs reported by the tester could lack in key information or bug format the client is expecting to get. Be attentive of what is expected of you.


Avoiding Duplicates

Before logging a bug, consider if a similar bug has already been raised by the others. Someone may have already reported the bug. Searching first helps to prevent duplicates. To be effective, try multiple synonyms and rephrasing of what the bug might have been called.

By not looking out for duplicates, the tester will lose out on the chance to log another valid bug. The known drawback is redundancy when there are many duplicates; however the impact is higher for a tester when the bug number is limited per tester in a contest.

Bugs rejected for being a duplicate will bring down the credibility of a tester.


Assigning Severity

Testers should assign the right severity for a bug. Clear understanding should be there to distinguish between a show stopper issue and a high issue.

I observed that many testers do not know this. Bugs like a field not accepting the input or a drop down not populating values are marked as show stoppers. Similarly some UI issues too are termed the same.

Show stopper bugs are those, which prevent the user from further accessing the application owing to an issue. In the examples stated above, the user can still access other areas of the application.


Bug Type

Be clear about what kind of bug it is. Is it a functional, GUI or a technical bug? Incorrect marking of bug types is a sure shot way to have the bug marked as invalid.


Managing Timeline

In order to accommodate more bugs within the specified testing timeframe, most testers log bugs in huge numbers without concentrating on how they are logging it. In this hurry, they just enter a one liner bug and submit it. They fail to realize that doing this way won’t fetch them any credibility. Being testers, one should never compromise on the quality. With practice, one will be able to learn how to manage the time.

Bug report should be complete inclusive of Bug description (one liner), steps to reproduce, expected results, actual results, screenshot or video and the environment details on which the testing was carried out. If there is a time limit allotted to test, that cannot be an excuse to log incomplete bugs. This shortcut will eventually lead to bugs being marked invalid or rejected owing to lack of details


Quality v/s quantity

Quality of the bugs is most important that the quantity of the bugs.  It serves no purpose to just keep filing bugs blindly.  Though it may appear that the tester is logging in many bugs, the credibility would definitely come down since the bugs would be lacking in content and quality.


Choice of Browsers

Ensure not to test on out dated or non-compatible browsers. It was surprising to see testers having tested the application on IE 6.


Environment Details

This information is one of the key aspects in a bug report that would aid in reproducing the issue.

Many testers fail to include the same while reporting the bug. This makes it very tedious for the client or the bug validator to figure out in which environment setup the issue occurred.


Bug Description

Bug descriptions are a sneak peak into what the bug is all about. They need to be short, crisp and in a line hinting on what the bug is all about. Inappropriate bug descriptions that don’t convey the right message might not catch the attention of the client or validator.


Communication Skills

Effective communication skills with a good proficiency in English are a must.


Spelling Mistakes

Before submitting the bug, do a proof reading. Check for spell errors. It is quite pathetic to see that as testers who should be finding the flaws, they themselves are unable to see the flaws in their own work.

Example: Sing in page not working

What the tester here meant to write was “Sign in page is not working”. There is a sea of difference between Sing & Sign. Hence care should be take to avoid spell errors.


Usage of short forms of words

Bugs reports are formal communications that help the client to know the issues in their product or application. So as per e-mail writing etiquette, no short forms of words must be used in a formal communication.


Example: One tester had written, the reset password page is not aval.


The tester had used the short form ‘aval’ for ‘available’. This is not a good practice.


Expecting the clients or validators to do mind reading

Don’t expect the client or validators to put their mind to figure out the entire issue just by logging in a bug with least information.



(a) Section ‘car’ accepts invalid inputs. The missing things in this are where is section car?

which page of section car(if the ‘car’ section is occurring in multiple places)


(b) Field is not accepting input in that form (which form?) How is the client

or stakeholder supposed to figure out which form in the entire application the tester is talking



All about screenshots

  • Do not save screenshots in .bmp format since they tend to occupy a very large space. This might irk the client since the screenshots might take a very long time to download.


  • When multiple screenshots are included in a word document, it is a good practice to include a one liner description above each image so that it helps the client to understand what that image is about.


  • When the tester mentions to refer the screenshot in the bug report, the tester should ensure that the screenshot is attached in the bug. Many a times it is seen that the tester forgets to include the screenshot attachment.


  • Supposing a particular functionality works fine in one browser and not in another, it is a good practice to attach screenshots for both: browser where it is working and browser where it is not working.


Do not live with the bug

At times testers might have a tendency to ignore a bug if it is a very very trivial GUI issue. They would be so focused reporting major stuff that this might get missed out.


It is good practice not to miss out such bugs based on their severity.



In the particular application where I was validating the bugs, the logout button did not have a tooltip. The icon was appearing such that the very appearance was like a refresh button. It was a good thought of the tester to log a low GUI defect highlighting the same.


Bug Report Contents

A bug report should essentially contain the proper steps to reproduce, expected result, actual result, environment details and screenshots.


Steps to reproduce should start off with the mention of the URL that was tested. In case the application navigates to a different URL owing to an issue, it is worthy to mention that URL in the bug report.


Avoiding invalid bugs

Before reporting bugs the testers should reproduce it themselves to confirm the same.

Also, when ruling out that the error messages are not fired for an invalid input, the testers should ensure that the form/page is submitted in order to confirm if the issue still exists.


Key characteristic trait for a tester

Learnability is an important characteristic trait for people who wish to better themselves.

If the validators have commented on the bugs with tips or suggestions for improvement, the testers should consider them in a positive stride and use them constructively to improve their  skills.


Further links to refer:





  • Bug Reporting etiquette from bugzilla:







The Open Bug Experience by Krishnaveni

Guest Post: By Krishnaveni, one of the Top Testers at 99tests

The Open Bug Experience

 Open Bug


A tester to some extent by nature of practice can effortlessly spot a flaw in a product or an application. And when this happens, it is very intriguing to voice out the same. But the question that bothers is – would the issue that was voiced out by the tester reach the right contact or not for the issue to be resolved?

A tester knows the impact, risk and consequence of a bug when it is present in a product or an application that is massively used by the public.

How about having a platform to voice the bugs that we come across in our daily activities, be sure that they reach the right contacts and be rewarded for the same too? Too good to resist isn’t it??

The recently launched “Open Bug” feature from 99tests facilities this.


About Open Bug

Open Bug is a very innovative initiative by 99tests, wherein the testers registered on 99tests can log bugs they come across. The bugs that are creative and worthy can be voted through which a tester is rewarded.

My Experience with Open Bug

The approach of Open Bug kindled my curiosity to give it a try.

Unlike the conventional process, wherein, we are accustomed to test a suggested app with a given mission, the interesting factor here is that a tester can choose any app and log the bugs. 99tests has always been a perfect platform for me to learn new things. Needless to say, I was sure that this open bug feature would definitely have more to offer me to learn.

What I learnt

Lesson #1

The first thing I learnt was to leverage my skills I had, testing a particular genre of website (e-commerce, travel etc.,) and looked for apps belonging to that genre.

Lesson #2

Secondly, I learnt to adapt my bug filing style based on the context for the open bug feature. Unlike the conventional ways, if the context was to just test a single app and log all the bugs for that app, then it would have made sense to file each bug individually. But the objective of the open bug feature was to test any app and expose the issues in them.

In order to accommodate this objective, I grouped the bugs based on its quality characteristic like usability, UI etc., and logged them as a single bug. However this would not apply in every case. Based on the need, some bugs like those of functionality had to be logged separately.

Lesson #3

I was encouraged to try out the different genre of websites that I hadn’t tried before.

Lesson #4

I got a couple of lessons from my mistakes. To err is human they say and I was no exception. And I committed mistakes too. Honestly it was an irony for me to realize that though I ventured to figure out flaws in other websites & products, I failed to ensure the same in what I was offering.

I have to admit that it was purely by oversight and I am not ashamed to admit it. I learnt what mistake I made and shall take care to overcome them going forward. And how did I realize it? Nah! I did not figure them out myself. Again by oversight. It was by other testers who viewed my bug and shared their comments on the same.

Guess at times one tends to have an oversight that they fail to notice the mistakes they make. Here are the mistakes I made.

Mistake #1: I had incorrectly keyed in the name of the website I tested.

Here, the bugs logged for the website were worthy, but how could one evaluate them when an incorrect website name is specified?

Mistake #2: For one bug, I had inadvertently included the link which was a report generated for another website.

Now though the bug was meant for one website and was valid, the incorrect report link I included for the same rendered the bug invalid.

Lesson #5

The criteria in the open bug feature were to log a minimum of 20 bugs. So I went on a bug hunting spree and started to log the bugs I unearthed.

But after I reached a good number, I paused by and pondered; if doing this way really served the purpose the open bug feature was meant for? This is not a contest wherein I can just keep filing all the bugs of just one app at a go.

Now a question arose in my mind – what catches more attention of business owners to accept their app to be tested? Is it the few bugs that occur in some parts of their app or some major severity issues for them to be convinced to go in for a testing effort?

This thought has inspired me to focus on high priority issues that affect a business than just a technical bug. So this would mean that it’s important that one needs to log most critical bugs first.

It is motivating that any new thing attempted has many things to offer to learn.So this is an on going process testing – learning and improvising.Come give Open Bug feature a try and experience it for yourself. It’s worth every effort and time spent on it.

Research Paper – Effect of Crowd Size in Software Testing

Source: More Testers – The Effect of Crowd Size and Time Restriction in Software Testing

Crowd Testing Team Size

Crowd Testing Team Size

We at 99tests have seen more than 23,000 bugs being logged by over 5000 testers in the past two year. Here are some of key finding we have uncovered over the past couple of year. Specially the effect of a Crowd Testing team on finding high quality bugs in a short time.

When does the performance of a Crowd Testing team over take the performance of a normal QA team?

We have seen that a Crowd Testing team has to be composed of a minimum of 20 testers of diverse backgrounds to see the positive effects of the Wisdom of Crowds. Now, in the book Wisdom of Crowds by James Surowiecki states that for a group of diverse individuals to come to an accurate prediction or decision, four conditions that need to be met, first is diversity of the crowd, the second is independence in thinking,  third is decentralization and the fourth is aggregation.

Now coming back to Software Testing, one of the inherent issues with in house QA teams, is familiarity with the product, this means that since they have seen the product evolve over time, they become resistant to looking at the product with a fresh set of eyes, so traditional QA teams are very good at executing test cases and letting the developers know what works. This is where a Crowd Testing team out performs in finding bugs, that a new user would find. The second key trait of Crowd Testing Teams is the team size. Once the number of testers is two to three time the size of normal QA team, we start seeing the kinds of bugs which only a crowd team can produce. The third key point in forming a crowd testing team, is the diversity of the testers, i.e. the team of 20-30 testers needs to be composed of very highly skilled exploratory testers, average testers and novice testers. This team dynamic ensures that a high bar is set for the quality of the bugs by top testers and the team follows their lead in logging high quality bugs. The third key trait for good crowd testing teams is time pressure, so having a fixed time in which to find the maximum number of bugs is important condition for getting the best results. The fourth key trait is the incentives or rewards need to matched to the testers who find the most number and high quality bugs.


Pre-print version (open access)