Prioritization: what to run first when time is short

level 2 · 20 min · test case prioritization

Prioritizing is deciding the order in which tests run so that what matters most runs first. The easiest order, the one on the list, ignores that some tests are worth far more than others.

There are three classic ways to order: by risk (probability of failing times impact), by coverage (the test that adds the most) and by the priority the business gives the requirement. On top of them, dependencies rule: a test that unblocks others comes before the ones that depend on it.

Rule of thumb. Order the queue by the criterion that reduces risk the most: what costs most if it fails, what unblocks other tests, what covers most and what the business demands. Whatever is left goes later, with the decision recorded.

Read the situation

Running the tests in list order is the easiest way and the worst: what matters most may be left until after the deadline.

The release ships tomorrow and there is time to run only half of the tests.

There are 40 test cases and the queue is in alphabetical order. Whoever runs in today’s order reaches the end of the day without having tested payment.

Questions to order the queue

  • Which failure would cost the business more, and which is more likely to happen?
  • Where did the code change in this release, or where has it had the most defects?
  • Does any test only make sense if another one passes first?
  • Does each test add coverage or repeat what another already does?

Exercise: decide the order

For each situation, choose what to run first (in the last one, what to postpone). The criteria are risk, dependencies, coverage and business priority.

RiskWhich to run first: what costs most if it fails and is most likely to fail?
DependencyChecking out, changing the password and viewing orders require being logged in. What comes first?
CoverageThree payment cases compete for a turn. Which to run first to gain the most coverage?
Business priorityTomorrow’s campaign depends on the launch coupon. What to run before it?
Defect historyShipping, profile and reports are in the queue. Which to test first?
What to postponeTime is up. Which case is left for the next cycle, with the risk recorded?

Choose an option for each situation.

References

  • ISTQB® Certified Tester Foundation Level v4.0, section 5.1, Test Planning. © ISTQB®.