Prioritization: what to run first when time is short
level 2 · 20 min · test case prioritization
Prioritizing is deciding the order in which tests run so that what matters most runs first. The easiest order, the one on the list, ignores that some tests are worth far more than others.
There are three classic ways to order: by risk (probability of failing times impact), by coverage (the test that adds the most) and by the priority the business gives the requirement. On top of them, dependencies rule: a test that unblocks others comes before the ones that depend on it.
Rule of thumb. Order the queue by the criterion that reduces risk the most: what costs most if it fails, what unblocks other tests, what covers most and what the business demands. Whatever is left goes later, with the decision recorded.
Read the situation
Running the tests in list order is the easiest way and the worst: what matters most may be left until after the deadline.
The release ships tomorrow and there is time to run only half of the tests.
There are 40 test cases and the queue is in alphabetical order. Whoever runs in today’s order reaches the end of the day without having tested payment.
Questions to order the queue
- Which failure would cost the business more, and which is more likely to happen?
- Where did the code change in this release, or where has it had the most defects?
- Does any test only make sense if another one passes first?
- Does each test add coverage or repeat what another already does?
Exercise: decide the order
For each situation, choose what to run first (in the last one, what to postpone). The criteria are risk, dependencies, coverage and business priority.
Choose an option for each situation.