A New Test Run Hub Is Not a Test Strategy
Recently I was asked whether a better Test Run Hub in Azure DevOps would improve UAT.
I think it will. I do however think people can expect rather too much from the tooling.
A dashboard can tell me a test passed. What I really want to know is what we proved by passing it.
Take something as simple as employee dates.
A test might pass because the date appeared in the new system. Nobody checked whether the format was UK rather than US, so 06/10 is suddenly June rather than October.
A calamity waiting to befoul us.

The test passed. The outcome is wrong.
That is my gripe with headline pass rates.
196 tests passing out of 200 means very little until I understand the other 4. If those failures concern payroll calculations, integrations or permissions, I am potentially more interested in the 4 than the 196.
For me, good UAT comes down to 3 questions:
What were we trying to prove?
What evidence proves it?
Who accepts the remaining risk?
Azure DevOps can organise the test execution, results, evidence and defects. A controlled HR/payroll UAT model then needs to carry that evidence through to the go-live decision.

Quick UAT check: Take your 3 highest-risk processes and trace each from requirement to test, evidence and decision owner. If you cannot complete that chain, I would fix that before worrying about the overall pass rate.
I like anything that makes evidence easier to find and failed tests easier to follow.
I do however see the Test Run Hub as supporting the testing process. Good requirements, meaningful tests and clear ownership still determine whether that evidence is useful.
A better view of weak testing is still weak testing.

Comments