handle 'softfailed' as a result (for now, treat as passed)
ClosedPublic

Authored by adamwill on Aug 26 2016, 11:52 PM.

Details

Summary

openQA upstream has changed to use 'softfailed' as an actual
result state which appears in the job dict returned by the API
(previously, soft failed jobs were 'passed' so far as the API
was concerned, and the web UI inferred the 'soft failed' state
from the test module results). So we should handle this. In
Fedora usage, so far, 'softfailed' tests are basically treated
like passes. Both the overall test result and each test module
result can be 'softfailed', so handle both these.

As a future improvement we might want to consider reporting
'warn' rather than 'pass' for these, and maybe even linking
workarounds to bug numbers so we can include a bug number in
the 'warn' result template.

Test Plan

Somehow generate a 'softfailed' test - you'll need
latest git openQA, there's only a scratch build for now - and
check that it gets reported as a pass. But really, this is a
pretty bleeding obvious change, I think you can just ack it
without testing. :) You can see an example of such a result
from openSUSE at
https://openqa.opensuse.org/api/v1/jobs?ids=249959 - that one
has both the overall result and some modules as 'softfailed'.

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
adamwill retitled this revision from to handle 'softfailed' as a result (for now, treat as passed).Aug 26 2016, 11:52 PM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: jskladan, garretraziel.
garretraziel accepted this revision.Aug 30 2016, 12:21 PM
This revision is now accepted and ready to land.Aug 30 2016, 12:21 PM
This revision was automatically updated to reflect the committed changes.