testcase_stats: handle test cases with multiple milestones
AbandonedPublic

Authored by adamwill on Jun 1 2016, 11:45 AM.

Details

Reviewers
kparal
Summary

If the same test case appears in several tables with different
milestones, display the most important one in the stats output, instead
of showing the last one encountered.

This was mostly visible in the "Default boot and install" section in Installation matrix.

Test Plan

tried generating F24 installation matrix, problem fixed

Diff Detail

Repository
rRELVAL relval
Branch
feature/milestone_prio
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 640
Build 640: arc lint + arc unit
kparal retitled this revision from to testcase_stats: handle test cases with multiple milestones.Jun 1 2016, 11:45 AM
kparal updated this object.
kparal edited the test plan for this revision. (Show Details)
kparal added a reviewer: adamwill.

Thanks for this! I've been meaning to do something about the problem but haven't had the round tuits. I'm not sure if this is enough, or if we need to do more; the actual counting of the test results here may not be what we want any more, I think we don't really want the rows for the same test case with different arch and priority to be combined...

kparal added a comment.Jun 2 2016, 8:30 AM

I agree it's not ideal. But changing that will probably require us to considerably change how to display the results. For example we could mirror the matrix structure from the wiki, so for each separate table on wiki we would produce the same stats table, with exactly the same headers and same test case names. Maybe you have a better idea.

In any case, I think the current patch improves the situation, i.e. makes it less incorrect. So I'd merge it until we come up with a better solution.

yeah, if I can't come up with anything else, I'll merge this. but I have some ideas.

Now that we're in the Beta cycle, this is getting slightly more important. Ehm ehm.

So I finally got back around to this. I think the way I'm going to handle it is to put these two tables of duplicated-for-i386 tests into their own sections. This will cause wikitcms to distinguish them better.

There's a couple of advantages to this approach:

  • It fixes relval report-results as well (currently it's a bit hard to tell why there are two of each test, in the report-results interface)
  • It'll make testcase-stats treat them as separate 'test instances' instead of combining them, which I think is better, as we don't really want to think these tests are 'covered' if only the i386 ones have been done, or 'not fully covered' if i386 has not been done

The only drawback is that I now get to hack up a temporary script to edit every single Installation results page since we changed this. le sigh. :)

adamwill commandeered this revision.Aug 30 2016, 9:41 PM
adamwill edited reviewers, added: kparal; removed: adamwill.

OK, this should be effectively addressed now by the page edits: the i386 results now show up separately in testcase_stats , and all have the appropriate milestones.

sorry not to use your code, it's nicely done for sure, but I don't think generally appropriate (usually, if a test has different milestones in different pages, we should prefer the milestone from the later pages, not whichever milestone is 'highest priority', and that's what the code does as written, since it always parses the pages in date order).

adamwill abandoned this revision.Aug 30 2016, 9:42 PM