This broadly looks fine, we're just trying to figure out the best approach for naming and grouping the results. I generally don't like the idea of using the wiki test case names: it makes sense for a world where the wiki test cases are our kind of 'canonical' reference for *all Fedora testing ever*, and presumably we'd require any system that reported into resultsdb to do the same kind of conversion, but I just don't think that's going to hold. We're not naming the rpm-validity tests after wiki test cases, after all. So I think we should just use the openQA test suite names.

@garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames and I talked it over, and I think we broadly agree on this scheme:

• We'll submit one result per openQA job (for now - possibility to report one result per openQA *test module* exists, but may be tricky)
• We won't include the test module info in the extradata for now as it's quite hard to use without talking to openQA in any case
• The result name will be based on the openQA test suite name, with appropriate transformation for resultsdb's conventions, and .uefi appended for UEFI tests
• We will have one big group named for the compose (build) into which all the results will go, and then groups with identifiers 'compose subvariant imagetype arch' into which relevant results go

The 'uefi' thing is a bit tricky. The 'generic' way to look this up is job['settings']['MACHINE'], a non-generic (and potentially Fedora-specific) way is job['settings']['UEFI']. Appending a machine identifier when needed but *only* when needed is the tricky bit, if you don't just hardcode if uefi or whatever; I think you can check whether MACHINE and arch differ, and append the identifier only if so, but make sure that's not the case for 32-bit x86 either (where you often run into fun with one thing saying 'i386' and another saying 'i686').

We should definitely do some trial runs of this first and discuss with @jskladan at least (and probably @mkrizek and @tflink , maybe even Dennis and/or @ralph), before putting into production, as we're kind of inventing some pretty important organizational principles on the fly here.

OK, after talking to Dennis, new plan!

The testcase stays as-is. The item will be the deliverable file name if there is an associated deliverable, otherwise (for tests that aren't associated with a specific deliverable) it will be the compose ID (build). There will be a group named for the compose ID with all results in it (as planned before).

If it's technically OK for resultsdb - we don't know if groups with a huge number of members are a problem - there will be smaller groups named for deliverable identifiers. By 'deliverable identifier' we mean something like check-compose uses, e.g. Kde live x86_64. This is something Dennis asked for - it will be valuable to be able to say 'just give me all the results for this image for all composes', so you can look for the most recent one that passed all the tests or whatever (fedora_nightlies does something very much like this, FWIW, so I have that use case too).

The check-compose identifier is constructed from subvariant, imagetype and arch. For this case it would probably be a good idea to add format, like Kde live iso x86_64, to try and get the best guarantee we can that the identifier will be unique. I don't think there's a policy for what's guaranteed to be a unique identifier, I just yell at the productmd devs when they screw up the ones I've chosen. Note: both - and _ occur in the values that go into these identifiers, so either of those is a bit tricky as a separator if we care much about letting people parse them. I think . would be a safe separator, but I haven't checked for sure.

Also, just a technical detail, @garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames correctly notes that the 'append MACHINE if it's different from ARCH' plan won't work, they actually differ in most cases. We're just going with the dumb, case-specific 'append .uefi if UEFI' approach for now, we can improve it later if there ever actually are other test scenarios like this.

A thought I just had: openQA has a 'test scenario' concept, which is something like 'all the information needed to identify a specific test across composes' - like, the same 'colored dot' on the overview page for two different composes. I'm pretty sure we don't want to expose that as the test name or the item or anything, but we should at least make sure that all the keys defined as 'scenario keys' (it's a couple of constants in the perl code, one including MACHINE and one not, and I have the definitions shadowed in openQA-python-client) are in the extradata. We may in future want to add a representation of the scenario as another grouping if we have use cases for it.

@jskladan says tests can actually have *multiple* items, which might be useful and let us refine the model a bit more. Let's talk it over with him tomorrow (if we can keep him out of meeting hell).

In D1102#20493, @adamwill wrote:

• The result name will be based on the openQA test suite name, with appropriate transformation for resultsdb's conventions, and .uefi appended for UEFI tests

Just to be sure - adding a .uefi to the testcase name makes more sense for you guys than adding boot=uefi or uefi=1 (or something in the likes of that) in to the result's extra-data? Not that it is a huge difference, just that when querying for results by testcase name (not sure whether it's the use-case for you) .../results?testcases=foobar wont' return results for foobar.uefi, but you sure could do .../results?testcases:like=foobar.* to get that, but it will return not only foobar.uefi but potentially also stuff like foobar.moo.
Whereas if the uefi part was in the extra data, you could .../results?testcases=foobar to get all the foobar results, and .../results?testcases=foobar&uefi=1 to get only the uefi ones.

Depends on what way you are going to consume it.

From what I'm seeing lately, I'll need to make the search filters much more capable, so this might be a non-issue in the future. Just let me know if you find something that would be usefull, and resutlsdb seems to not do it yet.

In D1102#20495, @adamwill wrote:

If it's technically OK for resultsdb - we don't know if groups with a huge number of members are a problem

You can have as many groups as you want, groups can have as many results as you want, and results can be in as many groups as you want. There is no arbitrary limit, and I'm not aware of any technical difficulty caused by the heavy usage of groups.

@ralph2 tagging you, as requested, just in case.

@garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames note: there's some vestigial resultsdb code left in scheduler which I can think can just go completely now (imports, a function arg, and a bit of the function docstring).

So I just proposed D1105 as a mechanism for abstracting the conventions of similar result scenarios between different reporting mechanisms. If we agree that's a good idea, we should tweak this to use that. It'll make the code much simpler.

The way I was expecting it to be used is for this module to subclass the most appropriate class from resultsdb_conventions and applying whatever additional bits of metadata, group memberships, system for creating the testcase name and link etc. are desired, but I haven't tried it yet so I haven't figured out the quirks. I'll try it for Autocloud and/or relval soon.

This looks great, me likey. Just going to try and hack up a way to test it out and will then ack.