Add resultsdb_conventions, module for sharing result conventions
ClosedPublic

Authored by adamwill on Feb 1 2017, 11:17 AM.

Details

Summary

This module provides a representation of some result reporting
conventions for ResultsDB (the ones we've just invented in the
last two days, so far). It's a class-based design - currently
hierarchical, but I suspect mixins might become a thing - and
the idea is that you can subclass (or even just use directly)
the class that most closely reflects the situation you want to
report a result for, and you get a lot of 'standard' stuff -
commonly-reported extradata and group memberships, mostly -
handled for you. This saves us boilerplating stuff around
between reporters for systems that do similar tests (e.g.
openQA, Autocloud and relval for compose tests) and ensures
they all use the same conventions for extradata naming, group
identification etc.

It's pretty easy to use, try this:

import logging
logging.basicConfig(level=logging.DEBUG)
import resultsdb_conventions
res = resultsdb_conventions.ImageResult(

'Fedora-Rawhide-20170131.n.1',
'Fedora-Server-dvd-x86_64-Rawhide-20170131.n.1.iso',
'compose.some.test',
'PASSED',
'note here',
'https://www.test.case/',
'https://www.job.link/'

)
res.report(None)

The argument to report is the ResultsDBapi instance, if you
set it to None, no reporting is done (but the attributes of the
result are logged at DEBUG loglevel).

Test Plan

Try it out! For now I've just used the result printing
as I don't have a local ResultsDB yet, will set one up now for
testing.

Diff Detail

Repository
rRSAPI resultsdb_api
Branch
conventions (branched from develop)
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 1002
Build 1002: arc lint + arc unit
adamwill retitled this revision from to Add resultsdb_conventions, module for sharing result conventions.Feb 1 2017, 11:17 AM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: jsedlak, jskladan, tflink, mkrizek.
adamwill added a comment.EditedFeb 1 2017, 11:41 AM

Notes: the naming conventions are still slightly up in the air, I'm downcasing some things but not others, do people have opinions on whether we should always downcase or not?

I'm intending to write tests for this next, it obviously should have some. And probably extend the validation a bit.

If you're questioning why the subclasses do some extradata stuff *right in __init__*, so it's just not skippable at all, those are things I think we're just about *always* going to want. Of course a subclass could just delete or change them if really desired.

Dunno if this is a bit too much inside baseball for @ralph @ralph2 , but tagging him anyway just in case. We'll have reporters hooked up to this RSN if you'd rather just look at the submitted data and see what you think.

adamwill updated this revision to Diff 2810.Feb 1 2017, 11:50 AM

Change license to GPLv2+ to match the rest of resultsdb_api

adamwill updated this revision to Diff 2811.Feb 1 2017, 11:58 AM

Add to setup.py (as separate module for now, HANDWAVE HANDWAVE)

I had a few nitpicks, but overall this looks good.

resultsdb_conventions.py
72–75

You could probably use regex for this.

86–89
if 'item' not in self.extradata:
    raise ValidationError('extradata', self.extradata, "is not a dict with 'item' key")
157–161

Again use 'ref_url' in rel?

187–188

I think that it would be better to put this on one line (120 lines anyone?). Also, document this, please.

jskladan accepted this revision.EditedFeb 1 2017, 12:53 PM

Like it overall, just a small thing in the comments. Acking, as I'm OK with it overall, but I'd like to have this (plus @jsedlak's comments) sorted out before the merge.
Thanks!

resultsdb_conventions.py
52

I know, and understand, why you split this up into testcase and tc_link, but I'd rather go with either of

  1. have testcase_name + testcase_url (or tc_name+tc_url)
  2. make the testcase param similarly 'smart' as it is in resultsdb, so it can either take string (and use it as testcase name) or a dict with name + ref_url keys in that.

Thoughts?

56

I'd rather stick with ref_url, at least in the general Result class, to mirror the "standard" ResultsDB stuff a bit more.

This revision is now accepted and ready to land.Feb 1 2017, 12:53 PM

Validation fails for test called compose.install_usbboot (it contains non-ASCII/dot characters) and I can't remember if we decided that test can be called install_usbboot or not...

adamwill updated this revision to Diff 2816.Feb 1 2017, 4:58 PM

Various tweaks and improvements

  • Add some help for creating groups (all the UUID stuff) This maybe could move to the API library?
  • Handle a source name, creating relevant groups in each class
  • Handle note properly
  • Kill the stupid validation stuff
  • Have default_extradata methods not override existing values
  • Rename ImageResult to FedoraImageResult as it's quite Fedora-y
adamwill updated this revision to Diff 2817.Feb 1 2017, 11:41 PM

Address review concerns

adamwill updated this revision to Diff 2818.Feb 1 2017, 11:54 PM

Add disc_number to image identifier

Per @lsedlar in https://pagure.io/pungi/issue/525 , disc_number
is needed for a truly unique image identifier.

Like it overall, just a small thing in the comments. Acking, as I'm OK with it overall, but I'd like to have this sorted out before the merge.
Thanks!

resultsdb_conventions.py
76

For the libtaskotron workflow (at least how it is now), we already have a group uuid created (by execdb) when reporting. It would be usefull to be able to either pass the uuid to the constructor directly.

Looking at the method header, I'd also like the order of the params changed so it is as close to the create_result as possible (def create_result(self, outcome, testcase, groups=None, note=None, ref_url=None, **data): for reference) so we are consistent. I know how much I want to tear my hair out when the helper methods for calling some underlying function unnecessarily change the param order.

I suggest result, tc_name, note, ref_url, tc_url, source, or result, tc_name, group_uuid, note, ref_url, tc_url, source if we decide to add the possibility to add the group directly.

I'd also make a case for renaming result to outcome - once again to be more consistent with the underlying api.

114

I know I came up with the names in the first place, but seeing this, it would make more sense to me to have group_name instead of group. Sorry for the trouble.

120–121

Makes sense to have it here. The resultsdb api is just a "stupid" mapping for the underlying RESTful API, and I'm trying to keep it as close to it as possible. Thus the resultsdb_api can take either list of strings and/or objects on input, and just plain passes that to the actual API.

jskladan requested changes to this revision.Feb 2 2017, 7:52 AM

See my previous comment. Changing away of the 'accepted' state, so the discussion is not incidentally skipped. Still like it overall, though!

This revision now requires changes to proceed.Feb 2 2017, 7:52 AM
adamwill updated this revision to Diff 2821.Feb 2 2017, 9:39 AM

Address review concerns, remove FIXMEs, add modeline and pylintrc

If we're using 120 line length, let's say so!

adamwill updated this revision to Diff 2822.Feb 2 2017, 9:47 AM

Pass the note correctly when submitting the result

kparal added a subscriber: kparal.Feb 2 2017, 10:35 AM
kparal added inline comments.
.pylintrc
2

As a minor nitpick, we have this set to 99, in case you want to have the conventions synchronized between projects :-)

jskladan accepted this revision.Feb 2 2017, 10:44 AM

Love it, thanks! Although we just decided that you will be flying the stealth mode on this one too, so... Who am I to say anything, really :D

Macro hypno:  All power to fedfind!

This revision is now accepted and ready to land.Feb 2 2017, 10:44 AM
jsedlak accepted this revision.Feb 2 2017, 12:30 PM

Because @tflink is some kind of goddamn wuss and doesn't want lovely, lovely fedfind in his crappy project, we made this into a new project . So now I get to commit without all you darn kids nitpicking my freaking naming schemes. MUAHAHAHAHAHAHA

oh and it has tests and docs and stuff now, like it's a real project!

.pylintrc
2

well, er, there are far longer lines in the API library, and @jsedlak asked me to unwrap a line that wound up being over 100 characters, saying that he likes 120 character lines, so that's why...

resultsdb_conventions.py
52

I'm fine with whatever naming you like; I *think* I just adopted the names @jsedlak was using in the openqa consumer, in fact, it was nothing very thought through. I'll change this. I'm not sure about the 'smart' dict, it feels like something left over from when you didn't have a URL but didn't want to change a function signature...I kinda prefer the more 'straightforward' approach here (even though pylint complains about too many args :>), so I'll just rename the args I think.

56

Sure, can do.

72–75

for the record, I was trying to avoid using regexes unnecessarily (AIUI they're a lot slower than this kind of thing), but I threw out this crap anyway so it's fine. :)

157–161

Not quite sure what you mean, but the reason we have to handle an exception here is because get_release can raise an error if it fails, and we want to handle that (though I'm not quite sure if we should maybe actually bail in that case after all...hm...)

187–188

oh, it's just a convenient way to get the first item without bothering to generate a list and index it - next(generator_expression) will get you the first item from the expression, unless there are no results, in which case you get StopIteration.

I usually use 100 character lines, but I'm happy to follow whatever the convention for the API module is...it seems to go up to at least 150, so I'll unwrap it :)

adamwill closed this revision.Feb 2 2017, 12:55 PM

So now I get to commit without all you darn kids nitpicking my freaking naming schemes. MUAHAHAHAHAHAHA

Macro devops: The lack of devops is killing kittens