modify the code to validate EPEL packages(fixed the bug reported in https://phab.qadevel.cloud.fedoraproject.org/T769)
Details
- Reviewers
lbrabec kparal lnie - Maniphest Tasks
- T769: rpm_utils.get_dist_tag() doesn't understand EPEL dist tag
just run the common rpmlint task with EPEL packages as the item
Diff Detail
- Repository
- rLTRN libtaskotron
- Branch
- epel-disttag
- Lint
Lint OK - Unit
Unit Test Errors - Build Status
Buildable 738 Build 738: arc lint + arc unit
Although running rpmlint works, the code is faulty. Note that running rpmlint does not even execute the code changed in the diff, so there is no surprise that it runs OK, even three times in a row.
Running a task that actually uses the code will show you the issue. Consider a minimal example like this:
name: testD944 input: args: - koji_build actions: - name: test download_koji_changes koji: action: download_latest_stable koji_build: ${koji_build} arch: ['all'] src: True
According to @mkrizek the distgit directive will probably be removed, as it's not being used at all.
Also, please add/fix unittests, failing unittests is a reason to nack the diff on its own.
actions:
- name: download rpms from koji
koji:
action: download_latest_stable
koji_bodhi: ${koji_bodhi}
arch: ['all']
src: True[libtaskotron] 08:43:17 INFO Execution started at: 2016-07-20 08:43:17 UTC
[libtaskotron] 08:43:17 DEBUG Using libtaskotron 0.4.13
[libtaskotron] 08:43:17 DEBUG Parsed arguments: Namespace(arch=['x86_64'], debug=True, disksize=None, item='SuperLUMT-3.1.0-5.el7', jobid='-1', libvirt=False, local=False, no_destroy=False, override=[], patch=None, ssh=None, ssh_privkey=None, task='/home/lnie/task-rpmlint/runtask1.yml', type='koji_build', uuid='20160720_084317_892137')
[libtaskotron] 08:43:17 DEBUG Using config file: /etc/taskotron/taskotron.yaml
[libtaskotron] 08:43:17 DEBUG Using config profile: development
{'task': '/home/lnie/task-rpmlint/runtask1.yml', 'no_destroy': False, 'uuid': '20160720_084317_892137', 'arch': ['x86_64'], 'local': False, 'artifactsdir': '/var/lib/taskotron/artifacts/20160720_084317_892137', 'patch': None, 'debug': True, 'jobid': '-1', 'item': 'SuperLUMT-3.1.0-5.el7', 'koji_build': 'SuperLUMT-3.1.0-5.el7', 'ssh_privkey': None, 'ssh': None, 'override': {}, 'disksize': None, 'libvirt': False, 'type': 'koji_build', 'koji_bodhi': 'SuperLUMT-3.1.0-5.el7', '_orig_args': {'task': '/home/lnie/task-rpmlint/runtask1.yml', 'no_destroy': False, 'uuid': '20160720_084317_892137', 'type': 'koji_build', 'local': False, 'jobid': '-1', 'item': 'SuperLUMT-3.1.0-5.el7', 'ssh_privkey': None, 'ssh': None, 'libvirt': False, 'override': [], 'debug': True, 'disksize': None, 'patch': None, 'arch': ['x86_64']}}
[libtaskotron:logger.py:184] 2016-07-20 08:43:17 DEBUG Stream logging enabled with level: DEBUG
[libtaskotron:overlord.py:86] 2016-07-20 08:43:17 INFO Task artifacts will be saved in: /var/lib/taskotron/artifacts/20160720_084317_892137
[libtaskotron:overlord.py:70] 2016-07-20 08:43:17 DEBUG Execution mode: local
[libtaskotron:rpm_utils.py:184] 2016-07-20 08:43:17 INFO Checking installed state of 1 packages...
[libtaskotron:rpm_utils.py:229] 2016-07-20 08:43:17 DEBUG Deciding whether DNF cache is available. Running: dnf --cacheonly repolist
[libtaskotron:rpm_utils.py:237] 2016-07-20 08:43:18 DEBUG ✔ DNF cache is available.
[libtaskotron:rpm_utils.py:198] 2016-07-20 08:43:18 DEBUG Running: dnf --cacheonly --assumeno --disableplugin=noroot install rpmlint
[libtaskotron:rpm_utils.py:206] 2016-07-20 08:43:20 DEBUG ✔ All specified packages are installed
[libtaskotron:executor.py:81] 2016-07-20 08:43:20 DEBUG Current workdir: /var/tmp/taskotron/lnie/task-Ps3Iv4
[libtaskotron:executor.py:144] 2016-07-20 08:43:20 DEBUG Executing directive: koji
[libtaskotron:koji_utils.py:165] 2016-07-20 08:43:21 INFO Querying Koji for a list of RPMS for: SuperLUMT-3.1.0-6.el7
[libtaskotron:koji_utils.py:231] 2016-07-20 08:43:22 INFO Fetching 12 RPMs for: SuperLUMT-3.1.0-6.el7 (into /var/tmp/taskotron/lnie/task-Ps3Iv4)
[libtaskotron:file_utils.py:112] 2016-07-20 08:43:23 DEBUG Already downloaded: /var/cache/taskotron/SuperLUMT-common-3.1.0-6.el7.noarch.rpm
[libtaskotron:file_utils.py:116] 2016-07-20 08:43:23 DEBUG Cached file /var/cache/taskotron/SuperLUMT-3.1.0-6.el7.src.rpm differs from its online version. Redownloading.
[libtaskotron:file_utils.py:120] 2016-07-20 08:43:23 DEBUG Downloading (cached): http://kojipkgs.fedoraproject.org/packages/SuperLUMT/3.1.0/6.el7/src/SuperLUMT-3.1.0-6.el7.src.rpm
[libtaskotron:file_utils.py:120] 2016-07-20 08:43:25 DEBUG Downloading (cached): http://kojipkgs.fedoraproject.org/packages/SuperLUMT/3.1.0/6.el7/x86_64/SuperLUMT-3.1.0-6.el7.x86_64.rpm
[libtaskotron:file_utils.py:120] 2016-07-20 08:43:25 DEBUG Downloading (cached): http://kojipkgs.fedoraproject.org/packages/SuperLUMT/3.1.0/6.el7/x86_64/SuperLUMT-complex-3.1.0-6.el7.x86_64.rpm
[libtaskotron:file_utils.py:120] 2016-07-20 08:43:25 DEBUG Downloading (cached): http://kojipkgs.fedoraproject.org/packages/SuperLUMT/3.1.0/6.el7/x86_64/SuperLUMT-complex16-3.1.0-6.el7.x86_64.rpm
[libtaskotron:file_utils.py:120] 2016-07-20 08:43:25 DEBUG Downloading (cached): http://kojipkgs.fedoraproject.org/packages/SuperLUMT/3.1.0/6.el7/x86_64/SuperLUMT-devel-3.1.0-6.el7.x86_64.rpm
[libtaskotron:file_utils.py:120] 2016-07-20 08:43:26 DEBUG Downloading (cached): http://kojipkgs.fedoraproject.org/packages/SuperLUMT/3.1.0/6.el7/x86_64/SuperLUMT-double-3.1.0-6.el7.x86_64.rpm
[libtaskotron:file_utils.py:120] 2016-07-20 08:43:26 DEBUG Downloading (cached): http://kojipkgs.fedoraproject.org/packages/SuperLUMT/3.1.0/6.el7/x86_64/SuperLUMT64-3.1.0-6.el7.x86_64.rpm
[libtaskotron:file_utils.py:120] 2016-07-20 08:43:26 DEBUG Downloading (cached): http://kojipkgs.fedoraproject.org/packages/SuperLUMT/3.1.0/6.el7/x86_64/SuperLUMT64-complex-3.1.0-6.el7.x86_64.rpm
[libtaskotron:file_utils.py:120] 2016-07-20 08:43:26 DEBUG Downloading (cached): http://kojipkgs.fedoraproject.org/packages/SuperLUMT/3.1.0/6.el7/x86_64/SuperLUMT64-complex16-3.1.0-6.el7.x86_64.rpm
[libtaskotron:file_utils.py:120] 2016-07-20 08:43:27 DEBUG Downloading (cached): http://kojipkgs.fedoraproject.org/packages/SuperLUMT/3.1.0/6.el7/x86_64/SuperLUMT64-devel-3.1.0-6.el7.x86_64.rpm
[libtaskotron:file_utils.py:120] 2016-07-20 08:43:27 DEBUG Downloading (cached): http://kojipkgs.fedoraproject.org/packages/SuperLUMT/3.1.0/6.el7/x86_64/SuperLUMT64-double-3.1.0-6.el7.x86_64.rpm
printing variables##############################################################################################################################################|ETA: 0:00:00 12.61 MB/s
{'artifactsdir': '/var/lib/taskotron/artifacts/20160720_084317_892137', 'jobid': '-1', 'koji_build': 'SuperLUMT-3.1.0-5.el7', 'uuid': '20160720_084317_892137', 'namespace': 'dist', '_orig_args': {'task': '/home/lnie/task-rpmlint/runtask1.yml', 'no_destroy': False, 'uuid': '20160720_084317_892137', 'arch': ['x86_64'], 'local': False, 'patch': None, 'debug': True, 'jobid': '-1', 'item': 'SuperLUMT-3.1.0-5.el7', 'ssh_privkey': None, 'ssh': None, 'override': [], 'disksize': None, 'libvirt': False, 'type': 'koji_build'}, 'override': {}, 'type': 'koji_build', 'workdir': '/var/tmp/taskotron/lnie/task-Ps3Iv4', 'no_destroy': False, 'libvirt': False, 'checkname': 'rpmlint', 'koji_bodhi': 'SuperLUMT-3.1.0-5.el7', 'ssh_privkey': None, 'ssh': None, 'arch': ['x86_64'], 'task': '/home/lnie/task-rpmlint/runtask1.yml', 'local': False, 'patch': None, 'item': 'SuperLUMT-3.1.0-5.el7', 'debug': True, 'disksize': None}
[libtaskotron:executor.py:144] 2016-07-20 08:43:27 DEBUG Executing directive: python
[libtaskotron:python_directive.py:154] 2016-07-20 08:43:27 INFO Executing Python: run_rpmlint.run() with args {'koji_bodhi': 'SuperLUMT-3.1.0-5.el7', 'artifactsdir': '/var/lib/taskotron/artifacts/20160720_084317_892137', 'workdir': '/var/tmp/taskotron/lnie/task-Ps3Iv4'}
[rpmlint:run_rpmlint.py:100] 2016-07-20 08:43:27 DEBUG Running command: ['rpmlint', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT-3.1.0-6.el7.src.rpm']
SuperLUMT.src: W: spelling-error %description -l en_US preordered -> reordered, p reordered, prerecorded
SuperLUMT.src: W: spelling-error %description -l en_US preordering -> reordering, p reordering, preordaining
SuperLUMT.src: W: patch-not-applied Patch2: %{name}64-build_shared.patch
SuperLUMT.src: W: patch-not-applied Patch3: %{name}64-fix_testsuite.patch
SuperLUMT.src: W: patch-not-applied Patch5: %{name}64-fix_examples.patch
1 packages and 0 specfiles checked; 0 errors, 5 warnings.
1 packages and 0 specfiles checked; 0 errors, 5 warnings.
[rpmlint:run_rpmlint.py:100] 2016-07-20 08:43:30 DEBUG Running command: ['rpmlint', '-o', 'NetworkEnabled False', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT-3.1.0-6.el7.x86_64.rpm', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT-common-3.1.0-6.el7.noarch.rpm', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT-complex-3.1.0-6.el7.x86_64.rpm', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT-complex16-3.1.0-6.el7.x86_64.rpm', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT-devel-3.1.0-6.el7.x86_64.rpm', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT-double-3.1.0-6.el7.x86_64.rpm', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT64-3.1.0-6.el7.x86_64.rpm', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT64-complex-3.1.0-6.el7.x86_64.rpm', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT64-complex16-3.1.0-6.el7.x86_64.rpm', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT64-devel-3.1.0-6.el7.x86_64.rpm', '/var/tmp/taskotron/lnie/task-Ps3Iv4/SuperLUMT64-double-3.1.0-6.el7.x86_64.rpm']
SuperLUMT.x86_64: W: spelling-error %description -l en_US preordered -> reordered, p reordered, prerecorded
SuperLUMT.x86_64: W: spelling-error %description -l en_US preordering -> reordering, p reordering, preordaining
SuperLUMT.x86_64: W: shared-lib-calls-exit /usr/lib64/libsuperlumt_s.so.3.1 exit@GLIBC_2.2.5
SuperLUMT.x86_64: W: no-documentation
SuperLUMT-complex.x86_64: W: shared-lib-calls-exit /usr/lib64/libsuperlumt_c.so.3.1 exit@GLIBC_2.2.5
SuperLUMT-complex.x86_64: W: no-documentation
SuperLUMT-complex16.x86_64: W: shared-lib-calls-exit /usr/lib64/libsuperlumt_z.so.3.1 exit@GLIBC_2.2.5
SuperLUMT-complex16.x86_64: W: no-documentation
SuperLUMT-devel.x86_64: W: only-non-binary-in-usr-lib
SuperLUMT-devel.x86_64: W: no-documentation
SuperLUMT-double.x86_64: W: shared-lib-calls-exit /usr/lib64/libsuperlumt_d.so.3.1 exit@GLIBC_2.2.5
SuperLUMT-double.x86_64: W: no-documentation
SuperLUMT64.x86_64: W: spelling-error %description -l en_US preordered -> reordered, p reordered, prerecorded
SuperLUMT64.x86_64: W: spelling-error %description -l en_US preordering -> reordering, p reordering, preordaining
SuperLUMT64.x86_64: W: shared-lib-calls-exit /usr/lib64/libsuperlumt64_s.so.3.1 exit@GLIBC_2.2.5
SuperLUMT64.x86_64: W: no-documentation
SuperLUMT64-complex.x86_64: W: shared-lib-calls-exit /usr/lib64/libsuperlumt64_c.so.3.1 exit@GLIBC_2.2.5
SuperLUMT64-complex.x86_64: W: no-documentation
SuperLUMT64-complex16.x86_64: W: shared-lib-calls-exit /usr/lib64/libsuperlumt64_z.so.3.1 exit@GLIBC_2.2.5
SuperLUMT64-complex16.x86_64: W: no-documentation
SuperLUMT64-devel.x86_64: W: only-non-binary-in-usr-lib
SuperLUMT64-devel.x86_64: W: no-documentation
SuperLUMT64-double.x86_64: W: shared-lib-calls-exit /usr/lib64/libsuperlumt64_d.so.3.1 exit@GLIBC_2.2.5
SuperLUMT64-double.x86_64: W: no-documentation
11 packages and 0 specfiles checked; 0 errors, 24 warnings.
11 packages and 0 specfiles checked; 0 errors, 24 warnings.
[rpmlint:run_rpmlint.py:76] 2016-07-20 08:43:31 INFO rpmlint PASSED for SuperLUMT-3.1.0-5.el7 (0 errors, 29 warnings)
[rpmlint:run_rpmlint.py:80] 2016-07-20 08:43:31 DEBUG Saving log to: /var/lib/taskotron/artifacts/20160720_084317_892137/SuperLUMT-3.1.0-5.el7.log
[libtaskotron:executor.py:158] 2016-07-20 08:43:31 DEBUG Variable ${rpmlint_output} was exported with value:
results:
- artifact: /var/lib/taskotron/artifacts/20160720_084317_892137/SuperLUMT-3.1.0-5.el7.log
item: SuperLUMT-3.1.0-5.el7
note: 0 errors, 29 warnings
outcome: PASSED
type: koji_buildAbove is what I get with this diff,and I'm wondering if there is something wrong with your machine?If so,please at least see the modified code,please.
Also, please add/fix unittests, failing unittests is a reason to nack the diff on its own.
Would you please see ' Excuse: the failures are prospective,as this diff is adding the EPEL dist tag which was unsupported.'?
The taks above can not work with the patch present in the review - you are obviously using a codebase that contains other changes, which can be easily seen by the simple fact, that you are using koji_bodhi, which is not possible with this diff applied on the develop branch. So no, the problem is not with my computer, but with the mess you most probably have in you git branch.
The diffs are (if not otherwise necessary and duly noted) required to work in a vacuum of the diffs itselves, not with random changes on top of that.
Would you please see ' Excuse: the failures are prospective,as this diff is adding the EPEL dist tag which was unsupported.'?
I guess I could have been more specific, but once again - by our development guide and consensus, the patches must pass the unittests. Introducing test-breaking changes necessarily means changing/adding the unittests. So no, I will not see the excuse and take it as a done issue. Please comply to our standards, if you want to contribute. Doing otherwise just makes things more complicated for all of us.
This is a bigger patch than I expected, I assumed the changes would be related to just rpm_utils.get_dist_tag(). It's great that you have identified further issues, I'm just saying that it's not as simple as I intended this to be :)
As for the existing discussion:
- You should always have a separate branch for each new feature, so e.g. if you're using koji_bodhi action in koji directive and it's not part of this patch, it's clear that you have some other changes applied. Please make sure you're working in a branch with just these changes that you submitted.
- The test suite has to pass, always, that's the rule. If some unit tests don't pass, you need to fix/adjust them. Also, it is expected that you add new unit tests to cover the new functionality. I understand writing unit tests can be hard, especially with all those mock/dingus objects, so please ask for help on irc if you need it. (The only time when it's OK for the test suite to fail is when you're working on some bigger changes and want to submit your patch as a proof of concept, and discuss the overall approach with others, without wasting time on fixing unit tests. But that has to be clearly marked as a proof of concept and you still need to have the test suite fixed when you post the final version of your patch.)
| libtaskotron/directives/koji_directive.py | ||
|---|---|---|
| 224 | It's safer to check for startswith('fc'). | |
| 226–231 | This is potentially duplicated with changes in distgit directive, and it would be better to have it somewhere separated and shared. Even better, it could be in a configuration file. I'll have to look into this deeper and think about it. | |
| libtaskotron/ext/fedora/rpm_utils.py | ||
| 244–247 | Let's modify the docstring as well. So, for example: :param str rpmstr: string to be manipulated in a format of N(E)VR
(``foo-1.2-3.fc20`` or ``bar-4:1.2-3.el7``) or N(E)VRA
(``foo-1.2-3.fc20.x86_64`` or ``bar-4:1.2-3.el7.i686``)
:return: string containing dist tag (e.g. ``fc20`` or ``el7``) | |
| 252 | This looks OK, but it needs to be covered with unit tests. Every time you add new functionality, it needs to have new unit tests. So please try to add new tests which will verify that if you call for example get_dist_tag('bar-4:1.2-3.el7.i686'), you'll receive 'el7' as the result. | |
| 256 | Let's update the comment as well: # there might be e.g. 'fc22' or 'el7' in git commit hash as part of the release, | |
| libtaskotron/taskformula.py | ||
| 211–213 | This is ugly and potentially unsafe. What about this? disttag = rpm_utils.get_dist_tag(item)
if disttag.startswith('fc'):
env['release'] = disttag[-2:]
else:
# do something elseNow, what should we do in case of non-Fedora dist tags? We can do what you proposed to do, use the default environment. Let's just improve the comment a bit, so that people reading it in the future know *why* it is there: # TODO: Only Fedora minions are supported at the moment (specifically, no RHEL support). Until that
# is resolved, let's fall back to the default minion environment for non-Fedora koji builds.
log.debug('Ignoring non-Fedora item dist tag: %s', disttag)Or, we could simply let the task crash, with error that no minion for e.g. el7 is available. That is something we claim in documentation that we do - crash if a minion is missing. Of course we *want* some tasks to handle EPEL builds, even though we have no RHEL minions (and probably won't have for some time). We could asks such tasks to provide a specific keyword in the formula to indicate that they can run on any supported minion (i.e. their execution environment doesn't have to be the same as the item dist tag). That applies to all rpmlint, abicheck, and probably rpmgrill. That seems as a better solution to me. What do you think, @jskladan? | |
The taks above can not work with the patch present in the review - you are obviously using a codebase that contains other changes, which can be easily seen >by the simple fact, that you are using koji_bodhi, which is not possible with this diff applied on the develop branch. So no, the problem is not with my >computer, but with the mess you most probably have in you git branch.
The thing is when I saw your comment, I'm using a different machine,and I got the log from it,but not as you said how messy my git branch is,though I admit it was.Reading here,you probably will continue reading and ,at the same time,start organizing your words quickly something like: why this branch is not on top of D942's .if I get you here,a friendly notice is :you may want to see the next paragraph first: ). As you said,you can easily find I'm using a different data base,ehich I thought it won't hurt,cause it only added a bodhi_id support,and won't change libtaskotron's reaction to koji_build.But,I have to admit,turns out that besides the difference from D942,there is also another trivial one,which had been ignored by me mistakenly,and that is exactly the reason why I thought it's odd that you say this diff dose not work at all.Anyway,I'm going to update this diff when a decision is made about libtaskotron's reaction to running task on disposable vm with EPEL packages,and you can check if it works well then.
The diffs are (if not otherwise necessary and duly noted) required to work in a vacuum of the diffs itselves, not with random changes on top of that.
I see what you are saying,but considering D942 is likely to be rejected,I preferred not to make a change on top of that diff's branch,otherwise, there is 99% chance that I have to submit this single diff once more just because of D942. Now I know, it kind of obeys the rule,and I will try to avoid this .Speaking of that,I'd like to remind you that I'm waiting for your comment on D898(plus,D909).I know you are very busy,just reminding you slightly,in case you forget it.
Glad to know,I thought there is only little thing need to be handled when I saw this roughly,too:)
As for the existing discussion:
- You should always have a separate branch for each new feature, so e.g. if you're using koji_bodhi action in koji directive and it's not part of this patch, it's clear that you have some other changes applied. Please make sure you're working in a branch with just these changes that you submitted.
- The test suite has to pass, always, that's the rule. If some unit tests don't pass, you need to fix/adjust them. Also, it is expected that you add new unit tests to cover the new functionality. I understand writing unit tests can be hard, especially with all those mock/dingus objects, so please ask for help on irc if you need it. (The only time when it's OK for the test suite to fail is when you're working on some bigger changes and want to submit your patch as a proof of concept, and discuss the overall approach with others, without wasting time on fixing unit tests.
It really makes sense for me now,and I'd like to welcome the rules,which I haven't known before.
| libtaskotron/ext/fedora/rpm_utils.py | ||
|---|---|---|
| 252 | Gonna to do | |
It's safer to check for startswith('fc').