adding dodo.py file for doit, allows easier reproduction of depcheck runs
AbandonedPublic

Authored by kparal on Apr 21 2015, 3:45 PM.

Details

Reviewers
tflink
Summary

allows easier reproduction of depcheck runs. after installing doit, 'doit buildid=<BUILDID> depcheck' will spit out the required command to reproduce the run specified by the BUILDID of the run you're interested in on taskotron.fp.o

Test Plan

not much, just try it out and see if it works

Diff Detail

Repository
rDEPCK task-depcheck
Branch
feature/reproducing-runs-doit
Lint
No Linters Available
Unit
No Unit Test Coverage
tflink retitled this revision from to adding dodo.py file for doit, allows easier reproduction of depcheck runs.Apr 21 2015, 3:45 PM
tflink updated this object.
tflink edited the test plan for this revision. (Show Details)
tflink added a reviewer: jskladan.
tflink added a subscriber: kparal.
tflink updated this object.Apr 21 2015, 4:21 PM

had a typo in the initial description - should have been doit buildid=<SOME BUILDID> depcheck

This uses doit to do the following:

  • get buildinfo from taskotron buildbot (run date, koji tag etc.)
  • find and download archived repodata that was used for that run
  • download all rpms used in the original run (by scraping the output in taskotron.log)
  • mash downloaded rpms
  • print out commands needed to do the depcheck run locally

The files will be stored in a local directory named for the date of original run and the tag used (localruns/<tag>-<date>/)

Example using a run on f22-updates-testing:

$ doit buildid=56193 depcheck
Getting closest branched base release repo
Finding closest mash for f22-updates-testing
.  makedirs
.  fetchtaskotronlog
.  downloadrpms
.  mash
.  depcheck
i386 depcheck command:
python run_depcheck.py -f updates -d -a i386 -r /home/tflink/code/taskotron/task-depcheck/localruns/f22-updates-testing-pending.201504081117/repos/f22/i386 -r /home/tflink/code/taskotron/task-depcheck/localruns/f22-updates-testing-pending.201504081117/repos/f22-updates-testing/i386 -r /home/tflink/code/taskotron/task-depcheck/localruns/f22-updates-testing-pending.201504081117/newrepo-i386 /home/tflink/code/taskotron/task-depcheck/localruns/f22-updates-testing-pending.201504081117/newrepo-i386
x86_64 depcheck command:
python run_depcheck.py -f updates -d -a x86_64 -r /home/tflink/code/taskotron/task-depcheck/localruns/f22-updates-testing-pending.201504081117/repos/f22/x86_64 -r /home/tflink/code/taskotron/task-depcheck/localruns/f22-updates-testing-pending.201504081117/repos/f22-updates-testing/x86_64 -r /home/tflink/code/taskotron/task-depcheck/localruns/f22-updates-testing-pending.201504081117/newrepo-x86_64 -r /home/tflink/code/taskotron/task-depcheck/localruns/f22-updates-testing-pending.201504081117/repos/f22/i386 -r /home/tflink/code/taskotron/task-depcheck/localruns/f22-updates-testing-pending.201504081117/repos/f22-updates-testing/i386 -r /home/tflink/code/taskotron/task-depcheck/localruns/f22-updates-testing-pending.201504081117/newrepo-i386 /home/tflink/code/taskotron/task-depcheck/localruns/f22-updates-testing-pending.201504081117/newrepo-x86_64
kparal added a comment.EditedApr 22 2015, 3:02 PM

Thanks, that's useful. It seems it fails to download repo metadata for me, but I haven't debugged why:

$ doit buildid=60951 depcheck
Finding closest mash for f21-updates
.  makedirs
.  fetchtaskotronlog
fetching https://taskotron.fedoraproject.org/taskmaster/builders/x86_64/builds/60951/steps/runtask/logs/taskotron.log/text as /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/taskotron.log
.  downloadrpms
.  mash
.  depcheck
i386 depcheck command:
python run_depcheck.py -f updates -d -a i386 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21/i386 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21-updates/i386 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/newrepo-i386 /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/newrepo-i386
x86_64 depcheck command:
python run_depcheck.py -f updates -d -a x86_64 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21/x86_64 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21-updates/x86_64 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/newrepo-x86_64 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21/i386 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21-updates/i386 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/newrepo-i386 /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/newrepo-x86_64

$ python run_depcheck.py -f updates -d -a i386 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21/i386 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21-updates/i386 -r /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/newrepo-i386 /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/newrepo-i386
DEBUG:depcheck:Preparing metadata download
DEBUG:depcheck:  - '/home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/newrepo-i386': 'i686', '/home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/newrepo-i386'
DEBUG:depcheck:  - '/home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21/i386': 'i686', '/home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21/i386'
DEBUG:depcheck:  - '/home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21-updates/i386': 'i686', '/home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21-updates/i386'
INFO:depcheck:Getting metadata...
Traceback (most recent call last):
  File "run_depcheck.py", line 59, in <module>
    depcheck.depcheck_cli()
  File "/home/kparal/devel/taskotron/task-depcheck/depcheck/__init__.py", line 168, in depcheck_cli
    run(args.rpm, arch, repos, args.format, args.workdir)
  File "/home/kparal/devel/taskotron/task-depcheck/depcheck/__init__.py", line 115, in run
    repomd_files = md_handle.repodata
  File "/home/kparal/devel/taskotron/task-depcheck/depcheck/metadata.py", line 253, in repodata
    self.verify()
  File "/home/kparal/devel/taskotron/task-depcheck/depcheck/metadata.py", line 243, in verify
    self._handle_repo(reponame)
  File "/home/kparal/devel/taskotron/task-depcheck/depcheck/metadata.py", line 228, in _handle_repo
    repo_targetdir)
  File "/home/kparal/devel/taskotron/task-depcheck/depcheck/metadata.py", line 192, in _handle_local_repofiles
    local_repofiles = self._find_repo_mdfiles(repopath)
  File "/home/kparal/devel/taskotron/task-depcheck/depcheck/metadata.py", line 147, in _find_repo_mdfiles
    raise Exception("Not able to find requested metadata file for %s in %s" % (mdtype, repodir))
Exception: Not able to find requested metadata file for filelists in /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21/i386

$ find /home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21/i386/
/home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21/i386/
/home/kparal/devel/taskotron/task-depcheck/localruns/f21-updates-pending.201504221402/repos/f21/i386/repodata

I wanted to fetch https://taskotron.fedoraproject.org/resultsdb/results/1380421 , because it is quite small.

dodo.py
6

Could you perhaps just very shortly document the purpose of this script as a module docstring? Just copy&paste the description from the phabricator. Thanks.

38

I guess this needs to be added to .gitignore. As well as .doit.db, it seems.

310

This print doesn't seem to work.

Could you also print some kind of progress, like Downloading (1/29): NVR?
(Also I wonder why our regular progress bar from KojiClient doesn't work).

looks like there's a missing dep for task_depcheck - if you run 'fetchrepodata', it should work

I've been futzing with the script today, trying to make it behave more like taskotron, will post changes when I get them working better

kparal requested changes to this revision.Sep 9 2015, 11:05 AM
kparal added a reviewer: kparal.

Moving back to you.

This revision now requires changes to proceed.Sep 9 2015, 11:05 AM
kparal commandeered this revision.Wed, Aug 2, 2:17 PM
kparal abandoned this revision.
kparal edited reviewers, added: tflink; removed: kparal.

Depcheck is dead!