Differential D806

reworking dodo.py a little bit to better support our upcoming ci setup
ClosedPublic
Actions

Authored by tflink on Apr 7 2016, 6:39 PM.

Details

Reviewers
kparal
mkrizek
jskladan
Commits
rLTRN7732574a7b0d: reworking dodo.py a little bit to better support our upcoming ci setup
Summary

this changes a few things around so that we can use doit for both local doc/rpm build and keep the buildbot config somewhat general. this should be mostly re-usable in most of our other projects once a few variables are changed

Test Plan

run doit locally, note what happens if you use envtype=ci or basedir=/some/other/path and make sure that's the kind of output pathing we'd want for CI

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
tflink retitled this revision from to reworking dodo.py a little bit to better support our upcoming ci setup.Apr 7 2016, 6:39 PM
tflink updated this object.
tflink edited the test plan for this revision. (Show Details)
tflink added reviewers: mkrizek, jskladan, kparal.
mkrizek added a comment.Apr 8 2016, 7:20 AM

Looks good to me, just couple nitpicks.

dodo.py
133

This will be always True, no? BRANCH.lower() perhaps?

253

Same as above.

269

I think 'else' is not needed here since Exception is raised, no? :)

tflink planned changes to this revision.Apr 8 2016, 9:08 PM
tflink added inline comments.
dodo.py
133

yeah, that needs to be fixed. thanks

tflink updated this revision to Diff 2063.Apr 8 2016, 9:09 PM
  • fixing dodo.py file per review comments
mkrizek accepted this revision.Apr 11 2016, 6:24 AM
This revision is now accepted and ready to land.Apr 11 2016, 6:24 AM
kparal added inline comments.Apr 12 2016, 2:53 PM
dodo.py
130

So what exactly is this variable for? I don't see it used anywhere.

tflink added inline comments.May 6 2016, 2:06 PM
dodo.py
130

it's an artifact from a previous revision that I forgot to remove, thanks for pointing it out

Closed by commit rLTRN7732574a7b0d: reworking dodo.py a little bit to better support our upcoming ci setup (authored by tflink). · Explain WhyMay 6 2016, 2:36 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathPackages
Mdodo.py (60 lines)

Diff 2161

View Options

dodo.py

Show First 20 Lines • Show All 75 Lines • ▼ Show 20 Line(s)74def get_gitbranch():
76 76​
77 gitbranch_command = ['git', 'rev-parse', '--abbrev-ref', 'HEAD'] 77​ gitbranch_command = ['git', 'rev-parse', '--abbrev-ref', 'HEAD']
78 raw_output = subprocess.check_output(gitbranch_command) 78​ raw_output = subprocess.check_output(gitbranch_command)
79 return raw_output.strip() 79​ return raw_output.strip()
80 80​
81 81​
82def maybe_use_virtualenv(commands): 82​def maybe_use_virtualenv(commands):
83 """Adds activate and deactivate commands to a chain of otherwise normal 83​ """Adds activate and deactivate commands to a chain of otherwise normal
84 bash commands if the ENVTYPE variable is set to 'ci'""" 84​ bash commands if the ENVTYPE variable is set to 'ci'. Note that this is
85​ making assumptions about what's installed as packages and you may need
86​ to install a few more things in the virtualenv (testcloud, sphinx etc.)"""
85 87​
86 if ENVTYPE != 'ci': 88​ if ENVTYPE != 'ci':
87 return commands 89​ return commands
88 90​
89 real_commands = [] 91​ real_commands = []
90 for command in commands: 92​ for command in commands:
91 real_commands.append('source {}/bin/activate; {}'.format(VENVNAME, command)) 93​ real_commands.append('source {}/bin/activate; {}'.format(VENVNAME, command))
92 94​
Show All 11 Lines
104RPMRELEASE = get_rpmrelease(os.path.abspath(SPECFILE)) 106​RPMRELEASE = get_rpmrelease(os.path.abspath(SPECFILE))
105NVR = '{}-{}-{}.{}'.format(NAME, VERSION, RPMRELEASE, TARGETDIST) 107​NVR = '{}-{}-{}.{}'.format(NAME, VERSION, RPMRELEASE, TARGETDIST)
106SHATYPE = 'sha256' 108​SHATYPE = 'sha256'
107 109​
108HERE = os.path.abspath(os.path.dirname(__file__)) 110​HERE = os.path.abspath(os.path.dirname(__file__))
109 111​
110ENVTYPE = get_var('envtype', 'dev') 112​ENVTYPE = get_var('envtype', 'dev')
111BRANCH = get_var('branch', 'HEAD') 113​BRANCH = get_var('branch', 'HEAD')
112BUILDTYPE = get_var('buildtype', 'NORMAL') ​
113 114​
114CURBRANCH = get_gitbranch() 115​CURBRANCH = get_gitbranch()
115 116​
116# many branches have slashes in them, make sure they'll be one dir 117​# many branches have slashes in them, make sure they'll be one dir
117SAFEBRANCH = CURBRANCH.replace('/', '_') 118​SAFEBRANCH = CURBRANCH.replace('/', '_')
118 119​
119BASEDIR = get_var('basedir', os.path.join(HERE, 'builds')) 120​BASEDIR = get_var('basedir', os.path.join(HERE, 'builds'))
120 121​
121 122​
122# if we're releasing, want to have the branch name in the basedir ​
123if BUILDTYPE.upper() == 'RELEASE': ​
124 BASEDIR = os.path.join(BASEDIR, SAFEBRANCH) ​
125 123​
126OUTDIR = os.path.join(BASEDIR, VERSION) 124​OUTDIR = os.path.join(BASEDIR, VERSION)
125​RELEASES_OUTDIR = OUTDIR
126​DOCS_OUTDIR = OUTDIR
127 127​
128​# We want to be doing things a little differently when we're in CI mode
129​if ENVTYPE == 'ci':
130​ outversion = VERSION
kparalUnsubmitted
Not Done

So what exactly is this variable for? I don't see it used anywhere.

tflinkAuthorUnsubmitted
Not Done

it's an artifact from a previous revision that I forgot to remove, thanks for pointing it out

131​
132​ # if we're not on master, add the branch name
133​ if BRANCH.upper() != 'MASTER':
mkrizekUnsubmitted
Not Done

This will be always True, no? BRANCH.lower() perhaps?

tflinkAuthorUnsubmitted
Not Done

yeah, that needs to be fixed. thanks

134​ outversion = SAFEBRANCH
135​
136​ RELEASES_OUTDIR = os.path.join(BASEDIR, 'releases', NAME, SAFEBRANCH)
137​ DOCS_OUTDIR = os.path.join(BASEDIR, 'docs', NAME, SAFEBRANCH)
128 138​
129 139​
130################################################################################ 140​################################################################################
131# PROJECT TASKS 141​# PROJECT TASKS
132# These tasks should be pretty much consistent for all qadevel projects, driven 142​# These tasks should be pretty much consistent for all qadevel projects, driven
133# by the variables listed above 143​# by the variables listed above
134################################################################################ 144​################################################################################
135 145​
136def task_chainbuild(): 146​def task_chainbuild():
137 """mockchain package using COPR repo to assist. Will fail if deps in repo are not up to date""" 147​ """mockchain package using COPR repo to assist. Will fail if deps in repo are not up to date"""
138 tmpdir_prefix = '{}-{}'.format(NAME, datetime.utcnow().strftime('%Y%m%d%H%M%S')) 148​ tmpdir_prefix = '{}-{}'.format(NAME, datetime.utcnow().strftime('%Y%m%d%H%M%S'))
139 mockchain_command = 'mockchain -r {} --tmp_prefix={} -a {} '\ 149​ mockchain_command = 'mockchain -r {} --tmp_prefix={} -a {} '\
140 '{}/{}.src.rpm'.format(MOCKENV, tmpdir_prefix, 150​ '{}/{}.src.rpm'.format(MOCKENV, tmpdir_prefix,
141 ' -a '.join(COPRREPOS), 151​ ' -a '.join(COPRREPOS),
142 OUTDIR, NVR) 152​ RELEASES_OUTDIR, NVR)
143 cp_command = 'cp /var/tmp/mock-chain-{}*/results/{}/{}/*.{}.rpm '\ 153​ cp_command = 'cp /var/tmp/mock-chain-{}*/results/{}/{}/*.{}.rpm '\
144 '{}/.'.format(tmpdir_prefix, MOCKENV, NVR, BUILDARCH, 154​ '{}/.'.format(tmpdir_prefix, MOCKENV, NVR, BUILDARCH,
145 OUTDIR) 155​ RELEASES_OUTDIR)
146 156​
147 return {'actions': [mockchain_command, cp_command], 157​ return {'actions': [mockchain_command, cp_command],
148 'task_dep': ['buildsrpm'], 158​ 'task_dep': ['buildsrpm'],
149 'targets': ['{}.rpm'.format(NVR)] 159​ 'targets': ['{}.rpm'.format(NVR)]
150 } 160​ }
151 161​
152 162​
153def task_buildsrpm(): 163​def task_buildsrpm():
154 """build SRPM using mock, the latest git shapshot and current specfile details""" 164​ """build SRPM using mock, the latest git shapshot and current specfile details"""
155 mocksrpm_command = 'mock -r {} --buildsrpm --spec={} '\ 165​ mocksrpm_command = '/usr/bin/mock -r {} --buildsrpm --spec={} '\
156 '--sources={}'.format(MOCKENV, SPECFILE, OUTDIR) 166​ '--sources={}'.format(MOCKENV, SPECFILE, RELEASES_OUTDIR)
157 cp_command = 'cp /var/lib/mock/{}/result/{}-{}-{}.{}.src.rpm '\ 167​ cp_command = 'cp /var/lib/mock/{}/result/{}-{}-{}.{}.src.rpm '\
158 '{}/.'.format(MOCKENV, NAME, VERSION, RPMRELEASE, TARGETDIST, 168​ '{}/.'.format(MOCKENV, NAME, VERSION, RPMRELEASE, TARGETDIST,
159 OUTDIR) 169​ RELEASES_OUTDIR)
160 170​
161 return {'actions': [mocksrpm_command, cp_command], 171​ return {'actions': [mocksrpm_command, cp_command],
162 'task_dep': ['snapshot'], 172​ 'task_dep': ['snapshot'],
163 'targets': ['{}.src.rpm'.format(NVR)] 173​ 'targets': ['{}.src.rpm'.format(NVR)]
164 } 174​ }
165 175​
166 176​
167def task_snapshot(): 177​def task_snapshot():
168 """Take snapshot of git specified git branch""" 178​ """Take snapshot of git specified git branch"""
169 archive_command = 'git archive {} --prefix={}/ | gzip -c9 > '\ 179​ archive_command = 'git archive {} --prefix={}/ | gzip -c9 > '\
170 '{}.tar.gz'.format(BRANCH, 180​ '{}.tar.gz'.format(BRANCH,
171 TARGETNAME, 181​ TARGETNAME,
172 os.path.join(OUTDIR, TARGETNAME)) 182​ os.path.join(RELEASES_OUTDIR, TARGETNAME))
173 183​
174 return {'actions': [(create_folder, [OUTDIR]), 184​ return {'actions': [(create_folder, [RELEASES_OUTDIR]),
175 archive_command], 185​ archive_command],
176 'targets': ['{}.tar.gz'.format(TARGETNAME)], 186​ 'targets': ['{}.tar.gz'.format(TARGETNAME)],
177 } 187​ }
178 188​
179 189​
180def task__checksum(): 190​def task__checksum():
181 """Generate checksum of git snapshot""" 191​ """Generate checksum of git snapshot"""
182 sha_command = '{shatype}sum {outdir}/{target}.tar.gz > '\ 192​ sha_command = '{shatype}sum {outdir}/{target}.tar.gz > '\
183 '{outdir}/{target}.tar.gz.{shatype}'.format(shatype=SHATYPE, 193​ '{outdir}/{target}.tar.gz.{shatype}'.format(shatype=SHATYPE,
184 target=TARGETNAME, 194​ target=TARGETNAME,
185 outdir=OUTDIR) 195​ outdir=RELEASES_OUTDIR)
186 return {'actions': [sha_command], 196​ return {'actions': [sha_command],
187 'task_dep': ['snapshot'], 197​ 'task_dep': ['snapshot'],
188 'targets': ['{}.tar.gz.{}'.format(TARGETNAME, SHATYPE)] 198​ 'targets': ['{}.tar.gz.{}'.format(TARGETNAME, SHATYPE)]
189 } 199​ }
190 200​
191 201​
192def task_builddocs(): 202​def task_builddocs():
193 """Generate sphinx html docs""" 203​ """Generate sphinx html docs"""
194 return {'actions': maybe_use_virtualenv(['pushd docs;make html']), 204​ return {'actions': maybe_use_virtualenv(['pushd docs;make html']),
205​ 'task_dep': ['_checkvirtualenv'],
195 'targets': ['docs/build/html'] 206​ 'targets': ['docs/build/html']
196 } 207​ }
197 208​
198 209​
199def task_cleandocs(): 210​def task_cleandocs():
200 """Clean generated sphinx html docs""" 211​ """Clean generated sphinx html docs"""
201 return {'actions': ['pushd docs;make clean'], 212​ return {'actions': ['pushd docs;make clean'],
202 } 213​ }
203 214​
204 215​
205def task_releasedocs(): 216​def task_releasedocs():
206 """Put generated docs in the release output""" 217​ """Put generated docs in the release output"""
207 return {'actions': ['mkdir -p {}'.format(OUTDIR), 218​ return {'actions': ['mkdir -p {}'.format(DOCS_OUTDIR),
208 'cp -r docs/build/html {}/docs'.format(OUTDIR)], 219​ 'cp -r docs/build/html {}'.format(DOCS_OUTDIR)],
209 'task_dep': ['builddocs'], 220​ 'task_dep': ['builddocs'],
210 'targets': ['{}/docs'.format(OUTDIR)] 221​ 'targets': ['{}/docs'.format(DOCS_OUTDIR)]
211 } 222​ }
212 223​
213 224​
214def set_symlink(basedir, source, linkdest): 225​def set_symlink(basedir, source, linkdest):
215 """make relative symlink between source and linkdest in basedir""" 226​ """make relative symlink between source and linkdest in basedir"""
216 227​
217 fullsource = os.path.join(basedir, source) 228​ fullsource = os.path.join(basedir, source)
218 fulldest = os.path.join(basedir, linkdest) 229​ fulldest = os.path.join(basedir, linkdest)
Show All 14 Lines
233 os.symlink(source, linkdest) 244​ os.symlink(source, linkdest)
234 os.chdir(cwd) 245​ os.chdir(cwd)
235 246​
236 247​
237def task_updatelatest(): 248​def task_updatelatest():
238 """Make sure that the 'latest' symlink in output directory points to the correct version""" 249​ """Make sure that the 'latest' symlink in output directory points to the correct version"""
239 250​
240 def updatelatest(): 251​ def updatelatest():
241 if not os.path.exists(OUTDIR): 252​ # this isn't needed for non-release branches
242 raise Exception("BASEDIR ({}) does not exist, cannot fix symlinks".format(BASEDIR)) 253​ if BRANCH.upper() != 'MASTER':
mkrizekUnsubmitted
Not Done

Same as above.

254​ return
255​
256​ if not os.path.exists(DOCS_OUTDIR):
257​ raise Exception("DOCS_OUTDIR ({}) does not exist, cannot fix ",
258​ "symlinks".format(DOCS_OUTDIR))
259​
260​ set_symlink(os.path.split(DOCS_OUTDIR)[0], VERSION, 'latest')
261​
262​ if not os.path.exists(RELEASES_OUTDIR):
263​ raise Exception("RELEASES_OUTDIR ({}) does not exist, cannot fix ",
264​ "symlinks".format(RELEASES_OUTDIR))
243 265​
244 set_symlink(BASEDIR, VERSION, 'latest') 266​ set_symlink(os.path.split(RELEASES_OUTDIR)[0], VERSION, 'latest')
245 267​
246 return {'actions': [updatelatest], 268​ return {'actions': [updatelatest],
247 'targets': [], 269​ 'targets': [],
mkrizekUnsubmitted
Not Done

I think 'else' is not needed here since Exception is raised, no? :)

248 'task_dep': ['releasedocs'], 270​ 'task_dep': ['releasedocs'],
249 } 271​ }
250 272​
251 273​
252def task_test(): 274​def task_test():
253 """Run the unit and functional tests""" 275​ """Run the unit and functional tests"""
254 return {'actions': maybe_use_virtualenv(['py.test -F testing/']), 276​ return {'actions': maybe_use_virtualenv(['py.test -F testing/']),
255 'task_dep': ['_checkvirtualenv'] 277​ 'task_dep': ['_checkvirtualenv']
Show All 34 Lines