Replace Makefile with doit dodo.py file
ClosedPublic
Actions

Authored by tflink on Jul 8 2015, 10:10 PM.

Details

Reviewers

kparal
mkrizek
jskladan

Summary

While make is great in many situations, it has its limitations. I've reworked the functionality previously covered by the Makefile with tasks in a dodo.py file used by doit. The advantage here is less in the individual project as it will be when we can chain them together from a master dodo.py file to do builds, releases, doc etc. in CI.

After installing doit (either from Fedora repos or pip), you run the tasks by running doit <taskname> where <taskname> is one of:

snapshot take a git snapshot of current HEAD and make tar.gz out of it
checksum create checksum of the snapshot
builddocs build documentation
releasedocs build docs and put them in the builds/<version>/ directory
buildsrpm build srpm from most current snapshot
chainbuild build rpm using mockchain and the current COPR repo

Test Plan

I've run through the various commands and while they are not identical to the old make commands, they are functionally the same.

Diff Detail

Repository

rLTRN libtaskotron

Branch

feature/replace-makefile

Lint

Lint OK

Unit

No Unit Test Coverage

tflink retitled this revision from to Replace Makefile with doit dodo.py file.Jul 8 2015, 10:10 PM

tflink updated this object.

tflink edited the test plan for this revision. (Show Details)

tflink added reviewers: kparal, mkrizek, jskladan.

tflink set the repository for this revision to rLTRN libtaskotron.

tflink added a project: libtaskotron.

Herald added a subscriber: tflink. · View Herald TranscriptJul 8 2015, 10:10 PM

Tried running a few commands, worked for me. LGTM

/me looks forward for automated releases

This revision is now accepted and ready to land.Jul 9 2015, 12:37 PM

Cool!

Looks good. A few remarks:

Could you please mention it in the readme, at least doit list command that lists all the available targets? It's certainly not as obvious and well-known as Makefile is.
python-doit should probably be listed in in readme in the package requirements section (along the other tools used for development)
It consumes output of certain commands. For example, buildsrpm produces output, but chainbuild doesn't. builddocs doesn't either, as well as test. I'd prefer to show all output by default, or at least have an option to do so (I haven't found it). For certain targets, like builddocs, the output often contains important warnings about invalid syntax, and currently we lost all of that info. For test, I'd prefer to see the output even if everything is OK, just to make me feel more at ease. Can that be configured?
In doit list, the description is cropped after the first line. See details:

$ doit list
builddocs     Generate sphinx html docs
buildsrpm     build SRPM using mock, the latest git shapshot and current specfile details
chainbuild    mockchain package using COPR repo to assist. Will fail if deps in repo are not up to date
checksum      Generate checksum of git snapshot
cleandocs     Clean generated sphinx html docs
details       The idea here is to provide a method to see some bits inside the
releasedocs   Put generated docs in the release output
snapshot      Take snapshot of git specified git branch
test          Run the unit and functional tests

In D422#7948, @kparal wrote:

Looks good. A few remarks:

Could you please mention it in the readme, at least doit list command that lists all the available targets? It's certainly not as obvious and well-known as Makefile is.

python-doit should probably be listed in in readme in the package requirements section (along the other tools used for development)

Agreed on the readme stuff, I'll make those changes and update the diff

It consumes output of certain commands. For example, buildsrpm produces output, but chainbuild doesn't. builddocs doesn't either, as well as test. I'd prefer to show all output by default, or at least have an option to do so (I haven't found it). For certain targets, like builddocs, the output often contains important warnings about invalid syntax, and currently we lost all of that info. For test, I'd prefer to see the output even if everything is OK, just to make me feel more at ease. Can that be configured?

The only task with a non-default verbosity is details. I think that buildsrpm has more output due to how mock handles console output. I can force everything to be more verbose if we really want it to be that way but I'd rather leave it as is - you can force more verbose output by adding --verbosity 2 to any command

In doit list, the description is cropped after the first line. See details:

Thanks for the reminder - details isn't really useful for more than debugging internally detected variables (spec version, release, etc.) so I wasn't paying as much attention to it

added doit notes to readme, hid a few tasks that wouldn't be needed regularly from 'doit list'

This revision is now accepted and ready to land.Jul 13 2015, 3:52 PM

In D422#8069, @tflink wrote:

The only task with a non-default verbosity is details. I think that buildsrpm has more output due to how mock handles console output. I can force everything to be more verbose if we really want it to be that way but I'd rather leave it as is - you can force more verbose output by adding --verbosity 2 to any command

Thanks, that seems to work. I've tried to inject invalid syntax into docs, and make some tests fail, and it seems to print the error lines (stderr) even with default verbosity. That's great. So the default values are probably fine.

However, could you please add a short note about --verbosity 2 into the readme as well? It's not documented in doit --help, so you can't figure it out easily. Thanks.

Overall, seems to be a nice build tool.

readme.rst
59–60	It was sorted by alphabet before ;-) (helps comparing with spec and requirements.txt)

making small changes to readme from comments in review

This was fixed with rLTRN58a26f8f9df7 but the change wasn't detected automagically - closing the revision.

Revision Contents

		Path
M		.gitignore (2 lines)
D	M	Makefile (92 lines)
A	M	dodo.py (167 lines)
M		readme.rst (29 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	1114	3efe3bf		Jul 8 2015, 5:12 PM	★	★
Diff 2	1146	3efe3bf	- added doit notes to readme, hid a few tasks that wouldn't be needed regularly from 'doit list'	Jul 13 2015, 3:52 PM	★	★
Diff 3	1180	3efe3bf	- making small changes to readme from comments in review	Jul 15 2015, 4:34 PM	★	★

Commit	Tree	Parents	Author	Summary	Date
3075fbd5fe08	af36cab9e61f	c9cac5f57b0f	Tim Flink	making small changes to readme from comments in review	Jul 15 2015, 4:34 PM
c9cac5f57b0f	3ae38fd380d4	985ccf9dc2fe	Tim Flink	added doit notes to readme, hid a few tasks that wouldn't be needed regularly… (Show More…)	Jul 13 2015, 3:50 PM
985ccf9dc2fe	1d35e9dca8af	3efe3bfaa837	Tim Flink	replace Makefile with dodo.py file used by doit	Jul 8 2015, 4:26 PM

Diff 1180

View Options

.gitignore

1	# Python	1	# Python
2	*.py[co]	2	*.py[co]
3		3
4	# building & packages	4	# building & packages
5	/build/	5	/build/
6	/dist/	6	/dist/
7	/*.egg-info	7	/*.egg-info
8	/*.tar.gz	8	/*.tar.gz
9	/*.rpm	9	/*.rpm
		10	.doit.db
		11	/builds/
10		12
11	# vim swap files	13	# vim swap files
12	.*.swp	14	.*.swp
13		15
14	# test coverage	16	# test coverage
15	/.coverage	17	/.coverage
16	/htmlcov/	18	/htmlcov/
17		19
Show All 10 Lines

View Options

Makefile

This file was deleted.

This file was completely deleted. Show File Contents

View Options

dodo.py

This file was added.

		1		#! /usr/bin/env python
		2		# encoding: utf-8
		3
		4		import os
		5		import codecs
		6		import re
		7		import subprocess
		8		from datetime import datetime
		9
		10		from doit.tools import create_folder
		11		from doit import get_var
		12
		13
		14		################################################################################
		15		# PROJECT DEFINITIONS
		16		# This dodo file is designed to be pretty much identical in all qadevel
		17		# projects. This section should be the only part which differs between them.
		18		################################################################################
		19
		20		NAME = 'libtaskotron'
		21		MOCKENV = 'fedora-21-x86_64'
		22		TARGETDIST = 'fc21'
		23		BUILDARCH = 'noarch'
		24		COPRREPO = 'https://copr-be.cloud.fedoraproject.org/results/tflink/taskotron/fedora-21-x86_64/'
		25
		26
		27		################################################################################
		28		# UTILITY FUNCTIONS
		29		# Functions used to generate some of the needed metadata
		30		################################################################################
		31
		32		def get_version():
		33		""" Assume that the version is stored as __version__ in $APPNAME/__init__.py
		34		so that it can be generically extracted
		35		"""
		36
		37		version_file = codecs.open(os.path.join(NAME, '__init__.py'), 'r').read()
		38		version_match = re.search(r"^__version__ = ['\"]([^'\"]*)['\"]",
		39		version_file, re.M)
		40		if version_match:
		41		return version_match.group(1)
		42		raise RuntimeError("Unable to find version string.")
		43
		44
		45		def get_rpmrelease(specfile):
		46		"""parse the rpm release out of the specfile that we're currently using"""
		47		rpmspec_command = ['rpmspec', '-q', '--queryformat="%{RELEASE}\\n"', specfile]
		48		raw_output = subprocess.check_output(rpmspec_command).split()[0].lstrip('"')
		49		return raw_output.split('.')[0]
		50
		51
		52		################################################################################
		53		# GENERAL VARIABLES
		54		# These variables are computed based on project definitions and current state
		55		# as a convenience for later functions
		56		################################################################################
		57
		58		VERSION = get_version()
		59		TARGETNAME = "{}-{}".format(NAME, VERSION)
		60		SPECFILE = '{}.spec'.format(NAME)
		61		RPMRELEASE = get_rpmrelease(os.path.abspath(SPECFILE))
		62		NVR = '{}-{}-{}.{}'.format(NAME, VERSION, RPMRELEASE, TARGETDIST)
		63		SHATYPE = 'sha256'
		64
		65		HERE = os.path.abspath(os.path.dirname(__file__))
		66		OUTDIR = os.path.join(HERE, 'builds/{}'.format(VERSION))
		67
		68		# these are temporary, should be options with defaults
		69		BRANCH = get_var('branch', 'develop')
		70
		71
		72		################################################################################
		73		# PROJECT TASKS
		74		# These tasks should be pretty much consistent for all qadevel projects, driven
		75		# by the variables listed above
		76		################################################################################
		77
		78		def task_chainbuild():
		79		"""mockchain package using COPR repo to assist. Will fail if deps in repo are not up to date"""
		80		tmpdir_prefix = '{}-{}'.format(NAME, datetime.utcnow().strftime('%Y%m%d%H%M%S'))
		81		mockchain_command = 'mockchain -r {} --tmp_prefix={} -a {} '\
		82		'{}/{}.src.rpm'.format(MOCKENV, tmpdir_prefix,
		83		COPRREPO, OUTDIR, NVR)
		84		cp_command = 'cp /var/tmp/mock-chain-{}*/results/{}/{}/{}.{}.rpm '\
		85		'{}/.'.format(tmpdir_prefix, MOCKENV, NVR, NVR, BUILDARCH,
		86		OUTDIR)
		87
		88		return {'actions': [mockchain_command, cp_command],
		89		'task_dep': ['buildsrpm'],
		90		'targets': ['{}.rpm'.format(NVR)]
		91		}
		92
		93
		94		def task_buildsrpm():
		95		"""build SRPM using mock, the latest git shapshot and current specfile details"""
		96		mocksrpm_command = 'mock -r {} --buildsrpm --spec={} '\
		97		'--sources={}'.format(MOCKENV, SPECFILE, OUTDIR)
		98		cp_command = 'cp /var/lib/mock/{}/result/{}-{}-{}.{}.src.rpm '\
		99		'{}/.'.format(MOCKENV, NAME, VERSION, RPMRELEASE, TARGETDIST,
		100		OUTDIR)
		101
		102		return {'actions': [mocksrpm_command, cp_command],
		103		'task_dep': ['snapshot'],
		104		'targets': ['{}.src.rpm'.format(NVR)]
		105		}
		106
		107
		108		def task_snapshot():
		109		"""Take snapshot of git specified git branch"""
		110		archive_command = 'git archive {} --prefix={}/ \| gzip -c9 > '\
		111		'{}.tar.gz'.format(BRANCH,
		112		TARGETNAME,
		113		os.path.join(OUTDIR, TARGETNAME))
		114
		115		return {'actions': [(create_folder, [OUTDIR]),
		116		archive_command],
		117		'targets': ['{}.tar.gz'.format(TARGETNAME)],
		118		}
		119
		120
		121		def task__checksum():
		122		"""Generate checksum of git snapshot"""
		123		sha_command = '{shatype}sum {outdir}/{target}.tar.gz > '\
		124		'{outdir}/{target}.tar.gz.{shatype}'.format(shatype=SHATYPE,
		125		target=TARGETNAME,
		126		outdir=OUTDIR)
		127		return {'actions': [sha_command],
		128		'task_dep': ['snapshot'],
		129		'targets': ['{}.tar.gz.{}'.format(TARGETNAME, SHATYPE)]
		130		}
		131
		132
		133		def task_builddocs():
		134		"""Generate sphinx html docs"""
		135		return {'actions': ['pushd docs;make html'],
		136		'targets': ['docs/build/html']
		137		}
		138
		139
		140		def task_cleandocs():
		141		"""Clean generated sphinx html docs"""
		142		return {'actions': ['pushd docs;make clean'],
		143		}
		144
		145
		146		def task_releasedocs():
		147		"""Put generated docs in the release output"""
		148		return {'actions': ['cp -r docs/build/html {}/docs'.format(OUTDIR)],
		149		'task_dep': ['builddocs'],
		150		'targets': ['{}/docs'.format(OUTDIR)]
		151		}
		152
		153
		154		def task_test():
		155		"""Run the unit and functional tests"""
		156		return {'actions': ['py.test -F testing/']
		157		}
		158
		159
		160		def task__details():
		161		""" Display some calculated variables used in the script for debug purposes"""
		162
		163		return {'actions': ['echo "taskotron version: {}"'.format(VERSION),
		164		'echo "target name: {}"'.format(TARGETNAME),
		165		'echo "output dir: {}"'.format(OUTDIR)],
		166		'verbosity': 2
		167		}

View Options

readme.rst

Show First 20 Lines • Show All 43 Lines • ▼ Show 20 Line(s)		42	Once the copr repo is available, install the necessary packages::
44	sudo yum install \	44		sudo yum install \
45	createrepo \	45		createrepo \
46	gcc \	46		gcc \
47	git \	47		git \
48	koji \	48		koji \
49	libtaskotron-config \	49		libtaskotron-config \
50	mash \	50		mash \
51	pyOpenSSL \	51		pyOpenSSL \
		52		python-doit \
52	python-hawkey \	53		python-hawkey \
53	python-pip \	54		python-pip \
54	python-pycurl \	55		python-pycurl \
55	python-rpmfluff \	56		python-rpmfluff \
56	python-urlgrabber \	57		python-urlgrabber \
57	python-virtualenv \	58		python-virtualenv \
58	rpm-python	59		rpm-python
59		60
			kparalUnsubmitted Not Done It was sorted by alphabet before ;-) (helps comparing with spec and requirements.txt)
60	If you have not yet cloned the repository, do it now::	61		If you have not yet cloned the repository, do it now::
61		62
62	git clone https://bitbucket.org/fedoraqa/libtaskotron	63		git clone https://bitbucket.org/fedoraqa/libtaskotron
63	cd libtaskotron	64		cd libtaskotron
64		65
65	Then, set up the virtualenv::	66		Then, set up the virtualenv::
66		67
67	virtualenv --system-site-packages env_taskotron	68		virtualenv --system-site-packages env_taskotron
▲ Show 20 Lines • Show All 96 Lines • ▼ Show 20 Line(s)
164	<http://docutils.sourceforge.net/rst.html>`_ and built using `Sphinx	165		<http://docutils.sourceforge.net/rst.html>`_ and built using `Sphinx
165	<http://sphinx-doc.org/>`_.	166		<http://sphinx-doc.org/>`_.
166		167
167	The documentation is easy to build if you have followed the instructions to set	168		The documentation is easy to build if you have followed the instructions to set
168	up a development environment.	169		up a development environment.
169		170
170	To actually build the documentation::	171		To actually build the documentation::
171		172
172	cd docs/	173		doit builddocs
173	make html	174
		175		Build Automation
		176		================
		177
		178		There are several development related tasks which are at least somewhat
		179		automated using `doit <http://pydoit.org/>`_ which has repaced our old Makefile.
		180
		181		After either installing the doit package (``python-doit``, ``python3-doit``) or
		182		via pip (``pip install doit``), you can see a list of available tasks and a short
		183		description of those tasks by running ``doit list``. Some of the available tasks
		184		are:
		185
		186		* ``buildsrpm`` takes a snapshot of current git repo and uses the in-repo spec
		187		file to build a matching srpm in the ``builds/<version>`` directory. Note
		188		that if a snapshot already exists for a given version, a new snapshot will
		189		not be generated until the existing one is deleted.
		190
		191		* ``chainbuild`` uses mockchain and the existing COPR repo to build a noarch
		192		binary rpm from the latest srpm.
		193
		194		* ``builddocs`` builds documentation from current git sources
		195
		196		By default, the tool is pretty quiet but if you would like to see more verbose
		197		output, add ``--verbosity 2`` to the doit command and all stdout/stderr output
		198		will be shown.

Replace Makefile with doit dodo.py fileClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 1180

.gitignore

Makefile

dodo.py

readme.rst

Replace Makefile with doit dodo.py file
ClosedPublic
Actions