Differential D1123

fix deps, improve develop and test workflow
ClosedPublic
Actions

Authored by kparal on Feb 10 2017, 10:26 AM.

Details

Reviewers

mkrizek

Group Reviewers

taskotron-trigger

Commits

rTRGR5389bf71469f: fix deps, improve develop and test workflow

Summary

Fix dependencies in requirements.txt: we need fedmsg[consumers], not

just fedmsg, and add some missing ones

Remove deps from setup.py, because it was not up-to-date anyway and

there's little sense to keep deps defined there as well, as long as we
don't publish this on PyPI

Make all Makefile targets phony (defined at the very same place, so

that it's not forgotten next time we add a target)

Print test coverage when the test suite is run
Use virtualenv --ignore-installed to make sure all the deps possible

are installed into virtualenv, regardless of system libs.

Extend instructions how to set up a development environment
Add testing instructions

Test Plan

Tried the dev setup steps, clean virtualenv, and make test, seems to work

Diff Detail

Repository

rTRGR taskotron-trigger

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

kparal retitled this revision from to fix deps, improve develop and test workflow.Feb 10 2017, 10:26 AM

kparal updated this object.

kparal edited the test plan for this revision. (Show Details)

kparal added a reviewer: taskotron-trigger.

Herald added a subscriber: tflink. · View Herald TranscriptFeb 10 2017, 10:26 AM

spec: use the same test approach as we do in development

mkrizek accepted this revision.Feb 10 2017, 12:41 PM

mkrizek added a reviewer: mkrizek.

This revision is now accepted and ready to land.Feb 10 2017, 12:41 PM

Closed by commit rTRGR5389bf71469f: fix deps, improve develop and test workflow (authored by Kamil Páral <kparal@redhat.com>). · Explain WhyFeb 10 2017, 12:54 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents

			Path	Packages
M			Makefile (30 lines)
M			README.rst (74 lines)
M			requirements.txt (23 lines)
M			setup.py (2 lines)
M			taskotron-trigger.spec (12 lines)
M			tox.ini (2 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	2874	f22c0ed		Feb 10 2017, 10:26 AM	★	★
Diff 2	2878	f22c0ed	- spec: use the same test approach as we do in development	Feb 10 2017, 11:57 AM	★	★
Diff 3	2880	f22c0ed	rTRGR5389bf71469ff22245b89e26f08254429b1c9c36	Feb 10 2017, 12:53 PM	★	★

Commit	Tree	Parents	Author	Summary	Date
2ca8dadb1e87	acd1477a8d0b	7cd19a681a34	Kamil Páral	spec: use the same test approach as we do in development	Feb 10 2017, 11:57 AM
7cd19a681a34	7b2acd881809	f22c0edc3ebb	Kamil Páral	fix deps, improve develop and test workflow (Show More…)	Feb 10 2017, 9:20 AM

Diff 2880

View Options

Makefile

Show All 10 Lines
11	# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the	11	# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12	# GNU General Public License for more details.	12	# GNU General Public License for more details.
13	#	13	#
14	# You should have received a copy of the GNU General Public License along	14	# You should have received a copy of the GNU General Public License along
15	# with this program; if not, write to the Free Software Foundation, Inc.,	15	# with this program; if not, write to the Free Software Foundation, Inc.,
16	# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.	16	# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
17	#	17	#
18		18
19	.PHONY: test test-ci pylint pep8 docs clean virtualenv
20
21	# general variables	19	# general variables
22	VENV=test_env	20	VENV=test_env
23	SRC=taskotron-trigger	21	SRC=taskotron-trigger
24	MODULENAME=jobtriggers	22	MODULENAME=jobtriggers
25		23
26	# Variables used for packaging	24	# Variables used for packaging
27	SPECFILE=$(SRC).spec	25	SPECFILE=$(SRC).spec
28	BASEARCH:=$(shell uname -i)	26	BASEARCH:=$(shell uname -i)
29	DIST:=$(shell rpm --eval '%{dist}')	27	DIST:=$(shell rpm --eval '%{dist}')
30	VERSION:=$(shell rpmspec -q --queryformat="%{VERSION}\n" $(SPECFILE) \| uniq)	28	VERSION:=$(shell rpmspec -q --queryformat="%{VERSION}\n" $(SPECFILE) \| uniq)
31	RELEASE:=$(subst $(DIST),,$(shell rpmspec -q --queryformat="%{RELEASE}\n" $(SPECFILE) \| uniq))	29	RELEASE:=$(subst $(DIST),,$(shell rpmspec -q --queryformat="%{RELEASE}\n" $(SPECFILE) \| uniq))
32	NVR:=$(SRC)-$(VERSION)-$(RELEASE)	30	NVR:=$(SRC)-$(VERSION)-$(RELEASE)
33	GITBRANCH:=$(shell git rev-parse --abbrev-ref HEAD)	31	GITBRANCH:=$(shell git rev-parse --abbrev-ref HEAD)
34	TARGETDIST:=fc24	32	TARGETDIST:=fc24
35	BUILDTARGET=fedora-24-x86_64	33	BUILDTARGET=fedora-24-x86_64
36		34
		35	.PHONY: test
37	test: $(VENV)	36	test: $(VENV)
38	sh -c "set -e; . $(VENV)/bin/activate; TEST='true' py.test --cov $(MODULENAME); deactivate"	37	sh -c "set -e; . $(VENV)/bin/activate; \
		38	TEST='true' py.test --cov-report=term-missing --cov $(MODULENAME); \
		39	deactivate"
39		40
		41	.PHONY: test-ci
40	test-ci: $(VENV)	42	test-ci: $(VENV)
41	sh -c "set -e; . $(VENV)/bin/activate; TEST='true' py.test --cov-report xml --cov $(MODULENAME); deactivate"	43	sh -c "set -e; . $(VENV)/bin/activate; \
		44	TEST='true' py.test --cov-report=xml --cov $(MODULENAME); deactivate"
42		45
		46	.PHONY: pylint
43	pylint:	47	pylint:
44	pylint -f parseable $(SRC) \| tee pylint.out	48	pylint -f parseable $(SRC) \| tee pylint.out
45		49
		50	.PHONY: pep8
46	pep8:	51	pep8:
47	pep8 $(SRC)/.py $(SRC)//*.py \| tee pep8.out	52	pep8 $(SRC)/.py $(SRC)//*.py \| tee pep8.out
48		53
		54	.PHONY: ci
49	ci: test-ci pylint pep8	55	ci: test-ci pylint pep8
50		56
		57	.PHONY: docs
51	docs:	58	docs:
52	sphinx-build -b html -d docs/_build/doctrees docs/source docs/_build/html	59	sphinx-build -b html -d docs/_build/doctrees docs/source docs/_build/html
53		60
		61	.PHONY: clean
54	clean:	62	clean:
55	rm -rf dist	63	rm -rf dist
56	rm -rf taskotron-trigger.egg-info	64	rm -rf taskotron-trigger.egg-info
57	rm -rf build	65	rm -rf build
58	rm -f pep8.out	66	rm -f pep8.out
59	rm -f pylint.out	67	rm -f pylint.out
60		68
		69	.PHONY: archive
61	archive: $(SRC)-$(VERSION).tar.gz	70	archive: $(SRC)-$(VERSION).tar.gz
62		71
		72	.PHONY: $(SRC)-$(VERSION).tar.gz
63	$(SRC)-$(VERSION).tar.gz:	73	$(SRC)-$(VERSION).tar.gz:
64	git archive $(GITBRANCH) --prefix=$(SRC)-$(VERSION)/ \| gzip -c9 > $@	74	git archive $(GITBRANCH) --prefix=$(SRC)-$(VERSION)/ \| gzip -c9 > $@
65		75
		76	.PHONY: mocksrpm
66	mocksrpm: archive	77	mocksrpm: archive
67	mock -r $(BUILDTARGET) --buildsrpm --spec $(SPECFILE) --sources .	78	mock -r $(BUILDTARGET) --buildsrpm --spec $(SPECFILE) --sources .
68	cp /var/lib/mock/$(BUILDTARGET)/result/$(NVR).$(TARGETDIST).src.rpm .	79	cp /var/lib/mock/$(BUILDTARGET)/result/$(NVR).$(TARGETDIST).src.rpm .
69		80
		81	.PHONY: mockbuild
70	mockbuild: mocksrpm	82	mockbuild: mocksrpm
71	mock -r $(BUILDTARGET) --no-clean --rebuild $(NVR).$(TARGETDIST).src.rpm	83	mock -r $(BUILDTARGET) --no-clean --rebuild $(NVR).$(TARGETDIST).src.rpm
72	cp /var/lib/mock/$(BUILDTARGET)/result/$(NVR).$(TARGETDIST).noarch.rpm .	84	cp /var/lib/mock/$(BUILDTARGET)/result/$(NVR).$(TARGETDIST).noarch.rpm .
73		85
		86	#.PHONY: kojibuild
74	#kojibuild: mocksrpm	87	#kojibuild: mocksrpm
75	# koji build --scratch dist-6E-epel-testing-candidate $(NVR).$(TARGETDIST).src.rpm	88	# koji build --scratch dist-6E-epel-testing-candidate $(NVR).$(TARGETDIST).src.rpm
76		89
		90	.PHONY: nvr
77	nvr:	91	nvr:
78	@echo $(NVR)	92	@echo $(NVR)
79		93
		94	.PHONY: cleanvenv
80	cleanvenv:	95	cleanvenv:
81	rm -rf $(VENV)	96	rm -rf $(VENV)
82		97
		98	.PHONY: virtualenv
83	virtualenv: $(VENV)	99	virtualenv: $(VENV)
84		100
		101	.PHONY: $(VENV)
85	$(VENV):	102	$(VENV):
86	virtualenv --distribute --system-site-packages $(VENV)	103	virtualenv --system-site-packages $(VENV)
87	sh -c "set -e; . $(VENV)/bin/activate; pip install --force-reinstall -r requirements.txt; deactivate"	104	sh -c "set -e; . $(VENV)/bin/activate; \
		105	pip install --ignore-installed -r requirements.txt; deactivate"

View Options

README.rst

1	Taskotron Job Trigger	1		Taskotron Job Trigger
2	=====================	2		=====================
3		3
4	This is very simple code that listens for fedmsg messages of specific types and	4		This is very simple code that listens for fedmsg messages of specific types and
5	triggers jobs in taskotron in buildbot as needed.	5		triggers jobs in taskotron in buildbot as needed.
6		6
		7		Project page: https://pagure.io/taskotron/taskotron-trigger
		8
7	Installation	9		Installation
8	------------	10		------------
9		11
10	Install the rpm dependencies on the system (only works on Fedora for now)::	12		Install the rpm dependencies on the system (only works on Fedora for now)::
11		13
12	yum install fedmsg fedmsg-hub	14		$ dnf install fedmsg-hub git koji
13		15
14	Parts of the trigger do need to be installed on the local system::	16		Parts of the trigger do need to be installed on the local system::
15		17
16	python setup.py install	18		$ python setup.py install
17		19
18	Updating Production Logging Configuration	20		Updating Production Logging Configuration
19	-----------------------------------------	21		-----------------------------------------
20		22
21	In /etc/fedmsg.d/logging.py, add the following handler::	23		In ``/etc/fedmsg.d/logging.py``, add the following handler::
22		24
23	filelog={	25		filelog={
24	"class": "logging.handlers.RotatingFileHandler",	26		"class": "logging.handlers.RotatingFileHandler",
25	"formatter": "bare",	27		"formatter": "bare",
26	"level": "INFO",	28		"level": "INFO",
27	"filename": "/var/log/fedmsg/taskotron-trigger.log",	29		"filename": "/var/log/fedmsg/taskotron-trigger.log",
28	"mode": "a",	30		"mode": "a",
29	}	31		}
Show All 10 Lines		35	console::
40	},	42		},
41		43
42	Running the Trigger	44		Running the Trigger
43	-------------------	45		-------------------
44		46
45	Once the fedmsg component has been installed, run the following command in the	47		Once the fedmsg component has been installed, run the following command in the
46	working directory with triggerjob.py::	48		working directory with triggerjob.py::
47		49
48	fedmsg-hub	50		$ fedmsg-hub
49		51
50	Setting Up a Dev Environment	52		Setting Up a Dev Environment
51	----------------------------	53		----------------------------
52		54
53	Install the following packages if they aren't already installed:	55		Install the RPM dependencies mentioned in the Installation section. Then also
54		56		install the following::
55	* koji
56		57
57	* fedmsg-hub	58		$ sudo dnf install python2-virtualenv python-pip
58		59
59	Create a virtualenv for the trigger, including system site-packages to make sure	60		Create a virtualenv for the trigger, including system site-packages to make sure
60	that we get koji::	61		that we get koji::
61		62
62	virtualenv --system-site-packages env_trigger/	63		$ virtualenv --system-site-packages env_trigger/
		64		$ source env_trigger/bin/activate
63		65
64	Install requirements::	66		Install requirements::
65	source env_trigger/bin/activate
66	pip install -r requirements.txt
67		67
68	Install fedmsg and taskotron-trigger into the virtualenv after activating (so	68		$ pip install --ignore-installed -r requirements.txt
69	that it picks up the trigger in the virtualenv's site-packages)::	69
		70		Install taskotron-trigger in a development mode::
		71
		72		$ pip install -e .
		73
		74		Create a local configuration file::
		75
		76		$ cp conf/trigger.cfg.example conf/trigger.cfg
		77
		78		Edit ``conf/trigger.cfg`` and set:
70		79
71	pip install -U --force-reinstall fedmsg	80		* ``job_logging = True``
72	pip install -e .	81		* ``joblog_file`` and ``git_cache_dir`` to a path where you have write access
		82		* ``rules_template`` to a file with your trigger rules, you can use
		83		``conf/trigger_rules.yml.example`` as a starting point
		84		* ``critpath_filepath`` to a critpath definition, you can use
		85		``docker_data/critpath_whitelist.example`` as a starting point
73		86
74	Once this is all done, you can run fedmsg-hub and it will pick up the local	87		Once this is all done, you can run fedmsg-hub and it will pick up the local
75	configuration without modifying the local system.	88		configuration without modifying the local system::
		89
		90		$ fedmsg-hub
		91
		92		If you use the default trigger rules, you should see similar messages to
		93		confirm they have been enabled::
		94
		95		[ fedmsg INFO] enabled by config - jobtriggers.dist_git_commit_msg:DistGitCommitJobConsumer
		96		[ fedmsg INFO] No backlog handling. status: None, url: None
		97		[ fedmsg INFO] enabled by config - jobtriggers.koji_build_msg:KojiBuildCompletedJobConsumer
		98		[ fedmsg INFO] No backlog handling. status: None, url: None
		99		[ fedmsg INFO] enabled by config - jobtriggers.compose_complete_msg:ComposeCompletedJobConsumer
		100		[ fedmsg INFO] No backlog handling. status: None, url: None
		101		[ fedmsg INFO] enabled by config - jobtriggers.koji_tag_msg:KojiTagChangedJobConsumer
		102		[ fedmsg INFO] No backlog handling. status: None, url: None
		103
		104		After a short time of waiting you should see new events being detected, e.g.::
		105
		106		[ fedmsg INFO] triggering rpmlint for kig-16.12.2-1.fc26 on x86_64
		107		[ fedmsg INFO] StreamRunner received:
		108		item: kig-16.12.2-1.fc26
		109		item_type: koji_build
		110		taskname: rpmlint
		111		arch: x86_64
		112
		113		Running the test suite
		114		----------------------
		115
		116		From inside the virtualenv run::
76		117
		118		$ py.test

View Options

requirements.txt

		1	# This is a list of pypi packages to be installed into virtualenv. Alternatively,
		2	# you can install these as RPMs instead of pypi packages. See the dependecies
		3	# with:
		4	# $ rpmspec -q --requires taskotron-trigger.spec
		5	# $ rpmspec -q --buildrequires taskotron-trigger.spec
		6
		7	# This list is in the same order as the requirements in the specfile.
		8
		9	fedmsg[consumers]
		10	mongoquery
		11	requests
		12	Twisted
		13	PyYAML == 3.11
		14
		15	# Test suite requirements
		16
1	dingus == 0.3.4	17	dingus == 0.3.4
2	mock==1.3.0	18	mock == 1.3.0
3	moksha.hub == 1.4.6
4	munch == 2.0.2	19	munch == 2.0.2
5	psutil == 3.2.1
6	Pygments == 2.1.3
7	pytest == 2.9.2	20	pytest == 2.9.2
8	pytest-cov == 2.2.1	21	pytest-cov == 2.2.1
9	PyYAML == 3.11
10	mongoquery

View Options

setup.py

Show All 29 Lines		23	setup(
30	url='https://fedorahosted.org/fedora-qa',	30		url='https://fedorahosted.org/fedora-qa',
31	packages=['jobtriggers'],	31		packages=['jobtriggers'],
32	entry_points={	32		entry_points={
33	'console_scripts': ['jobrunner=jobtriggers.jobrunner:main'],	33		'console_scripts': ['jobrunner=jobtriggers.jobrunner:main'],
34	'moksha.consumer': moksha_consumer,	34		'moksha.consumer': moksha_consumer,
35	},	35		},
36	include_package_data=True,	36		include_package_data=True,
37	install_requires=[	37		install_requires=[
38	'fedmsg',
39	'requests',
40	],	38		],
41	cmdclass={'test': PyTest}	39		cmdclass={'test': PyTest}
42	)	40		)

View Options

taskotron-trigger.spec

1	Name: taskotron-trigger	1	Name: taskotron-trigger
2	# NOTE: if you update version, make sure to also update `setup.py`	2	# NOTE: if you update version, make sure to also update `setup.py`
3	Version: 0.4.4	3	Version: 0.4.4
4	Release: 1%{?dist}	4	Release: 2%{?dist}
5	Summary: Triggering Taskotron jobs via fedmsg	5	Summary: Triggering Taskotron jobs via fedmsg
6		6
7	License: GPLv2+	7	License: GPLv2+
8	URL: https://pagure.io/taskotron/taskotron-trigger	8	URL: https://pagure.io/taskotron/taskotron-trigger
9	Source0: https://qa.fedoraproject.org/releases/%{name}/%{name}-%{version}.tar.gz	9	Source0: https://qa.fedoraproject.org/releases/%{name}/%{name}-%{version}.tar.gz
10		10
11	BuildArch: noarch	11	BuildArch: noarch
12		12
13	Requires: fedmsg	13	Requires: fedmsg
14	Requires: fedmsg-hub	14	Requires: fedmsg-hub
15	Requires: git	15	Requires: git
16	Requires: koji	16	Requires: koji
17	Requires: python-mongoquery	17	Requires: python-mongoquery
18	Requires: python-requests	18	Requires: python-requests
19	Requires: python-twisted	19	Requires: python-twisted
20	Requires: PyYAML	20	Requires: PyYAML
21	BuildRequires: fedmsg	21	BuildRequires: fedmsg
22	BuildRequires: koji	22	BuildRequires: koji
23	BuildRequires: pytest
24	BuildRequires: python2-devel	23	BuildRequires: python2-devel
25	BuildRequires: python-dingus	24	BuildRequires: python-dingus
26	BuildRequires: python-mock	25	BuildRequires: python-mock
27	BuildRequires: python-mongoquery	26	BuildRequires: python-mongoquery
28	BuildRequires: python-munch	27	BuildRequires: python-munch
		28	BuildRequires: python2-pytest
		29	BuildRequires: python-pytest-cov
29	BuildRequires: python-setuptools	30	BuildRequires: python-setuptools
30	BuildRequires: PyYAML	31	BuildRequires: PyYAML
31		32
32	%description	33	%description
33	Triggering Taskotron jobs via fedmsg.	34	Triggering Taskotron jobs via fedmsg.
34		35
35	%prep	36	%prep
36	%setup -q	37	%setup -q
37		38
38	%check	39	%check
39	py.test testing	40	py.test
40	rm -f %{buildroot}%{_sysconfdir}/fedmsg.d/*.py{c,o}	41	rm -f %{buildroot}%{_sysconfdir}/fedmsg.d/*.py{c,o}
41		42
42	%build	43	%build
43	%{__python} setup.py build	44	%{__python} setup.py build
44		45
45	%install	46	%install
46	%{__python} setup.py install -O1 --skip-build --root %{buildroot}	47	%{__python} setup.py install -O1 --skip-build --root %{buildroot}
47		48
Show All 22 Lines
70		71
71	%dir %{_sysconfdir}/taskotron	72	%dir %{_sysconfdir}/taskotron
72	%{_sysconfdir}/fedmsg.d/taskotron-trigger.py*	73	%{_sysconfdir}/fedmsg.d/taskotron-trigger.py*
73	%config(noreplace) %{_sysconfdir}/taskotron/trigger.cfg	74	%config(noreplace) %{_sysconfdir}/taskotron/trigger.cfg
74	%config(noreplace) %{_sysconfdir}/taskotron/trigger_rules.yml	75	%config(noreplace) %{_sysconfdir}/taskotron/trigger_rules.yml
75	%config(noreplace) %{_sysconfdir}/logrotate.d/taskotron-trigger	76	%config(noreplace) %{_sysconfdir}/logrotate.d/taskotron-trigger
76		77
77	%changelog	78	%changelog
		79	* Fri Feb 10 2017 Kamil Páral <kparal@redhat.com> - 0.4.4-2
		80	- add python-pytest-cov dependency
		81	- use the same test approach we use in development (just run py.test and
		82	inherit everything from tox.ini)
		83
78	* Mon Jan 23 2017 Martin Krizek <mkrizek@redhat.com> - 0.4.4-1	84	* Mon Jan 23 2017 Martin Krizek <mkrizek@redhat.com> - 0.4.4-1
79	- Fix CLI to work with the trigger rules (D1081)	85	- Fix CLI to work with the trigger rules (D1081)
80		86
81	* Mon Oct 31 2016 Tim Flink <tflink@fedoraproject.org> - 0.4.3-1	87	* Mon Oct 31 2016 Tim Flink <tflink@fedoraproject.org> - 0.4.3-1
82	- Improve task discovery for dist-git (D1031)	88	- Improve task discovery for dist-git (D1031)
83	- Add support for triggering on dist-git commit events (D1064)	89	- Add support for triggering on dist-git commit events (D1064)
84	- Use proper dist-git namespaces (D1062)	90	- Use proper dist-git namespaces (D1062)
85	- Require mongoquery (D1068)	91	- Require mongoquery (D1068)
▲ Show 20 Lines • Show All 107 Lines • Show Last 20 Lines

View Options

tox.ini

1	# This is a common file where different test suites/linters can be configured.	1	# This is a common file where different test suites/linters can be configured.
2	# Phabricator uses this file when running `arc unit` or `arc lint`.	2	# Phabricator uses this file when running `arc unit` or `arc lint`.
3		3
4	[flake8]	4	[flake8]
5	max-line-length=99	5	max-line-length=99
6		6
7	[pep8]	7	[pep8]
8	max-line-length=99	8	max-line-length=99
9		9
10	[pytest]	10	[pytest]
11	minversion=2.0	11	minversion=2.0
12	python_functions=test should	12	python_functions=test should
13	python_files=test_* functest_*	13	python_files=test_* functest_*
14	addopts=--functional testing/	14	addopts=--functional testing/ --cov-report=term-missing --cov jobtriggers

fix deps, improve develop and test workflowClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 2880

Makefile

README.rst

requirements.txt

setup.py

taskotron-trigger.spec

tox.ini

fix deps, improve develop and test workflow
ClosedPublic
Actions