Adding task writing guide to documentation and doing a bunch of other doc cleanup. Fixes T168 and T159
ClosedPublic
Actions

Authored by tflink on May 15 2014, 6:36 PM.

Details

Reviewers

kparal
roshi

Maniphest Tasks

T159: write 'how to write tasks for taskotron' documentation
T168: Write temporary directive docs

Commits

rLTRN742c92c3ccc3: Adding task writing guide to documentation and doing a bunch of other doc…

Summary

Lots of doc cleanup that isn't directly related to T159 but needed to be done

Test Plan

built docs, viewable at http://tflink.fedorapeople.org/taskotron/initial-taskotron-docs/

Diff Detail

Lint

Lint Skipped

Unit

Unit Tests Skipped

Overall looks great - just noticed some literary changes to make before pushing.

Sorry I didn't catch these earlier :-/

docs/source/writingtasks.rst
81	"... that you keep limit..." needs to read either "keep" or "limit."
109	This sounds a little off. Perhaps, "Download rpms included in the bodhi_id argument"

changes in wording from review comments

There are some small things I'd adjust, but apart from that, thanks for great work. Once you fix some of those issues, or decide not to fix them, please push the patch. Thanks.

docs/source/index.rst
92	This should be ``foo-1.2-3.fc99`` Single backquotes are undefined in reST; in Sphinx they are used for an inline default role.
docs/source/taskyaml.rst
177	Should T142 be a link?
185–186	All of these are directives, but only some of them are called that in the section title. That's a bit confusing. Maybe remove the "directive" word and leave it just as "bodhi" and "dummy"? If we do that, "bodhi comment" would move under "bodhi" description.
188	In this case (first introduction), I think we should provide a link: `Bodhi <https://admin.fedoraproject.org/updates/>`_
205	Again, I'd provide a link to Koji.
228–229	And a link to ResultsDB, either our instance or a project page.
docs/source/writingtasks.rst
122	`%s` calls `str()`, AFAIK. Not needed to call it twice.
139	I wasn't immediately clear what "under python" means. Probably better to use `under "python:"`.
206	Again, `str` not needed IMO. More of that below.
224	`rpmfile` is not defined at this point. Or have I overlooked something?

replied to some of the comments, will update the diff shortly

docs/source/taskyaml.rst
177	it could be, sure but that's a non-rendered comment. anyone reading that and especially anyone capable of fixing it should know how to find T142
185–186	that's what I get for pasting stuff without reading it closely :-/ the bodhi directive and the bodhi_comment directive are 2 different things, though so I don't think combining them is a good idea
docs/source/writingtasks.rst
122	that's a habit I've gotten into after having been bitten too many times by expecting automagical-ness when passing a tuple or a list into string formatting. I don't see any harm in leaving it there - while bodhi_id isn't a list or tuple in this case, I'd argue it's not harming anything
224	it's a parameter for the function. the code should run with libtaskotron - it hits bugs that make the output a garbled mess but it runs as well as it can

updates for comments from review, cleaning up improper monospace syntax for sphinx, some other cleanup and clarifications

Updated rendered docs at http://tflink.fedorapeople.org/taskotron/initial-taskotron-docs/

Everything looks good, but I would add the links to koji and resultsdb @kparal noted.

Everything looks good, but I would add the links to koji and resultsdb @kparal noted.

I don't understand - what needs to be changed about the links that have already been added?

NVM - refreshing the page didn't update to the lastest diff for me. Apologies. Looks good :)

Closed by commit rLTRN742c92c3ccc3 (authored by @tflink).

docs/source/writingtasks.rst
122	Oh wow, a tuple really breaks it (a list works fine, though)! I didn't know: >>> a = (1,2) >>> print 'foo %s' % a Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: not all arguments converted during string formatting >>> print 'foo %s' % (a,) foo (1, 2) Interesting.
224	No, `rpmfiles` is a parameter for the function. `rpmfile` is undefined. I still think this should crash.

Filed T170 to fix the issue.

docs/source/writingtasks.rst
224	yeah, you're right. I forgot to pull one of the code changes I was using to test it. it should be `str(bodhi_id)` instead of rpmfile

Revision Contents

		Path
M		docs/source/devguide.rst (28 lines)
M		docs/source/index.rst (100 lines)
M		docs/source/taskyaml.rst (76 lines)
A	M	docs/source/writingtasks.rst (264 lines)
M		readme.rst (119 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	278	eb3ece6		May 15 2014, 6:36 PM	★	★
Diff 2	280	eb3ece6	- changes in wording from review comments	May 16 2014, 1:03 PM	★	★
Diff 3	281	eb3ece6	- updates for comments from review, cleaning up improper monospace syntax for sphinx, some other cleanup and clarifications	May 16 2014, 3:30 PM	★	★
Diff 4	282	1af4d1c	Commit rLTRN742c92c3ccc3d87c9e151a6b12b4bc60790f1f14	May 16 2014, 6:03 PM	★	★

Commit	Tree	Parents	Author	Summary	Date
bdfca841cb50	d1927e543cdd	b518e1f1d54e	Tim Flink	updates for comments from review, cleaning up improper monospace syntax for… (Show More…)	May 16 2014, 3:29 PM
b518e1f1d54e	7f146971b4e7	661a71cd6cc0	Tim Flink	changes in wording from review comments	May 16 2014, 1:03 PM
661a71cd6cc0	c1390c2ef177	eb3ece62ce18	Tim Flink	Adding task writing guide to documentation and doing a bunch of other doc… (Show More…)	May 14 2014, 5:15 PM

Diff 282

View Options

docs/source/devguide.rst

Show All 13 Lines
14	Source Code	14	Source Code
15	-----------	15	-----------
16		16
17	The source code for libtaskotron is in a git repository on bitbucket:	17	The source code for libtaskotron is in a git repository on bitbucket:
18	https://bitbucket.org/fedoraqa/libtaskotron	18	https://bitbucket.org/fedoraqa/libtaskotron
19		19
20	If you submit patches, please use the process of :ref:`submitting-code`.	20	If you submit patches, please use the process of :ref:`submitting-code`.
21		21
		22	.. _taskotron-bugs:
		23
22	Bugs, Issues and Tasks	24	Bugs, Issues and Tasks
23	----------------------	25	----------------------
24		26
25	We use `Phabricator <http://phabricator.org/>`_ to track issues and facilitate	27	We use `Phabricator <http://phabricator.org/>`_ to track issues and facilitate
26	code reviews for several projects related to libtaskotron and Fedora QA.	28	code reviews for several projects related to libtaskotron and Fedora QA.
27		29
28	Our phabricator instance can be found at https://phab.qadevel.cloud.fedoraproject.org/	30	Our phabricator instance can be found at https://phab.qadevel.cloud.fedoraproject.org/
29		31
▲ Show 20 Lines • Show All 71 Lines • ▼ Show 20 Line(s)
101		103
102	.. _submitting-code:	104	.. _submitting-code:
103		105
104	Submitting code	106	Submitting code
105	---------------	107	---------------
106		108
107	Libtaskotron follows the `gitflow <http://nvie.com/posts/a-successful-git-branching-model/>`_	109	Libtaskotron follows the `gitflow <http://nvie.com/posts/a-successful-git-branching-model/>`_
108	branching model and with the exception of hotfixes, all changes made should be	110	branching model and with the exception of hotfixes, all changes made should be
109	against the `develop` branch.	111	against the ``develop`` branch.
110		112
111	While not required, using the `gitflow plugin for git <https://github.com/nvie/gitflow>`_	113	While not required, using the `gitflow plugin for git <https://github.com/nvie/gitflow>`_
112	is recommended as it makes the process significantly easier. See	114	is recommended as it makes the process significantly easier. See
113	:ref:`installing-gitflow` for instructions on installing gitflow on Fedora.	115	:ref:`installing-gitflow` for instructions on installing gitflow on Fedora.
114		116
115		117
116	Start a new feature	118	Start a new feature
117	^^^^^^^^^^^^^^^^^^^	119	^^^^^^^^^^^^^^^^^^^
118		120
119	To start work on a new feature, use::	121	To start work on a new feature, use::
120		122
121	git flow feature start TXXX-short-description	123	git flow feature start TXXX-short-description
122		124
123	Where `TXXX` is the issue number in Phabricator and `short-description` is a	125	Where ``TXXX`` is the issue number in Phabricator and ``short-description`` is a
124	short, human understandable description of the change contained in this branch.	126	short, human understandable description of the change contained in this branch.
125		127
126	In general, short reviews are better than long reviews. If you can, please break	128	In general, short reviews are better than long reviews. If you can, please break
127	up features into chunks that are smaller and easier to manage.	129	up features into chunks that are smaller and easier to manage.
128		130
129		131
130	Submitting code for review	132	Submitting code for review
131	^^^^^^^^^^^^^^^^^^^^^^^^^^	133	^^^^^^^^^^^^^^^^^^^^^^^^^^
Show All 10 Lines
142	is recommended.	144	is recommended.
143		145
144	You do not need to push code anywhere public before submitting a review. Unless	146	You do not need to push code anywhere public before submitting a review. Unless
145	there is a good reason to do so (and there are very few), pushing a feature	147	there is a good reason to do so (and there are very few), pushing a feature
146	branch to origin is frowned on as it makes repository maintenance more difficult	148	branch to origin is frowned on as it makes repository maintenance more difficult
147	for no real benefit.	149	for no real benefit.
148		150
149	To submit code for review, make sure that your code has been updated with respect	151	To submit code for review, make sure that your code has been updated with respect
150	to `origin/develop` and run the following from your checked-out feature branch::	152	to ``origin/develop`` and run the following from your checked-out feature branch::
151		153
152	arc diff develop	154	arc diff develop
153		155
154	The first time that you use Arcanist, it will ask for an API key which can be	156	The first time that you use Arcanist, it will ask for an API key which can be
155	retrieved from a link contained in that prompt.	157	retrieved from a link contained in that prompt.
156		158
157	Arcanist will create a new code review on our Phabricator instance and prompt	159	Arcanist will create a new code review on our Phabricator instance and prompt
158	you for information about the testing which has been done, a description of the	160	you for information about the testing which has been done, a description of the
159	code under review and people to whom the review should be assigned. If you're	161	code under review and people to whom the review should be assigned. If you're
160	not clear on who should review your code, leave the `reviewers` section blank	162	not clear on who should review your code, leave the ``reviewers`` section blank
161	and someone will either review your code or assign the review task to someone	163	and someone will either review your code or assign the review task to someone
162	who will.	164	who will.
163		165
164		166
165	Updating code reviews	167	Updating code reviews
166	^^^^^^^^^^^^^^^^^^^^^	168	^^^^^^^^^^^^^^^^^^^^^
167		169
168	There will often be requests for changes to the code submitted for review. Once	170	There will often be requests for changes to the code submitted for review. Once
169	the requested changes have been made in your feature branch, commit them and	171	the requested changes have been made in your feature branch, commit them and
170	make sure that your branch is still up to date with respect to `origin/develop`.	172	make sure that your branch is still up to date with respect to ``origin/develop``.
171		173
172	To update the existing review, use::	174	To update the existing review, use::
173		175
174	arc diff develop --update DXXX	176	arc diff develop --update DXXX
175		177
176	Where `DXXX` is the Differential ID assigned to your review when it was	178	Where ``DXXX`` is the Differential ID assigned to your review when it was
177	originally created.	179	originally created.
178		180
179		181
180	Pushing code	182	Pushing code
181	^^^^^^^^^^^^	183	^^^^^^^^^^^^
182	.. note::	184	.. note::
183		185
184	Changes to git are very difficult to undo or cleanup once they have been	186	Changes to git are very difficult to undo or cleanup once they have been
185	pushed to a central repository. If you are not comfortable going through the	187	pushed to a central repository. If you are not comfortable going through the
186	process listed here, please ask for help.	188	process listed here, please ask for help.
187		189
188	If you run into any problems before pushing code to origin, please ask for	190	If you run into any problems before pushing code to origin, please ask for
189	help before pushing.	191	help before pushing.
190		192
191		193
192	Once the review has been accepted, the code needs to be merged into `develop`	194	Once the review has been accepted, the code needs to be merged into ``develop``
193	and pushed to `origin`.	195	and pushed to ``origin``.
194		196
195	Make sure that your local `develop` branch is up-to-date with `origin/develop`	197	Make sure that your local ``develop`` branch is up-to-date with ``origin/develop``
196	before starting the merge process, else messy commits and merges may ensue.	198	before starting the merge process, else messy commits and merges may ensue.
197	Once `develop` is up-to-date, the basic workflow to use is::	199	Once ``develop`` is up-to-date, the basic workflow to use is::
198		200
199	git checkout feature/TXXX-some-feature	201	git checkout feature/TXXX-some-feature
200	git rebase develop	202	git rebase develop
201		203
202	To merge the code into develop, use one of two commands. If the feature can be	204	To merge the code into develop, use one of two commands. If the feature can be
203	reasonably expressed in one commit (most features), use::	205	reasonably expressed in one commit (most features), use::
204		206
205	git flow feature finish --squash TXXX-some-feature	207	git flow feature finish --squash TXXX-some-feature
Show All 21 Lines
227	If you want to look at the code under review locally to run tests or test	229	If you want to look at the code under review locally to run tests or test
228	suggestions prior to posting them, use Arcanist to apply the review code.	230	suggestions prior to posting them, use Arcanist to apply the review code.
229		231
230	Make sure that your local repo is at the same base revision as the code under	232	Make sure that your local repo is at the same base revision as the code under
231	review (usually origin/develop) and run the following command::	233	review (usually origin/develop) and run the following command::
232		234
233	arc patch DXXX	235	arc patch DXXX
234		236
235	Where `DXXX` is the review id that you're interested in. Arcanist will grab the	237	Where ``DXXX`` is the review id that you're interested in. Arcanist will grab the
236	code under review and apply it to a local branch named `arcpatch-DXXX`. You can	238	code under review and apply it to a local branch named ``arcpatch-DXXX``. You can
237	then look at the code locally or make modifications.	239	then look at the code locally or make modifications.
238		240
239	Writing directives	241	Writing directives
240	==================	242	==================
241		243
242	TODO Write documentation on writing directives for libtaskotron	244	TODO Write documentation on writing directives for libtaskotron
243		245
244	Documentation	246	Documentation
▲ Show 20 Lines • Show All 93 Lines • ▼ Show 20 Line(s)
338		340
339	To be determined - likely a variation on the docstring convention	341	To be determined - likely a variation on the docstring convention
340		342
341		343
342	reST Documentation	344	reST Documentation
343	------------------	345	------------------
344		346
345	The majority of libtaskotron's documentation is in the form of reST files which	347	The majority of libtaskotron's documentation is in the form of reST files which
346	live in the `docs/source` directory in the git repository.	348	live in the ``docs/source`` directory in the git repository.
347		349
348	For more information see:	350	For more information see:
349		351
350	* `reStructuredText Primer <http://sphinx-doc.org/rest.html#rst-primer>`_	352	* `reStructuredText Primer <http://sphinx-doc.org/rest.html#rst-primer>`_
351	* `Sphinx Markup Constructs <http://sphinx-doc.org/markup/index.html#sphinxmarkup>`_	353	* `Sphinx Markup Constructs <http://sphinx-doc.org/markup/index.html#sphinxmarkup>`_
352		354
353		355
354	ReST Headings	356	ReST Headings
Show All 25 Lines

View Options

docs/source/index.rst

1	==========================	1		==========================
2	Libtaskotron Documentation	2		Libtaskotron Documentation
3	==========================	3		==========================
4		4
5	What is libtaskotron?	5		What is libtaskotron?
6	=====================	6		=====================
7		7
8	Libtaskotron is one of the core components which make up the `Taskotron`_	8		Libtaskotron is one of the core components which make up the `Taskotron`_
9	system. Libtaskotron is responsible for running tasks described in the	9		system. Libtaskotron is responsible for running tasks written in
10	:doc:`taskyaml` section.	10		:ref:`taskotron-yaml-format`.
11		11
12	While libtaskotron was designed for use with `Fedora <http://fedoraproject.org>`_,	12		While libtaskotron was designed for use with `Fedora <http://fedoraproject.org>`_,
13	the Fedora specific parts are isolated and the core should be usable with any	13		the Fedora specific parts are isolated and the core should be usable with any
14	recent Linux distribution.	14		recent Linux distribution.
15		15
16	Libtaskotron and Taskotron should be considered Alpha software. They are	16		Libtaskotron and Taskotron should be considered Alpha software. They are
17	currently under very heavy development and will likely change until we stabilize	17		currently under very heavy development and will likely change until we stabilize
18	the interfaces.	18		the interfaces.
19		19
20	Support	20		Contents
21	=======	21		--------
		22
		23		.. toctree::
		24		:maxdepth: 1
		25
		26		writingtasks
		27		taskyaml
		28		library
		29		devguide
		30
		31
		32
		33		.. _what-can-i-do-with-taskotron:
		34
		35		What Can I Do With Taskotron?
		36		=============================
		37
		38		In theory, there is little that cannot be done in Taskotron; by design, it is
		39		a system and framework for running semi-arbitrary tasks. That being said, there
		40		is a limit to what we currently support. If there is some feature or language
		41		that you'd like to see added to Taskotron, let us know through :ref:`contact-us`
		42		or filing a feature request (:ref:`taskotron-bugs`).
		43
		44		Task Languages
		45		--------------
		46
		47		Currently, the only language supported for tasks is Python. However, Taskotron
		48		is designed to work on text-based interactions between the runner and other
		49		units of work. Once the runner matures more, support for additional languages
		50		may be added.
		51
		52		Directives
		53		----------
		54
		55		A Taskotron task is made up of :ref:`taskotron-directives` which control what
		56		can be done in a task. See the list of available :ref:`taskotron-directives`
		57		to learn more about what is currently available.
		58
		59
		60		.. _install-libtaskotron:
		61
		62		Install Libtaskotron
		63		====================
		64
		65		In order to run any tasks locally, libtaskotron must first be installed. While
		66		not currently in the main Fedora repositories, it is available in a side repo
		67		which is updated as new versions are released. To enable this repo::
		68
		69		sudo wget http://copr-fe.cloud.fedoraproject.org/coprs/tflink/taskotron/repo/fedora-20-i386/tflink-taskotron-fedora-20-i386.repo -O /etc/yum.repos.d/tflink-taskotron-fedora-20-i386.repo
		70
		71		.. TODO - replace this link with a non-copr repo that's more stable over time
		72
		73		Once the repository is enabled, install libtaskotron with::
		74
		75		sudo yum install libtaskotron
		76
		77
		78		.. _running-tasks:
		79
		80		Running Tasks
		81		=============
		82
		83		Using `rpmlint <https://bitbucket.org/fedoraqa/task-rpmlint>`_ as an example,
		84		the task repository contains the following::
		85
		86		task-rpmlint/
		87		readme.rst
		88		rpmlint.py
		89		rpmlint.yml
		90
		91		Rpmlint runs off of koji builds, so to run it against the x86_64 build of
		92		``foo-1.2-3.fc99`` (substitute a real build if you're going to run this)::
			kparalUnsubmitted Not Done This should be ``foo-1.2-3.fc99`` Single backquotes are undefined in reST; in Sphinx they are used for an inline default role.
		93
		94		runtask -i foo-1.2-3.fc99 -t koji_build -a x86_64 rpmlint.yml
		95
		96		The build will be downloaded to a local tempdir, rpmlint will be run against
		97		that build and the results will be printed to the console in TAP13 format.
		98
		99
		100		.. _contact-us:
		101
		102		Contact Us
		103		==========
22	Please direct questions and comments to either the `Fedora QA Devel List	104		Please direct questions and comments to either the `Fedora QA Devel List
23	<https://admin.fedoraproject.org/mailman/listinfo/qa-devel>`_ or the #fedora-qa	105		<https://admin.fedoraproject.org/mailman/listinfo/qa-devel>`_ or the #fedora-qa
24	IRC channel on `Freenode <http://freenode.net/>`_.	106		IRC channel on `Freenode <http://freenode.net/>`_.
25		107
26		108
27		109
28	.. _Taskotron: https://fedoraproject.org/wiki/Taskotron	110		.. _Taskotron: https://fedoraproject.org/wiki/Taskotron
29		111
30	Contents
31	========
32
33	.. toctree::
34	:maxdepth: 1
35
36	devguide
37	library
38	taskyaml
39
40		112
41	Indices and tables	113		Indices and tables
42	==================	114		==================
43		115
44	* :ref:`genindex`	116		* :ref:`genindex`
45	* :ref:`modindex`	117		* :ref:`modindex`
46	* :ref:`search`	118		* :ref:`search`
47		119

View Options

docs/source/taskyaml.rst

		1		.. _taskotron-yaml-format:
		2
1	==========================	3		==========================
2	Taskotron Task YAML Format	4		Taskotron Task YAML Format
3	==========================	5		==========================
4		6
5	One of the core bits of Taskotron is the yaml file used to describe tasks to be	7		One of the core bits of Taskotron is the yaml file used to describe tasks to be
6	run. As Taskotron is still a very young project, some components of the task yaml	8		run. As Taskotron is still a very young project, some components of the task yaml
7	files are not yet used by the task runner but are useful to human readers of	9		files are not yet used by the task runner but are useful to human readers of
8	the task. Those sections will be noted and those notes updated as things change.	10		the task. Those sections will be noted and those notes updated as things change.
9		11
10	The documentation here is a description of how things are implemented but are	12		The documentation here is a description of how things are implemented but are
11	a little light on the practical creation of tasks. link to task creation howto	13		a little light on the practical creation of tasks. :ref:`writing-tasks-for-taskotron`
12	and some existing tasks are also good references:	14		and some existing tasks are also good references:
13		15
14	* `rpmlint <https://bitbucket.org/fedoraqa/task-rpmlint>`_	16		* `rpmlint <https://bitbucket.org/fedoraqa/task-rpmlint>`_
15	* `examplebodhi <https://bitbucket.org/fedoraqa/task-examplebodhi>`_	17		* `examplebodhi <https://bitbucket.org/fedoraqa/task-examplebodhi>`_
16		18
17	Task Description	19		Task Description
18	================	20		================
19		21
▲ Show 20 Lines • Show All 59 Lines • ▼ Show 20 Line(s)		80	.. note::
79		81
80	Future versions will also support specifying dependencies in other formats	82		Future versions will also support specifying dependencies in other formats
81	than RPM.	83		than RPM.
82		84
83		85
84	Task Execution	86		Task Execution
85	==============	87		==============
86		88
87	Every task is defined in the `task` block of the task yaml file. The task	89		Every task is defined in the ``task`` block of the task yaml file. The task
88	consists of one or more steps which consist of a name, directive and optional	90		consists of one or more steps which consist of a name, directive and optional
89	export of data.	91		export of data.
90		92
91	.. code-block:: yaml	93		.. code-block:: yaml
92		94
93	task:	95		task:
94	- name: using directivename for something	96		- name: using directivename for something
95	directivename:	97		directivename:
Show All 30 Lines
126	The task runner provides the following variables without a need for them to be	128		The task runner provides the following variables without a need for them to be
127	explicitly specified:	129		explicitly specified:
128		130
129	* workdir	131		* workdir
130	* jobid	132		* jobid
131		133
132	.. note::	134		.. note::
133		135
134	The `jobid` is used primarially for reporting to render proper log and job	136		The ``jobid`` is used primarially for reporting to render proper log and job
135	urls. It can be specified on the command line but isn't required for local	137		urls. It can be specified on the command line but isn't required for local
136	execution. If a `jobid` is not explicitly provided at execution time, a	138		execution. If a ``jobid`` is not explicitly provided at execution time, a
137	sane default value will be provided by the runner.	139		sane default value will be provided by the runner.
138		140
139	Created Variables	141		Created Variables
140	-----------------	142		-----------------
141		143
142	There are many situations in which the output from one step is used as input	144		There are many situations in which the output from one step is used as input
143	for another step which is executed later. In order for a step's output to be	145		for another step which is executed later. In order for a step's output to be
144	used later, it must first be exported.	146		used later, it must first be exported.
145		147
146	.. code-block:: yaml	148		.. code-block:: yaml
147		149
148	task:	150		task:
149	- name: first step	151		- name: first step
150	directive_one:	152		directive_one:
151	arg1: value1	153		arg1: value1
152	export: firststep_output	154		export: firststep_output
153		155
154	- name: second step	156		- name: second step
155	directive_two:	157		directive_two:
156	arg1: "{{ firststep_output }}"	158		arg1: "{{ firststep_output }}"
157		159
158	In this example, the output from `first step` is stored as `firststep_output`	160		In this example, the output from ``first step`` is stored as ``firststep_output``
159	and later used by `second step`. Once the `first step` is executed, its	161		and later used by ``second step``. Once the ``first step`` is executed, its
160	output is stored as a variable using the name following `export:`.	162		output is stored as a variable using the name following ``export:``.
		163
161		164
		165		.. _taskotron-directives:
162		166
163	Directives	167		Directives
164	==========	168		==========
165		169
166	The directives which are used in task steps are defined in libtaskotron. These	170		The directives which are used in task steps are defined in libtaskotron. These
167	directives are grouped into three types: preparation, execution and reporting.	171		directives are grouped into three types: preparation, execution and reporting.
		172		Below is a list of the current directives and what they can help you accomplish.
		173		This is not the complete documentation for each of these directives, but it
		174		should provide a good starting point for understanding what they do.
		175
		176		.. note::
		177
			kparalUnsubmitted Not Done Should T142 be a link?
			tflinkAuthorUnsubmitted Not Done it could be, sure but that's a non-rendered comment. anyone reading that and especially anyone capable of fixing it should know how to find T142
		178		This directive documentation is incomplete and is meant to be temporary until
		179		we resolve `some issues <https://phab.qadevel.cloud.fedoraproject.org/T142>`_
		180		around generating documenation for the directives. For now, please see the
		181		source or ask questions about directives directly.
		182
		183		bodhi_comment
		184		-------------
		185
		186		This is an interface to `bodhi <https://admin.fedoraproject.org/updates/>`_
			kparalUnsubmitted Not Done All of these are directives, but only some of them are called that in the section title. That's a bit confusing. Maybe remove the "directive" word and leave it just as "bodhi" and "dummy"? If we do that, "bodhi comment" would move under "bodhi" description.
			tflinkAuthorUnsubmitted Not Done that's what I get for pasting stuff without reading it closely :-/ the bodhi directive and the bodhi_comment directive are 2 different things, though so I don't think combining them is a good idea
		187		comments. Allows the posting of comments on a specific update as well as karma
		188		for the update.
			kparalUnsubmitted Not Done In this case (first introduction), I think we should provide a link: `Bodhi <https://admin.fedoraproject.org/updates/>`_
		189
		190		bodhi
		191		-----
		192
		193		This directive is used to download specific packages from
		194		`bodhi <https://admin.fedoraproject.org/updates/>`_.
		195
		196		createrepo
		197		----------
		198
		199		Takes a path from yaml input and creates a repository on that path.
		200
		201		dummy
		202		-----
		203
		204		Primarily a testing directive, it simply returns a status and
		205		optionally a message. Also works as a useful placeholder while
			kparalUnsubmitted Not Done Again, I'd provide a link to Koji.
		206		writing new tasks.
		207
		208		koji
		209		----
		210
		211		Provides an interface to `koji <http://koji.fedoraproject.org/koji/>`_ for
		212		downloading rpms with a specific envr or rpms with a specific tag.
		213
		214		mash
		215		----
		216
		217		Creates repositories from locally downloaded rpms.
		218
		219		python
		220		------
		221
		222		Allows the execution of a python script as part of task execution.
		223		Offers several methods of executing python code for a task.
		224
		225		resultsdb
		226		---------
168		227
169	Needs some sort of link or list of directives and their documentation	228		Provides an interface to `resultsdb <https://bitbucket.org/fedoraqa/resultsdb>`_
		229		for storing results of executed tasks.
			kparalUnsubmitted Not Done And a link to ResultsDB, either our instance or a project page.

View Options

docs/source/writingtasks.rst

This file was added.

1		.. _writing-tasks-for-taskotron:
2
3		===========================
4		Writing Tasks for Taskotron
5		===========================
6
7		While the eventual target for most tasks will be to run in a production system,
8		one of libtaskotron's features is that it can be run without the overhead of
9		a full production-like system.
10
11		To learn more about what can be done with taskotron, see :ref:`what-can-i-do-with-taskotron`.
12
13		.. warning::
14
15		Taskotron is still very young. At the moment, this document is both the
16		direction that we want to take in the near future and features that currently
17		exist in libtaskotron.
18
19		If you follow this tutorial, be aware that things will be changing. Do not
20		expect libtaskotron to have a stable API yet and be aware that the interfaces
21		may change drastically with little notice.
22
23		Once the interfaces stabilize a bit more, this warning will be edited or
24		removed.
25
26		In order to write tasks, you must :ref:`install-libtaskotron` on your local
27		machine if you aren't running from git.
28
29		Examples
30		========
31
32		examplebodhi
33		-------------
34
35		`examplebodhi task git repository <https://bitbucket.org/fedoraqa/task-examplebodhi>`_
36
37		The examplebodhi task is a trivial example which takes an update and determines
38		a PASS/FAIL results in a pseudo-random fashion. This task wasn't designed to be
39		used in production but is useful as a starting point and for development
40		purposes.
41
42		rpmlint
43		-------
44
45		`rpmlint task git repository <https://bitbucket.org/fedoraqa/task-rpmlint>`_
46
47		rpmlint is the simplest of the production tasks. It takes a koji build and runs
48		rpmlint on it, reporting results as TAP13.
49
50
51		Creating a New Task
52		===================
53
54		.. warning::
55
56		With the current libtaskotron code, the following example code does run but
57		its output does not match what is shown here. We are aware of the issue and
58		will be fixing it soon. See the `phabricator task
59		<https://phab.qadevel.cloud.fedoraproject.org/T169>`_ for details.
60
61		While running rpmlint is an easy and trivial example, that doesn't help with
62		creating something new. To write a new task, you'll need some information:
63
64		* What type of object will the task be run against?
65
66		* What will be the name of the task?
67
68		* What are the dependencies?
69
70
71		The taskotron runner works on specially formatted YAML files (see
72		:ref:`taskotron-yaml-format` for details) which describe the task that is to
73		be executed. In addition to the task YAML file, any other code or resources needed
74		for task execution should be accessible by the runner.
75
76		A directory layout could look like::
77
78		mytask/
79		mytask.yml
80		mytask.py
81		someresource.txt
	roshiUnsubmitted Not Done "... that you keep limit..." needs to read either "keep" or "limit."
82
83		.. note::
84
85		Taskotron was designed to work with tasks stored in git repostitories and
86		while this is not strictly required for the local execution case, it is
87		highly recommended that you limit tasks to one per directory.
88
89		Non-Executed Task Information
90		-----------------------------
91
92		For this example, let's write a task that runs on bodhi ids. Looking at the
93		non-executed task data needed in :file:`mytask.yml`:
94
95		.. code-block:: yaml
96
97		---
98		name: mytask
99		desc: do something stupendous
100		maintainer: somefasuser
101
102		input:
103		args: arch, bodhi_id
104
105		environment:
106		rpm:
107		- libtaskotron
108
109
	roshiUnsubmitted Not Done This sounds a little off. Perhaps, "Download rpms included in the bodhi_id argument"
110		Executed Task Information
111		-------------------------
112
113		Let's have this new task do the following:
114
115		* download the rpms contained in the indicated bodhi update
116
117		* run the function ``run_mytask`` in :file:`mytask.py` using a list of the
118		downloaded rpms as input
119
120		* report the results to resultsdb
121
122
	kparalUnsubmitted Not Done `%s` calls `str()`, AFAIK. Not needed to call it twice.
	tflinkAuthorUnsubmitted Not Done that's a habit I've gotten into after having been bitten too many times by expecting automagical-ness when passing a tuple or a list into string formatting. I don't see any harm in leaving it there - while bodhi_id isn't a list or tuple in this case, I'd argue it's not harming anything
	kparalUnsubmitted Not Done Oh wow, a tuple really breaks it (a list works fine, though)! I didn't know: >>> a = (1,2) >>> print 'foo %s' % a Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: not all arguments converted during string formatting >>> print 'foo %s' % (a,) foo (1, 2) Interesting.
123		.. code-block:: python
124
125		# this is a trivial example that doesn't do everything we want it to do, more
126		# details will be added later in the example
127		def run_mytask(rpmfiles, bodhi_id):
128		print "Running mytask on %s" % (str(bodhi_id))
129
130
131		.. code-block:: yaml
132
133		task:
134		# we can use bodhi_id and arch as variables here because they're defined
135		# for us by the runner which also verifies that all required input listed
136		# above is present before executing the task
137		- name: download bodhi update
138		bodhi:
139		action: download
	kparalUnsubmitted Not Done I wasn't immediately clear what "under python" means. Probably better to use `under "python:"`.
140		bodhi_id: "{{ bodhi_id }}"
141		arch: "{{ arch }}"
142		export: bodhi_downloads
143
144
145		# notice how the third item under python is 'rpmfiles' and this matches
146		# the arg name in mytask.py. Anything outside of 'file' and 'callable'
147		# are passed in as kwargs to the specified python callable.
148		- name: run mytask
149		python:
150		file: mytask.py
151		callable: run_mytask
152		rpmfiles: "{{ bodhi_downloads }}"
153		bodhi_id: "{{ bodhi_id }}"
154		export: mytask_output
155
156
157		# this assumes that the output from mytask is valid TAP
158		- name: report mytask results to resultsdb
159		resultsdb:
160		results: "{{ mytask_output }}"
161		checkname: 'mytask'
162
163		.. this needs to be fixed, this output is doctored since it doesn't actually
164		work.
165
166		We can execute this new task using::
167
168		$ python runtask.py -i foo-1.2-3.fc99 -t bodhi_id -a x86_64 ../task-mytask/mytask.yml
169
170
171		The output from this task should end with::
172
173		Running mytask on foo-1.2-3.fc99.x86_64.rpm
174
175		[libtaskotron:logger.py:34] 2014-05-14 21:30:30 CRITICAL Traceback (most recent call last):
176		File "runtask.py", line 4, in <module>
177		runner.main()
178		File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/runner.py", line 192, in main
179		task_runner.run()
180		File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/runner.py", line 31, in run
181		self.do_actions()
182		File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/runner.py", line 105, in do_actions
183		self.do_single_action(action)
184		File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/runner.py", line 94, in do_single_action
185		self.envdata)
186		File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/directives/resultsdb_directive.py", line 136, in process
187		raise TaskotronDirectiveError("Failed to load 'results': %s" % e.message)
188		TaskotronDirectiveError: Failed to load 'results': Failed to parse TAP contents: Missing plan in the TAP source
189
190		We're not actually passing TAP output from our task, so let's get that fixed.
191
192		Actually Doing Something
193		------------------------
194
195		The Python code for ``mytask.py`` above doesn't really do anything at all, so let's
196		make it do something:
197
198		* Change the task to create a result from the input bodhi update id and list
199		all of the rpms contained in that update as details.
200
201		There is a bit of complexity to the `TAP13 specification <http://podwiki.hexten.net/TAP/TAP13.html?page=TAP13>`_
202		but generating TAP from python tasks in Taskotron isn't very difficult.
203
204		Our original task looked like:
205
206		.. code-block:: python
	kparalUnsubmitted Not Done Again, `str` not needed IMO. More of that below.
207
208		# this is a trivial example that doesn't do everything we want it to do, more
209		# details will be added later in the example
210		def run_mytask(rpmfiles):
211		for rpmfile in rpmfiles:
212		print "Running mytask on %s" % (str(rpmfile))
213
214		To create valid TAP, we want to populate a :py:class:`libtaskotron.check.CheckDetail`
215		object and use :py:meth:`libtaskotron.check.export_TAP` to generate valid TAP13
216		with the data required for reporting.
217
218		.. code-block:: python
219
220		from libtaskotron import check
221
222		def run_mytask(rpmfiles, bodhi_id):
223		"""run through all passed in rpmfiles and emit TAP13 saying they all passed"""
224
	kparalUnsubmitted Not Done `rpmfile` is not defined at this point. Or have I overlooked something?
	tflinkAuthorUnsubmitted Not Done it's a parameter for the function. the code should run with libtaskotron - it hits bugs that make the output a garbled mess but it runs as well as it can
	kparalUnsubmitted Not Done No, `rpmfiles` is a parameter for the function. `rpmfile` is undefined. I still think this should crash.
	tflinkAuthorUnsubmitted Not Done yeah, you're right. I forgot to pull one of the code changes I was using to test it. it should be `str(bodhi_id)` instead of rpmfile
225		print "Running mytask on %s" % (str(bodhi_id))
226
227		details = []
228		result = 'PASSED'
229		summary = 'mycheck %s for %s' % (result, str(bodhi_id))
230		detail = check.CheckDetail(rpmfile, check.ReportType.BODHI_UPDATE,
231		result, summary)
232		for rpmfile in rpmfiles:
233		detail.store(rpmfile, False)
234
235		return check.export_TAP(detail)
236
237		Now if we run the task we get output that ends with::
238
239		[libtaskotron:resultsdb_directive.py:123] 2014-05-15 09:17:44 INFO Reporting to ResultsDB is disabled.
240		INFO:libtaskotron:Reporting to ResultsDB is disabled.
241		[libtaskotron:runner.py:198] 2014-05-15 09:17:44 INFO
242		TAP version 13
243		1..1
244		ok - $CHECKNAME for Bodhi update openvswitch-2.1.2-1.fc19
245		---
246		details:
247		output: " foo-1.2-3.fc99.x86_64.rpm"
248		item: foo-1.2-3.fc99
249		outcome: PASSED
250		summary: mycheck PASSED for foo-1.2-3.fc99
251		type: bodhi_update
252		...
253
254		.. note::
255		Reporting is disabled in the default configuration for Taskotron. While it is
256		possible to set up a local resultsdb instance to check reporting, we want
257		to make it easier than that and will update these docs when that easier method
258		is available.
259
260		For now, if the resultsdb directive doesn't throw exceptions, it's reasonable
261		to assume that the TAP output was constructed correctly and is reasonably valid.
262
263
264

View Options

readme.rst

		1		============
1	libtaskotron	2		libtaskotron
2	============	3		============
3		4
4	This is a proof-of-concept for libtaskotron. The purpose of this is to illustrate	5		libtaskotron is a library for running automated tasks as part of the
5	where we're going with taskotron and get input on any perceived problems early on	6		`Taskotron <https://fedoraproject.org/wiki/Taskotron>` system. It is in very
6	before anything is final.	7		early stages of development with the initial objective of (but not limited to)
7		8		replacing AutoQA for automating selected package checks in Fedora.
8	Please keep in mind that this is proof-of-concept code - there are a lot of things	9
9	which are hard-coded and nothing is final. Please don't develop against this code	10		Libtaskotron should be considered as very early alpha code - the features
10	unless you've talked to us first.	11		contained are generally funcitonal but we will likely continue to make major
		12		changes in how libtaskotron works.
11		13
12	Please direct questions and comments to either `#fedora-qa` on freenode or the	14		Please direct questions and comments to either `#fedora-qa` on freenode or the
13	`qa-devel mailing list <https://admin.fedoraproject.org/mailman/listinfo/qa-devel>`_.	15		`qa-devel mailing list <https://admin.fedoraproject.org/mailman/listinfo/qa-devel>`_.
14		16
15	For more information, visit `Taskotron wiki page <https://fedoraproject.org/wiki/Taskotron>`_.	17		For more information, please reference the online documentation:
		18
		19		* `Development Version <http://libtaskotron.readthedocs.org/en/develop/>`_
		20
		21		* `Latest Release <http://libtaskotron.readthedocs.org/en/latest/`_
16		22
17	Installing the Code	23
18	-------------------	24		Installing a Development Environment
		25		====================================
19		26
20	For the moment, libtaskotron can't be fully installed by either pip or rpm and	27		For the moment, libtaskotron can't be fully installed by either pip or rpm and
21	needs a bit of both for now.	28		needs a bit of both for now.
22		29
23	Some of the packages required for using libtaskotron are not yet available from	30		Some of the packages required for using libtaskotron are not yet available from
24	standard Fedora repos and the taskotron-copr repo is required::	31		standard Fedora repos and the taskotron-copr repo is required::
25		32
26	sudo wget http://copr-fe.cloud.fedoraproject.org/coprs/tflink/taskotron/repo/fedora-20-i386/tflink-taskotron-fedora-20-i386.repo -O /etc/yum.repos.d/tflink-taskotron-fedora-20-i386.repo	33		sudo wget http://copr-fe.cloud.fedoraproject.org/coprs/tflink/taskotron/repo/fedora-20-i386/tflink-taskotron-fedora-20-i386.repo -O /etc/yum.repos.d/tflink-taskotron-fedora-20-i386.repo
Show All 25 Lines		57	use a standard installation, if you want to just use this project::
52	pip install .	59		pip install .
53		60
54	Or, if you intend to work on this project, you can install it in the editable	61		Or, if you intend to work on this project, you can install it in the editable
55	mode. This way you don't need to reinstall the project every time you make some	62		mode. This way you don't need to reinstall the project every time you make some
56	changes to it, the code changes are reflected immediately::	63		changes to it, the code changes are reflected immediately::
57		64
58	pip install -e .	65		pip install -e .
59		66
60	Running the Demo
61	----------------
62		67
63	For now, the only supported task is `rpmlint <https://bitbucket.org/fedoraqa/task-rpmlint>`_.	68		Configuration
		69		=============
		70
		71		The `libtaskotron-config` package installs config files with default values
		72		into `/etc/taskotron`. To change those default values, you can either change
		73		the files in `/etc/taskotron` or you can create config files inside your
		74		checkout with::
		75
		76		cp conf/taskotron.yaml.example conf/taskotron.yaml
		77		cp conf/yumrepoinfo.conf.exampe conf/yumrepoinfo.conf
		78
		79		The configuration files in `./conf` take precedence over anything in `/etc/`,
		80		so make sure that you're editing the correct file if you create local copies.
		81
		82
		83		Running a Task
		84		==============
		85
		86		A relatively simple example task is `rpmlint <https://bitbucket.org/fedoraqa/task-rpmlint>`_.
64		87
65	To run that task against a koji build with envr `<envr>` for some arch `<arch>`	88		To run that task against a koji build with envr `<envr>` for some arch `<arch>`
66	, do the following::	89		, do the following::
67		90
68	git clone https://bitbucket.org/fedoraqa/task-rpmlint.git	91		git clone https://bitbucket.org/fedoraqa/task-rpmlint.git
69	runtask -i <envr> -t koji_build -a <arch> task-rpmlint/rpmlint.yml	92		runtask -i <envr> -t koji_build -a <arch> task-rpmlint/rpmlint.yml
70		93
71	This will download the `<envr>` from koji into a temp directory under `/var/tmp/`,	94		This will download the `<envr>` from koji into a temp directory under `/var/tmp/`,
72	run rpmlint on the downloaded rpms and print output in TAP format to stdout.	95		run rpmlint on the downloaded rpms and print output in TAP format to stdout.
73		96
74	Example::	97		Example::
75		98
76	runtask -i xchat-2.8.8-21.fc20 -t koji_build -a x86_64 task-rpmlint/rpmlint.yml	99		runtask -i xchat-2.8.8-21.fc20 -t koji_build -a x86_64 task-rpmlint/rpmlint.yml
77		100
78	Task Description Format
79	-----------------------
80
81	The task description format is mostly yaml with support for variable replacement::
82
83	---
84	name: rpmlint
85	desc: download specified rpms and run rpmlint on them
86	maintainer: tflink
87
88	input:
89	args: envr,arch
90
91	environment:
92	rpm:
93	- rpmlint
94	- libtaskbot
95
96	task:
97	- name: download rpms from koji
98	koji:
99	action: download
100	envr: "{{ envr }}"
101	arch: "{{ arch }}"
102
103	- name: run rpmlint on downloaded rpms
104	python:
105	file: run_rpmlint.py
106	callable: run
107	workdir: "{{ workdir }}"
108	export: rpmlint_output
109
110	In this case, `envr` and `arch` are passed in to the runner as an argument and
111	`workdir` is the directory in which the task is running.
112		101
113	For the moment, what you see in the rpmlint task is the limit of what is supported	102		Running the Test Suite
114	for task description syntax.	103		======================
115
116	Running Test Suite
117	------------------
118		104
119	You can run the included test suite of unit and functional tests.	105		You can run the included test suite of unit and functional tests.
120	From the root checkout directory, execute::	106		From the root checkout directory, execute::
121		107
122	py.test testing/	108		py.test testing/
123		109
124	to run unit tests, or::	110		to run unit tests, or::
125		111
126	py.test --functional testing/	112		py.test --functional testing/
127		113
128	to run both unit and functional tests (recommended). Make sure to use the	114		to run both unit and functional tests (recommended). Make sure to use the
129	virtualenv-provided version of py.test, not the system one, otherwise you'll	115		virtualenv-provided version of py.test, not the system one, otherwise you'll
130	encounter import errors.	116		encounter import errors.
131		117
132
133	You can also see a test coverage of the code, just run::	118		You can also see a test coverage of the code, just run::
134		119
135	py.test --functional --cov-report term-missing --cov libtaskotron testing/	120		py.test --functional --cov-report term-missing --cov libtaskotron testing/
136		121
137	A nice HTML-based representation is available if you use `--cov-report html`	122		A nice HTML-based representation is available if you use `--cov-report html`
138	parameter instead.	123		parameter instead.
139		124
140	If you write new tests, be sure to run this to see whether the code is	125		If you write new tests, be sure to run this to see whether the code is
141	sufficiently covered by your tests.	126		sufficiently covered by your tests.
142
143	Building Documentation
144	----------------------
145
146	Libtaskotron's documentation is written in `reStructuredText
147	<http://docutils.sourceforge.net/rst.html>`_ and built using `Sphinx
148	<http://sphinx-doc.org/>`_.
149
150	The documentation is easy to build once deps are installed::
151
152	yum install python-sphinx
153
154	To actually build the documentation::
155
156	cd docs/
157	make html
158
159	Configuration
160	-------------
161
162	The `libtaskotron-config` package installs config files with default values
163	into `/etc/taskotron`. To change those default values, you can either change
164	the files in `/etc/taskotron` or you can create config files inside your
165	checkout with::
166
167	cp conf/taskotron.yaml.example conf/taskotron.yaml
168	cp conf/yumrepoinfo.conf.exampe conf/yumrepoinfo.conf
169
170	The configuration files in `./conf` take precedence over anything in `/etc/`,
171	so make sure that you're editing the correct file if you create local copies.

Adding task writing guide to documentation and doing a bunch of other doc cleanup. Fixes T168 and T159ClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 282

docs/source/devguide.rst

docs/source/index.rst

docs/source/taskyaml.rst

docs/source/writingtasks.rst

readme.rst

Adding task writing guide to documentation and doing a bunch of other doc cleanup. Fixes T168 and T159
ClosedPublic
Actions