Differential D1148

docs: add base-design documentation
ClosedPublic
Actions

Authored by kparal on Feb 22 2017, 12:43 PM.

Details

Reviewers
mkrizek
jskladan
Group Reviewers
taskotron-trigger
Commits
rTRGRacff4f36bd33: docs: add base-design documentation
Summary

The document was created by jskladan. Let's put it into the project.

Test Plan

none

Diff Detail

Repository
rTRGR taskotron-trigger
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
kparal retitled this revision from to docs: add base-design documentation.Feb 22 2017, 12:43 PM
kparal updated this object.
kparal edited the test plan for this revision. (Show Details)
kparal added reviewers: taskotron-trigger, jskladan, mkrizek.
jskladan accepted this revision.Feb 23 2017, 9:02 AM
This revision is now accepted and ready to land.Feb 23 2017, 9:02 AM
kparal updated this revision to Diff 2926.Feb 23 2017, 9:05 AM
  • use ascii quotes
Closed by commit rTRGRacff4f36bd33: docs: add base-design documentation (authored by Kamil Páral <kparal@redhat.com>). · Explain WhyFeb 23 2017, 9:11 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathPackages
MREADME.rst (9 lines)
AMdocs/base-design.rst (99 lines)

Diff 2927

View Options

README.rst

1Taskotron Job Trigger 1​Taskotron Job Trigger
2===================== 2​=====================
3 3​
4This is very simple code that listens for fedmsg messages of specific types and 4​Taskotron Trigger listens for fedmsg messages of specific types and triggers
5triggers jobs in taskotron in buildbot as needed. 5​Taskotron jobs (currently using Buildbot) as needed.
6 6​
7Project page: https://pagure.io/taskotron/taskotron-trigger 7​Project page: https://pagure.io/taskotron/taskotron-trigger
8 8​
9​Documentation
10​-------------
11​
12​Look into ``docs/`` directory.
13​
9Installation in Production 14​Installation in Production
10-------------------------- 15​--------------------------
11 16​
12Install the rpm dependencies on the system (only works on Fedora for now):: 17​Install the rpm dependencies on the system (only works on Fedora for now)::
13 18​
14 $ sudo dnf install fedmsg-hub git koji 19​ $ sudo dnf install fedmsg-hub git koji
15 20​
16Then install the trigger. Please note that the following installation is 21​Then install the trigger. Please note that the following installation is
▲ Show 20 Lines • Show All 113 Lines • Show Last 20 Lines
View Options

docs/base-design.rst

  • This file was added.
1​Taskotron-trigger base design
2​=============================
3​
4​Trigger consists of a set of *Fedmsg consumers*, *Triggering engine*, and
5​*Runners*.
6​
7​
8​Fedmsg consumers
9​----------------
10​
11​*Fedmsg consumers* are the "input source" for trigger - we listen on the fedmsg
12​hub for a known set of message topics (compose completed, distgit commit, koji
13​build, koji tag, pagure commit). When a message is received, the data is
14​parsed, and additional information is/can be gathered from external sources
15​(e.g. getting the list of critpath path packages in koji build trigger).
16​
17​Example data from the koji build trigger::
18​
19​ data = {
20​ "_msg": {... the raw fedmsg },
21​ "message_type": "KojiBuildPackageCompleted",
22​ "item": "stage-4.1.1-3.fc18",
23​ "item_type": "koji_build",
24​ "name": "stage",
25​ "version": "4.1.1.",
26​ "release": "3.fc18",
27​ "critpath_pkgs": [... list of critpath package names …],
28​ "distgit_branch": "f18",
29​ }
30​
31​
32​Triggering engine
33​-----------------
34​
35​The data from the consumer are passed into the *Triggering engine*, which takes
36​a list of rules + the data on input, and returns a list of tasks that are
37​supposed to be triggered based on that. Behind the curtains, the rules are
38​mongo-db-style queries, that are executed over the consumer-provided data. If
39​the query "passes", it means that the data match the rule, and the respective
40​tasks are to be executed.
41​
42​Some example rules for the Trigger engine::
43​
44​ - when:
45​ message_type: KojiBuildPackageCompleted
46​ do:
47​ - {tasks: [awesometask, cooltask]}
48​
49​ - when:
50​ message_type: KojiBuildPackageCompleted
51​ name: firefox
52​ do:
53​ - {tasks: [ff_testsuite]}
54​
55​ - when:
56​ message_type: KojiBuildPackageCompleted
57​ name:
58​ $in: ${critpath_pkgs}
59​ do:
60​ - {tasks: [othertask]}
61​
62​ - when:
63​ message_type: DistGitCommit
64​ do:
65​ - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-${namespace}/${repo}.git', branch: "${distgit_branch}"}}
66​
67​With these rules, and the data provided by the Fedmsg Consumer, ``awesometask``
68​and ``cooltask`` would be executed, because the rules (``when`` section) match
69​all the constraints (``message_type`` equals ``KojiBuildPackageCompleted``).
70​
71​The ``ff_testsuite`` is not ran, as the ``name`` field is different (``stage``
72​does not equal ``firefox``).
73​
74​The next example shows "dynamic constraints" - the ``${critpath_pkgs}`` is
75​populated with the value of the ``critpath_pkgs`` value passed in the data
76​structure prior to the rules being tested. So the ``othertask`` is executed
77​only if the package name is in the list of critpath packages.
78​
79​And in the last case, instead of having the task-name hardcoded, the trigger
80​is able to search a git repo of given url (here it is inferred from the data)
81​for ``runtask.yml`` (the Taskotron formula) files, and schedules those as jobs,
82​using runners.
83​
84​
85​Runners
86​-------
87​
88​The last bit here are the *Runners* - at the moment we have *StreamRunner*
89​(basically just an echo-runner that puts what it received on stdout and does
90​nothing), and *BuildbotRunner*, which takes care of scheduling the runs in
91​Buildbot.
92​
93​The runners, in general are the interface between the Trigger and the execution
94​engine - be it Buildbot, Jenkins, Beaker, or any other thing you like. They are
95​in charge of doing the right (e.g. RESTful) calls, and passing the right data,
96​so the actual tasks can be executed.
97​
98​At the moment, we only allow for using one Runner for all the tasks, based on
99​the general Trigger configuration.