This started as simple bike-shedding to make more sense in naming (so
everything is not named "Trigger"), but it went further :D
The main change here is, what I call "configurable trigger" - at the moment,
every time we want to add support for even the most basic new task (like the
package-specific task for docker), changes are needed in the trigger's source
code.
These changes add a concept of a "rules engine", that decides what tasks to
schedule based on data extacted from the received FedMessage, and a set of rules.
The rules-engine is YAML, in a format like this::
- do: - {tasks: [depcheck, upgradepath]} when: {message_type: KojiTagChanged} - do: - {tasks: [dockerautotest]} when: {message_type: KojiBuildCompleted, name: docker} - do: - {tasks: [abicheck]} when: message_type: KojiBuildCompleted name: $in: ${critpath_pkgs} $nin: ['docker'] # critpath excludes
The rules are split in two parts when and do, the when clause is
a mongo query that will get evaluated against the dataset provided by
the FedMsg consumer. For example, the KojiBuildCompletedJobTrigger now
publishes this (values are fake, to make it more descriptive::
message_data = { "_msg": {...snipped...}, "message_type": "KojiBuildCompleted", "item": "docker-1.9.1-6.git6ec29ef.fc23", "item_type": "koji_build", "name": "docker", "version": "1.9.1-6.git6ec29ef", "release": "fc23", "critpath_pkgs": [..., "docker", ...] "distgit_branch": "f23", }
So taking the rules, and the data, going from the top:
- First rule's when is False as message_type is not KojiTagChanged
- Second rule is True because both the message_type and name in the when clause match the data
- Third rule does _not_ schedule anything, because even though docker is in critpath_pkgs, it also is part of the critpath excludes list, and so the rule is ignored
The when clauses are in fact mongo queries https://docs.mongodb.com/manual/tutorial/query-documents/,
evaluated using a Python library that implements it for querying Python objects.
The rules engine then takes the do clauses of the 'passed' rules, and
produces arguments for the trigger_tasks() calls. By default, item, and
item_type are taken from the message_data, arches is set to
config.valid_arches, and then all the key/values from the do's body are
added on top. This means, that we can have a task, that for example forces
an architecture different than default::
- do: - {tasks: [awesome_arm_check], arches: [armhfp]} when: {message_type: KojiBuildCompleted}
The do clause can have multiple items in it, so something like this is
possible::
- do: - {tasks: [rpmlint]} - {tasks: [awesome_arm_check], arches: [armhfp]} when: {message_type: KojiBuildCompleted}
Triggering rpmlint on the default architectures, and awesome_arm_check
on armhfp for each package built in Koji.
This means, that when we want to trigger new (somewhat specific) tasks,
no changes are needed in the trigger's code, but just in the configuration,
to alter the rules. If we come to the point where more functionality is
needed, than it obviously calls for changes in the underlying code, in order
to add more key/values to the data provided by the Fedmsg consumer, or
adding more general functionality overall.
A good example of this is the dist-git style tasks problem. To solve it
I have added a new command ($discover)to the do section, that crawls the
provided git repo/branch, and schedules jobs for all runtask.yml's found::
- do: - {$discover: {repo: 'http://pkgs.fedoraproject.org/git/rpms-checks/${name}.git', branch: '${distgit_branch}'}} when: {message_type: KojiBuildCompleted}
In the bigger picture, this 'rules engine' functionality can be used to
make (for example) a web interface, that allows creating/altering the rules,
instead of changing the config file (the rules can as easily be taken from
a database, as from the config file), or even to provide a per-user triggering
capability - we could add a piece of code, that checks (selected) users'
Fedorapeople profile for a file, that contains rules in this format, and
then could simply run the engine on those rules+data from Fedmsg to decide
whether the user-defined tasks should be run.
line too long (113 > 99 characters)