PoC of "Better" Trigger
AbandonedPublic

Authored by jskladan on May 31 2016, 1:28 PM.

Details

Reviewers
None
Group Reviewers
taskotron-trigger
Summary

This started as simple bike-shedding to make more sense in naming (so
everything is not named "Trigger"), but it went further :D

The main change here is, what I call "configurable trigger" - at the moment,
every time we want to add support for even the most basic new task (like the
package-specific task for docker), changes are needed in the trigger's source
code.

These changes add a concept of a "rules engine", that decides what tasks to
schedule based on data extacted from the received FedMessage, and a set of rules.

The rules-engine is YAML, in a format like this::

- do:
  - {tasks: [depcheck, upgradepath]}
  when: {message_type: KojiTagChanged}
- do:
  - {tasks: [dockerautotest]}
  when: {message_type: KojiBuildCompleted, name: docker}
- do:
  - {tasks: [abicheck]}
  when:
    message_type: KojiBuildCompleted
    name:
      $in: ${critpath_pkgs}
      $nin: ['docker'] # critpath excludes

The rules are split in two parts when and do, the when clause is
a mongo query that will get evaluated against the dataset provided by
the FedMsg consumer. For example, the KojiBuildCompletedJobTrigger now
publishes this (values are fake, to make it more descriptive::

message_data = {
        "_msg": {...snipped...},
        "message_type": "KojiBuildCompleted",
        "item": "docker-1.9.1-6.git6ec29ef.fc23",
        "item_type": "koji_build",
        "name": "docker",
        "version": "1.9.1-6.git6ec29ef",
        "release": "fc23",
        "critpath_pkgs": [..., "docker", ...]
        "distgit_branch": "f23",
        }

So taking the rules, and the data, going from the top:

  1. First rule's when is False as message_type is not KojiTagChanged
  2. Second rule is True because both the message_type and name in the when clause match the data
  3. Third rule does _not_ schedule anything, because even though docker is in critpath_pkgs, it also is part of the critpath excludes list, and so the rule is ignored

The when clauses are in fact mongo queries https://docs.mongodb.com/manual/tutorial/query-documents/,
evaluated using a Python library that implements it for querying Python objects.

The rules engine then takes the do clauses of the 'passed' rules, and
produces arguments for the trigger_tasks() calls. By default, item, and
item_type are taken from the message_data, arches is set to
config.valid_arches, and then all the key/values from the do's body are
added on top. This means, that we can have a task, that for example forces
an architecture different than default::

- do:
  - {tasks: [awesome_arm_check], arches: [armhfp]}
  when: {message_type: KojiBuildCompleted}

The do clause can have multiple items in it, so something like this is
possible::

- do:
  - {tasks: [rpmlint]}
  - {tasks: [awesome_arm_check], arches: [armhfp]}
  when: {message_type: KojiBuildCompleted}

Triggering rpmlint on the default architectures, and awesome_arm_check
on armhfp for each package built in Koji.

This means, that when we want to trigger new (somewhat specific) tasks,
no changes are needed in the trigger's code, but just in the configuration,
to alter the rules. If we come to the point where more functionality is
needed, than it obviously calls for changes in the underlying code, in order
to add more key/values to the data provided by the Fedmsg consumer, or
adding more general functionality overall.

A good example of this is the dist-git style tasks problem. To solve it
I have added a new command ($discover)to the do section, that crawls the
provided git repo/branch, and schedules jobs for all runtask.yml's found::

- do:
  - {$discover: {repo: 'http://pkgs.fedoraproject.org/git/rpms-checks/${name}.git', branch: '${distgit_branch}'}}
  when: {message_type: KojiBuildCompleted}

In the bigger picture, this 'rules engine' functionality can be used to
make (for example) a web interface, that allows creating/altering the rules,
instead of changing the config file (the rules can as easily be taken from
a database, as from the config file), or even to provide a per-user triggering
capability - we could add a piece of code, that checks (selected) users'
Fedorapeople profile for a file, that contains rules in this format, and
then could simply run the engine on those rules+data from Fedmsg to decide
whether the user-defined tasks should be run.

Test Plan

Added testcases to cover the changes, but have not yet run the code for real

Diff Detail

Repository
rTRGR taskotron-trigger
Branch
pony
Lint
Lint ErrorsExcuse: PoC me no cares now
SeverityLocationCodeMessage
Errortesting/test_jobtrigger.py:1F401flake8 1
Warningjobtriggers/config.py:51E501flake8 100
Warningjobtriggers/jobtrigger.py:10E302flake8 1
Warningjobtriggers/jobtrigger.py:27E302flake8 1
Warningjobtriggers/jobtrigger.py:36E302flake8 1
Warningjobtriggers/jobtrigger.py:64E126flake8 21
Warningjobtriggers/jobtrigger.py:70E225flake8 72
Warningjobtriggers/jobtrigger.py:78E501flake8 100
Warningjobtriggers/jobtrigger.py:92E303flake8 5
Warningjobtriggers/jobtrigger.py:109E303flake8 5
Warningjobtriggers/koji_build_msg.py:46E221flake8 35
Warningjobtriggers/koji_build_msg.py:53E222flake8 25
Warningjobtriggers/koji_build_msg.py:56E126flake8 17
Warningjobtriggers/koji_build_msg.py:67W391flake8 1
Warningjobtriggers/utils.py:80W391flake8 1
Warningtesting/functest_cli.py:29E501flake8 100
Warningtesting/functest_cli.py:30E501flake8 100
Warningtesting/test_compose_trigger.py:45E501flake8 100
Warningtesting/test_compose_trigger.py:46E501flake8 100
Warningtesting/test_jobtrigger.py:12E302flake8 1
Warningtesting/test_jobtrigger.py:15E302flake8 1
Warningtesting/test_jobtrigger.py:30E501flake8 100
Warningtesting/test_jobtrigger.py:49E303flake8 5
Warningtesting/test_jobtrigger.py:53E501flake8 100
Warningtesting/test_jobtrigger.py:118E501flake8 100
Unit
Unit Tests OK
Build Status
Buildable 637
Build 637: arc lint + arc unit
jskladan retitled this revision from to PoC of "Better" Trigger.May 31 2016, 1:28 PM
jskladan updated this object.
jskladan edited the test plan for this revision. (Show Details)
jskladan added a reviewer: taskotron-trigger.
jskladan updated this object.May 31 2016, 1:33 PM
jskladan abandoned this revision.Aug 9 2016, 10:57 AM

abanodoned in favor of D963