Differential D1150

Decouple ExecDB from Buildbot
Needs ReviewPublic
Actions

Authored by jskladan on Feb 23 2017, 12:35 PM.

Details

Reviewers
lbrabec
kparal
jsedlak
tflink
mkrizek
Maniphest Tasks
T951: Decouple ExecDB from Buildbot
Summary

Instead of consuming the Buildbot push notification stream, ExecDB provides a comprehensive API to

  • create new job (returns uuid)
  • update job state (running, finished, ...)
  • create and update steps (set to finished, duration, outcome)

The default behavior tries to make stuff as reasonable as possibe:

  • creating a new step marks the previous one as succesfully completed (if not alredy marked as finished)
    • so does setting the job to finished
  • on top of setting the job's outcome to a specific value, there is _COMPUTED_ special value that inferres the outcome based on the steps

UI still updates dynamically in the job detail view.

I'd like to discuss some of the implementation details

  1. Does it make sense to have more than COMPLETED and FAILED outcomes initially?
  2. Should status and outcome be separated, as they are now (not in DB, but in the user-facing objects)?
    • My idea here is that status would be able to be SCHEDULED and RUNNING as now, but instead of FINISHED it would return the outcome value (COMPLETED, FAILED, ...)
    • The current state needs you to 'care about' two different values, but the more I think about it, the less sense it makes to have that separation
  3. Does it make sense to store the item and/or item_type? I see some value there to be able to search for "did any jobs run on this stuff, without reporting results" so this is why I kept it so far. But I guess we can revisit this once we decide on the way to solve passing 'random arguments' to libtaskotron.

UNFINISHED:

  • API documentation
  • search over jobs (name, item, state, ...)
Test Plan

Tested locally via test_api.py

Diff Detail

Repository
rEXECDB execdb
Branch
feature/pony
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 1075
Build 1075: arc lint + arc unit
jskladan retitled this revision from to Decouple ExecDB from Buildbot.Feb 23 2017, 12:35 PM
jskladan updated this object.
jskladan edited the test plan for this revision. (Show Details)
jskladan added reviewers: kparal, tflink, mkrizek, lbrabec, jsedlak.
kparal added a comment.Feb 23 2017, 2:39 PM

Macro idontalways: I don't always understand the code but I can always complain

Does it make sense to have more than COMPLETED and FAILED outcomes initially?

I think having just these two is fine, at least for now. The idea is that these include the crash/failure reason in the future, right? Like FAILED NETWORK, etc (do we have that documented somehwhere? I already forgot it)?

There might be some confusion between FAILED outcome in execdb and FAILED in resultsdb (also, we use the same term for them).

Should status and outcome be separated, as they are now (not in DB, but in the user-facing objects)?
My idea here is that status would be able to be SCHEDULED and RUNNING as now, but instead of FINISHED it would return the outcome value (COMPLETED, FAILED, ...)
The current state needs you to 'care about' two different values, but the more I think about it, the less sense it makes to have that separation

That depends on the answer for the question above. IIUIC, I'd do it like: SCHEDULED, RUNNING, COMPLETED or FAILED. And then have the failure reason as a separate field.

Does it make sense to store the item and/or item_type? I see some value there to be able to search for "did any jobs run on this stuff, without reporting results" so this is why I kept it so far. But I guess we can revisit this once we decide on the way to solve passing 'random arguments' to libtaskotron.

The first time a build/update doesn't receive any result and somebody asks us "why my htop-1.2-3.fc25 didn't receive any test feedback?", we'll need to be able to search that up somehow. Do you know about any better ways to do it outside of storing item/item_type (and possibly other properties) in execdb, at least for failed jobs?

execdb/cli.py
125–134

We added a new step to dev yesterday.

152

This will crash if the step is not listed. Let's not.

mkrizek resigned from this revision.Thu, Jul 13, 9:03 AM

Revision Contents

Diff 2928

View Options

alembic/versions/104a0806520e_bulidbot_decoupling_step_1.py

  • This file was added.
1​"""Bulidbot Decoupling - step 1
2​
3​Revision ID: 104a0806520e
4​Revises: 3b51d15f0f87
5​Create Date: 2017-02-22 15:39:40.333356
6​
7​"""
8​
9​# revision identifiers, used by Alembic.
10​revision = '104a0806520e'
11​down_revision = '3b51d15f0f87'
12​branch_labels = None
13​depends_on = None
14​
15​from alembic import op
16​import sqlalchemy as sa
17​
18​
19​def upgrade():
20​ op.add_column('build_step', sa.Column('outcome', sa.Enum(
21​ 'COMPLETED', 'FAILED', 'CRASHED', 'ABORTED', name='execoutcome'), nullable=True))
22​ op.add_column(
23​ 'build_step', sa.Column('description', sa.Text(), nullable=True))
24​
25​
26​def downgrade():
27​ op.drop_column('build_step', 'outcome')
28​ op.drop_column('build_step', 'description')
View Options

alembic/versions/3b7629fd849f_bulidbot_decoupling_step_2.py

  • This file was added.
1​"""Bulidbot Decoupling - step 2
2​
3​Revision ID: 3b7629fd849f
4​Revises: 104a0806520e
5​Create Date: 2017-02-22 15:43:40.895604
6​
7​"""
8​
9​# revision identifiers, used by Alembic.
10​revision = '3b7629fd849f'
11​down_revision = '104a0806520e'
12​branch_labels = None
13​depends_on = None
14​
15​from alembic import op
16​import sqlalchemy as sa
17​from sqlalchemy.ext.declarative import declarative_base
18​from sqlalchemy.orm import relation, sessionmaker
19​from sqlalchemy.sql import text
20​import logging
21​
22​Session = sessionmaker()
23​Base = declarative_base()
24​
25​
26​EXEC_OUTCOME = ('COMPLETED', 'FAILED', 'CRASHED', 'ABORTED')
27​
28​
29​class BuildStep(Base):
30​ __tablename__ = 'build_step'
31​ id = sa.Column(sa.Integer, primary_key=True)
32​ status = sa.Column(sa.String(10))
33​ outcome = sa.Column(sa.Enum(*EXEC_OUTCOME, name='execoutcome'))
34​
35​
36​OUTCOME_MAP = {'OK': 'COMPLETED', 'NOT OK': 'FAILED', 'INPROGRESS': None}
37​DESCRIPTION_MAP = {
38​ "rm_tmp": "Cleaning-up temporary files",
39​ "rm_log": "Cleaning-up log files",
40​ "git": "Cloning the task repository",
41​ "runtask": "Running the testsuite",
42​ "MasterShellCommand": "Creating artifacts directory on master",
43​ "upload": "Uploading the artifacts to master",
44​ "MasterShellCommand_1": "Compressing artifacts",
45​ "MasterShellCommand_2": "Publishing the artifacts",
46​}
47​
48​
49​def upgrade():
50​ logger = logging.getLogger('alembic')
51​ connection = op.get_bind()
52​ session = Session(bind=connection)
53​
54​ # Alter buildsteps
55​ i = 0
56​ for step in session.query(BuildStep).yield_per(100):
57​ i += 1
58​ step.outcome = OUTCOME_MAP[step.status]
59​ step.description = DESCRIPTION_MAP[step.name]
60​ session.add(step)
61​ if i % 1000 == 0:
62​ logger.info("Traversed %s steps", i)
63​ session.commit()
64​ logger.info("Final steps commit")
65​ session.commit()
66​
67​ op.alter_column('build_step', 'finished_at', new_column_name='t_finished')
68​ op.alter_column('build_step', 'started_at', new_column_name='t_started')
69​ op.alter_column('build_step', 'name', existing_type=sa.String(20), type_=sa.Text())
70​ op.drop_index('buildstep_idx_status', table_name='build_step')
71​ op.drop_column('build_step', 'status')
72​ op.drop_column('build_step', 'data')
73​
74​ op.add_column('job', sa.Column('outcome', sa.Enum(
75​ 'COMPLETED', 'FAILED', 'CRASHED', 'ABORTED', name='execoutcome'), nullable=True))
76​ op.alter_column('job', 'link_build_log', new_column_name='ref_url')
77​ op.alter_column('job', 't_build_ended', new_column_name='t_finished')
78​ op.alter_column('job', 't_build_started', new_column_name='t_started')
79​ op.alter_column('job', 'taskname', existing_type=sa.String(20), type_=sa.Text())
80​ op.alter_column('job', 'item_type', existing_type=sa.String(20), type_=sa.Text())
81​ op.create_index('job_idx_type', 'job', [
82​ 'item_type'], unique=False, postgresql_ops={'item_type': 'text_pattern_ops'})
83​ op.drop_column('job', 'fedmsg_data')
84​ op.drop_column('job', 'slavename')
85​ op.drop_column('job', 'arch')
86​
87​
88​def downgrade():
89​ pass
90​ # FIXME - put together later on
View Options

execdb/cli.py

Show First 20 Lines • Show All 115 Lines • ▼ Show 20 Line(s)108 if destructive or not db.session.query(User).count():
116 db.session.commit() 116​ db.session.commit()
117 else: 117​ else:
118 print " - skipped User" 118​ print " - skipped User"
119 119​
120def mock_data_live(destructive): 120​def mock_data_live(destructive):
121 import time 121​ import time
122 import datetime 122​ import datetime
123 import random 123​ import random
124​
125​ dsc_map = {
126​ "rm_tmp": "Cleaning-up temporary files",
127​ "rm_log": "Cleaning-up log files",
128​ "git": "Cloning the task repository",
129​ "runtask": "Running the testsuite",
130​ "MasterShellCommand": "Creating artifacts directory on master",
131​ "upload": "Uploading the artifacts to master",
132​ "MasterShellCommand_1": "Compressing artifacts",
133​ "MasterShellCommand_2": "Publishing the artifacts",
134​ }
kparalUnsubmitted
Not Done

We added a new step to dev yesterday.

135​
136​
124 if destructive or not db.session.query(Job).count(): 137​ if destructive or not db.session.query(Job).count():
125 job = Job() 138​ job = Job()
126 job.uuid = '249cb6f4-e471-11e6-a77f-525400cb0b45' 139​ job.uuid = '249cb6f4-e471-11e6-a77f-525400cb0b45'
127 job.t_triggered = datetime.datetime.utcnow() 140​ job.t_triggered = datetime.datetime.utcnow()
128 job.taskname = 'rpmgrill' 141​ job.taskname = 'rpmgrill'
129 job.item = 'libtaskotron-0.4.18-1.fc25' 142​ job.item = 'libtaskotron-0.4.18-1.fc25'
130 job.item_type = 'koji_build' 143​ job.item_type = 'koji_build'
131 job.arch = 'x86_64' ​
132 job.slavename = 'qa11.qa-2' ​
133 db.session.add(job) 144​ db.session.add(job)
134 db.session.commit() 145​ db.session.commit()
135 time.sleep(10) 146​ time.sleep(5)
136 job.start() 147​ job.start()
137 db.session.add(job) 148​ db.session.add(job)
138 db.session.commit() 149​ db.session.commit()
139 time.sleep(3) 150​ time.sleep(3)
140 for step_name, sleep_time in (('rm_tmp', 2), ('git', 8), ('runtask', 17), ('MasterShellCommand', 1), ('upload', 6), ('MasterShellCommand_1', 3), ('MasterShellCommand_2', 5)): 151​ for step_name, sleep_time in (('rm_tmp', 2), ('git', 8), ('runtask', 17), ('MasterShellCommand', 1), ('upload', 6), ('MasterShellCommand_1', 3), ('MasterShellCommand_2', 5)):
141 step = BuildStep(step_name) 152​ step = BuildStep(dsc_map[step_name], step_name)
kparalUnsubmitted
Not Done

This will crash if the step is not listed. Let's not.

142 step.job = job 153​ step.job = job
143 step.status = 'INPROGRESS' ​
144 step.start() 154​ step.start()
145 db.session.add(step) 155​ db.session.add(step)
146 db.session.commit() 156​ db.session.commit()
147 time.sleep(sleep_time) 157​ time.sleep(sleep_time)
148 step.status = 'OK' ​
149 step.finish() 158​ step.finish()
150 db.session.add(step) 159​ db.session.add(step)
151 db.session.commit() 160​ db.session.commit()
152 job.finish() 161​ job.finish()
153 db.session.add(job) 162​ db.session.add(job)
154 db.session.commit() 163​ db.session.commit()
155 164​
156 165​
Show All 37 Lines
View Options

execdb/controllers/main.py

Show All 21 Lines
22from sqlalchemy.orm import exc as orm_exc 22​from sqlalchemy.orm import exc as orm_exc
23 23​
24from flask.ext.restful import reqparse 24​from flask.ext.restful import reqparse
25from werkzeug.exceptions import HTTPException 25​from werkzeug.exceptions import HTTPException
26from werkzeug.exceptions import BadRequest as JSONBadRequest 26​from werkzeug.exceptions import BadRequest as JSONBadRequest
27 27​
28 28​
29from execdb import app, db 29​from execdb import app, db
30from execdb.models.job import Job, BuildStep 30​from execdb.models.job import Job, BuildStep, EXEC_OUTCOME
31​from execdb.serializers.api_v1 import Serializer
31from sqlalchemy import desc 32​from sqlalchemy import desc
32 33​
33import json 34​import json
34import re 35​import re
36​import datetime
35 37​
36from pprint import pformat 38​from pprint import pformat
37 39​
38main = Blueprint('main', __name__) 40​main = Blueprint('main', __name__)
39BB_URL = app.config['BUILDBOT_FRONTPAGE_URL'] ​
40RESULTSDB_URL = app.config['RESULTSDB_FRONTPAGE_URL'] 41​RESULTSDB_URL = app.config['RESULTSDB_FRONTPAGE_URL']
41 42​
43​QUERY_LIMIT = 20
44​
42RE_PAGE = re.compile(r"([?&])page=([0-9]+)") 45​RE_PAGE = re.compile(r"([?&])page=([0-9]+)")
46​RE_CALLBACK = re.compile(r"([?&])callback=[^&]*&?")
47​RE_CLEAN_AMPERSANDS = re.compile(r'&+')
48​
43RP = {} 49​RP = {}
44RP['get_jobs'] = reqparse.RequestParser() 50​SERIALIZE = Serializer().serialize
45RP['get_jobs'].add_argument('page', default=0, type=int, location='args') ​
46RP['get_jobs'].add_argument('limit', default=30, type=int, location='args') ​
47 51​
52​# =============================================================================
53​# GLOBAL METHODS
54​# =============================================================================
48 55​
49def pagination(q, page, limit): 56​def pagination(q, page, limit):
57​ """
58​ Sets the offset/limit for the DB query.
59​ limit+1 is purposely set as 'limit' so we can later on decide whether 'next'
60​ page link should be provided or set to None.
61​ """
50 # pagination offset 62​ # pagination offset
51 try: 63​ try:
52 page = int(page) 64​ page = int(page)
53 if page > 0: 65​ if page > 0:
54 offset = page * limit 66​ offset = page * limit
55 q = q.offset(offset) 67​ q = q.offset(offset)
56 except (TypeError, ValueError): 68​ except (TypeError, ValueError):
57 pass 69​ pass
58 70​
59 # apply the query limit 71​ q = q.limit(limit + 1)
60 try: ​
61 limit = int(limit) ​
62 except (ValueError, TypeError): ​
63 limit = QUERY_LIMIT ​
64 ​
65 q = q.limit(limit) ​
66 return q 72​ return q
67 73​
68# TODO: find a better way to do this ​
69 ​
70 74​
71def prev_next_urls(): 75​def prev_next_urls(data, limit=QUERY_LIMIT):
72 global RE_PAGE 76​ global RE_PAGE
77​
73 try: 78​ try:
74 match = RE_PAGE.findall(request.url) 79​ match = RE_PAGE.findall(request.url)
75 flag, page = match[0][0], int(match[0][1]) 80​ flag, page = match[0][0], int(match[0][1])
76 except IndexError: # page not found 81​ except IndexError: # page not found
77 if '?' in request.url: 82​ page = None
78 return None, "%s&page=1" % request.url ​
79 else: ​
80 return None, "%s?page=1" % request.url ​
81 83​
82 prev = None 84​ prev = None
83 next = None 85​ next = None
84 prevpage = page - 1 86​ placeholder = "[!@#$%^&*PLACEHOLDER*&^%$#@!]"
85 nextpage = page + 1 87​
88​ if page is None:
89​ if "?" in request.url:
90​ baseurl = "%s&page=%s" % (request.url, placeholder)
91​ else:
92​ baseurl = "%s?page=%s" % (request.url, placeholder)
93​ page = 0
94​ else:
95​ baseurl = RE_PAGE.sub("%spage=%s" % (flag, placeholder), request.url)
96​
97​ baseurl = RE_CALLBACK.sub(r"\1", baseurl)
98​ baseurl = RE_CLEAN_AMPERSANDS.sub('&', baseurl)
86 99​
87 if page > 0: 100​ if page > 0:
88 prev = RE_PAGE.sub("%spage=%s" % (flag, prevpage), request.url) 101​ prev = baseurl.replace(placeholder, str(page - 1))
89 next = RE_PAGE.sub("%spage=%s" % (flag, nextpage), request.url) 102​ if len(data) > limit:
103​ next = baseurl.replace(placeholder, str(page + 1))
104​ data = data[:limit]
90 105​
91 return prev, next 106​ return data, prev, next
92 107​
93 108​
94@main.route('/') 109​# =============================================================================
95@main.route('/index') 110​# API
96@main.route('/jobs', methods=['GET']) 111​# =============================================================================
97def index(): 112​
113​# =============================================================================
114​# JOBS
115​# =============================================================================
116​
117​@main.route('/api/v1/jobs/<uuid>', methods=['GET'])
118​def get_job(uuid):
119​ job = Job.query.filter_by(uuid=uuid).first()
120​ if not job:
121​ return jsonify({'message': 'Job not found'}), 404
122​ return jsonify(SERIALIZE(job))
123​
124​
125​RP['get_jobs'] = reqparse.RequestParser()
126​RP['get_jobs'].add_argument('page', default=0, type=int, location='args')
127​RP['get_jobs'].add_argument('limit', default=QUERY_LIMIT, type=int, location='args')
128​
129​@main.route('/api/v1/jobs', methods=['GET'])
130​def get_jobs():
98 try: 131​ try:
99 args = RP['get_jobs'].parse_args() 132​ args = RP['get_jobs'].parse_args()
100 except JSONBadRequest as error: 133​ except JSONBadRequest as error:
101 return jsonify({"message": "Bad Request"}), error.code 134​ return jsonify({"message": "Bad Request"}), error.code
102 except HTTPException as error: 135​ except HTTPException as error:
103 return jsonify(error.data), error.code 136​ return jsonify(error.data), error.code
104 137​
105 query = db.session.query(Job).order_by(desc(Job.t_triggered)) 138​ query = db.session.query(Job).order_by(desc(Job.t_triggered))
106 query = pagination(query, args['page'], args['limit']) ​
107 139​
108 prev, next = prev_next_urls() 140​ query = pagination(query, args['page'], args['limit'])
109 jobs = query.all() 141​ data, prev, next = prev_next_urls(query.all(), args['limit'])
110 142​
111 return render_template('index.html', 143​ return jsonify(dict(
112 jobs=jobs, 144​ prev=prev,
113 buildbot_url=BB_URL, 145​ next=next,
114 prev=prev, 146​ data=[SERIALIZE(o) for o in data],
115 next=next) 147​ ))
116 148​
149​RP['create_job'] = reqparse.RequestParser()
150​RP['create_job'].add_argument('taskname', required=True, location='json')
151​RP['create_job'].add_argument('item', required=True, location='json')
152​RP['create_job'].add_argument('item_type', location='json')
153​RP['create_job'].add_argument('ref_url', location='json')
117 154​
118@main.route('/jobs/<uuid>', methods=['GET']) 155​@main.route('/api/v1/jobs', methods=['POST'])
119def show_job(uuid): 156​def create_job():
120 try: 157​ try:
121 job = db.session.query(Job).filter(Job.uuid == uuid).one() 158​ args = RP['create_job'].parse_args()
122 except orm_exc.NoResultFound: 159​ except JSONBadRequest as error:
123 return 'UUID not found', 404 160​ return jsonify({"message": "Malformed Request: %s" % error.data['message']}), error.code
124 job.t_triggered = str(job.t_triggered).split('.')[0] 161​ except HTTPException as error:
125 return render_template('show_job.html', 162​ return jsonify(error.data), error.code
126 job=job, 163​
127 buildbot_url=BB_URL, 164​ job = Job(**args)
128 resultsdb_url=RESULTSDB_URL, 165​ db.session.add(job)
129 artifacts_base_url=app.config['ARTIFACTS_BASE_URL']) 166​ db.session.commit()
167​ # FIXME - add resultsdb-like serializer
168​ retval = {
169​ 'uuid': job.uuid,
170​ 'taskname': job.taskname,
171​ 'item': job.item,
172​ 'item_type': job.item_type,
173​ 'ref_url': job.ref_url,
174​ 'outcome': job.outcome,
175​ 'status': job.status
176​ }
130 177​
178​ return jsonify(SERIALIZE(job)), 201
131 179​
132@main.route('/jobs/<uuid>/steps', methods=['GET']) 180​
133def show_steps(uuid): 181​RP['update_job'] = reqparse.RequestParser()
182​RP['update_job'].add_argument('start', type=bool, default=False, location='json')
183​RP['update_job'].add_argument('ref_url', location='json')
184​RP['update_job'].add_argument('outcome', location='json')
185​RP['update_job'].add_argument('duration', location='json')
186​
187​@main.route('/api/v1/jobs/<uuid>', methods=['POST'])
188​def update_job(uuid):
134 try: 189​ try:
135 job = db.session.query(Job).filter(Job.uuid == uuid).one() 190​ args = RP['update_job'].parse_args()
136 except orm_exc.NoResultFound: 191​ except JSONBadRequest as error:
137 return 'UUID not found', 404 192​ return jsonify({"message": "Malformed Request: %s" % error.data['message']}), error.code
193​ except HTTPException as error:
194​ return jsonify(error.data), error.code
138 195​
139 steps = dict( 196​ if args['duration']:
140 buildbot_url=BB_URL, 197​ try:
141 steps=[], 198​ args['duration'] = int(args['duration'])
142 job_status=job.current_state, 199​ except ValueError:
143 job_duration=str(job.build_took), 200​ return jsonify({'message': 'duration must be int or None'}), 400
144 ) 201​
145 name_map = { 202​ job = Job.query.filter_by(uuid=uuid).first()
146 "rm_tmp": "Cleaning-up temporary files", 203​ if not job:
147 "rm_log": "Cleaning-up log files", 204​ return jsonify({'message': 'Job not found'}), 404
148 "git": "Cloning the task repository", 205​
149 "runtask": "Running the testsuite", 206​ if args['outcome'] and args['outcome'] not in EXEC_OUTCOME+('_COMPUTED_', ):
150 "MasterShellCommand": "Creating artifacts directory on master", 207​ return jsonify({'message': "outcome %r not one of %r" % (args['outcome'], EXEC_OUTCOME+['_COMPUTE_'],)}), 400
151 "upload": "Uploading the artifacts to master", 208​
152 "MasterShellCommand_1": "Compressing artifacts", 209​ if args['start']:
153 "MasterShellCommand_2": "Publishing the artifacts", 210​ job.start()
154 } 211​
155 for step in job.build_steps: 212​ if args['outcome']:
156 if step.status not in ['OK', 'NOT OK', 'INPROGRESS']: 213​ try:
157 continue 214​ # mark last step as finished, if not done already
158 s = {} 215​ last_step = job.steps[-1]
159 s['name'] = step.name 216​ if last_step.status != 'FINISHED':
160 s['description'] = name_map.get(step.name, step.name) 217​ last_step.finish()
161 s['status'] = step.status 218​ db.session.add(last_step)
162 s['log_url'] = step.log_url 219​ except IndexError:
163 duration = step.step_took 220​ pass
164 minutes = duration.seconds / 60 221​ job.finish(args['outcome'])
165 seconds = duration.seconds - (minutes*60) ​
166 duration = [] ​
167 if minutes: ​
168 duration.append('%s minutes' % minutes) ​
169 duration.append('%s seconds' % seconds) ​
170 s['duration'] = ' '.join(duration) ​
171 steps['steps'].append(s) ​
172 ​
173 return jsonify(steps) ​
174 ​
175 ​
176 ​
177@main.route('/jobs', methods=['POST']) ​
178def create_job(): ​
179 job = Job() ​
180 222​
181 data = request.json 223​ if args['duration'] is not None:
182 job.fedmsg_data = json.dumps(data) 224​ if job.status != 'FINISHED':
225​ job.finish()
226​ job.t_finished = datetime.datetime.utcnow()
227​ job.t_started = job.t_finished - datetime.timedelta(seconds=args['duration'])
183 228​
184 # FIXME - add validation 229​
185 job.taskname = data['taskname'] 230​ if args['ref_url']:
186 job.item = data['item'] 231​ job.ref_url = args['ref_url']
187 job.item_type = data['item_type'] ​
188 job.arch = data['arch'] ​
189 232​
190 db.session.add(job) 233​ db.session.add(job)
191 db.session.commit() 234​ db.session.commit()
192 # FIXME - add resultsdb-like serializer ​
193 retval = { ​
194 'id': job.id, ​
195 'uuid': job.uuid, ​
196 't_triggered': job.t_triggered.isoformat() ​
197 } ​
198 235​
199 return jsonify(retval), 201 236​ return jsonify(SERIALIZE(job))
200 237​
201 238​
202def process_event(data): 239​# =============================================================================
240​# STEPS
241​# =============================================================================
203 242​
204 def bb_convert_properties(prop): 243​RP['create_step'] = reqparse.RequestParser()
205 """Converts list of lists to dict""" 244​RP['create_step'].add_argument('name', location='json')
206 return dict([(key, value) for key, value, _ in prop]) 245​RP['create_step'].add_argument('description', required=True, location='json')
246​RP['create_step'].add_argument('duration', location='json')
247​RP['create_step'].add_argument('outcome', location='json')
207 248​
208 # at the moment, we act just on these events 249​@main.route('/api/v1/jobs/<uuid>/steps', methods=['POST'])
209 event = data['event'] 250​def create_step(uuid):
210 known_events = ['changeAdded', 'buildStarted', 'stepStarted', 251​ try:
211 'stepFinished', 'buildFinished'] 252​ args = RP['create_step'].parse_args()
212 253​ except JSONBadRequest as error:
213 if event not in known_events: 254​ return jsonify({"message": "Malformed Request: %s" % error.data['message']}), error.code
214 # FIXME remove 255​ except HTTPException as error:
215 if 'uuid' in json.dumps(data): 256​ return jsonify(error.data), error.code
216 app.logger.debug("UUID found in %s", event) ​
217 257​
218 return 'Skipping event', 204 258​ if args['duration']:
259​ try:
260​ args['duration'] = int(args['duration'])
261​ except ValueError:
262​ return jsonify({'message': 'duration must be int or None'}), 400
219 263​
220 # grab the 'properties' field 264​ job = Job.query.filter_by(uuid=uuid).first()
221 if event == 'changeAdded': 265​ if not job:
222 properties = bb_convert_properties(data['payload']['change']['properties']) 266​ return jsonify({'message': 'Job not found'}), 404
223 elif event in ['buildStarted', 'buildFinished']: ​
224 properties = bb_convert_properties(data['payload']['build']['properties']) ​
225 elif event in ['stepStarted', 'stepFinished']: ​
226 properties = bb_convert_properties(data['payload']['properties']) ​
227 267​
228 # abort if uuid is not provided ​
229 try: 268​ try:
230 uuid = properties['uuid'] 269​ # mark last step as finished, if not done already
231 except KeyError: 270​ last_step = job.steps[-1]
232 return 'Missing `uuid` field in properties', 400 271​ if last_step.status != 'FINISHED':
272​ last_step.finish()
273​ db.session.add(last_step)
274​ except IndexError:
275​ pass
276​
277​ step = BuildStep(args['description'], args['name'], args['outcome'])
278​ step.start()
279​ if args['outcome']:
280​ if step.status != 'FINISHED':
281​ step.finish(args['outcome'])
282​ step.outcome = args['outcome']
283​ if args['duration'] is not None:
284​ if step.status != 'FINISHED':
285​ step.finish()
286​ step.t_finished = datetime.datetime.utcnow()
287​ step.t_started = step.t_finished - datetime.timedelta(seconds=args['duration'])
288​
289​ job.steps.append(step)
290​
291​ db.session.add(step)
292​ db.session.add(job)
293​ db.session.commit()
294​
295​ return jsonify(SERIALIZE(step)), 201
296​
233 297​
234 if uuid is None: 298​RP['update_step'] = reqparse.RequestParser()
235 return 'UUID set to None', 400 299​RP['update_step'].add_argument('duration', location='json')
300​RP['update_step'].add_argument('outcome', default='COMPLETED', location='json')
236 301​
302​@main.route('/api/v1/jobs/<uuid>/steps/<id>', methods=['POST'])
303​def update_step(uuid, id):
237 try: 304​ try:
238 job = db.session.query(Job).filter(Job.uuid == uuid).one() 305​ args = RP['update_step'].parse_args()
239 except orm_exc.NoResultFound: 306​ except JSONBadRequest as error:
240 return 'UUID not found', 400 307​ return jsonify({"message": "Malformed Request: %s" % error.data['message']}), error.code
308​ except HTTPException as error:
309​ return jsonify(error.data), error.code
241 310​
242 if event == 'changeAdded': 311​ if args['duration']:
243 # FIXME ? 312​ try:
244 pass 313​ args['duration'] = int(args['duration'])
314​ except ValueError:
315​ return jsonify({'message': 'duration must be int or None'}), 400
245 316​
246 elif event == 'buildStarted' and job.current_state == 'Triggered': 317​ job = Job.query.filter_by(uuid=uuid).first()
247 job.start() 318​ if not job:
319​ return jsonify({'message': 'Job not found'}), 404
248 320​
249 job.taskname = properties['taskname'] 321​ if id == 'last':
250 job.item = properties['item'] ​
251 job.item_type = properties['item_type'] ​
252 job.arch = properties['arch'] ​
253 job.slavename = properties['slavename'] ​
254 job.link_build_log = '/builders/%s/builds/%s' % ( ​
255 data['payload']['build']['builderName'], ​
256 properties['buildnumber']) ​
257 ​
258 db.session.add(job) ​
259 ​
260 # add 'empty' steps for the build (since we know them already) ​
261# app.logger.debug("%s: %s" % (uuid, data['payload']['build']['steps'])) ​
262# app.logger.debug("%s - Build Started" % uuid) ​
263 for step_info in data['payload']['build']['steps']: ​
264 # app.logger.debug("%s -- adding step %s"% (uuid, step_info['name'])) ​
265 step = BuildStep(name=step_info['name']) ​
266 step.job = job ​
267 db.session.add(step) ​
268 ​
269 db.session.commit() ​
270 ​
271 elif event == 'stepStarted' and job.current_state == 'Running': ​
272 step_info = data['payload']['step'] ​
273# app.logger.debug("%s - Step Started - %s"% (uuid, step_info['name'])) ​
274 try: 322​ try:
275 step = job.get_build_step(step_info['name']) 323​ step = job.steps[-1]
276 except KeyError: 324​ except IndexError:
277 app.logger.debug("Job %s had missing step %s", job.uuid, step_info) 325​ return jsonify({'message': "Step not found"}), 404
278 step = BuildStep(name=step_info['name']) 326​ else:
279 step.job = job ​
280 ​
281 step.start() ​
282 step.status = 'INPROGRESS' ​
283 step.data = json.dumps(data['payload']) # FIXME - store sensible subset of data ​
284 db.session.add(step) ​
285 db.session.commit() ​
286# app.logger.debug("%s - Step Started - %s - written to db"% (uuid, step_info['name'])) ​
287 ​
288 elif event == 'stepFinished' and job.current_state == 'Running': ​
289 step_info = data['payload']['step'] ​
290# app.logger.debug("%s - Step Finished - %s"% (uuid, step_info['name'])) ​
291 try: 327​ try:
292 step = job.get_build_step(step_info['name']) 328​ id = int(id)
293 except KeyError: 329​ except ValueError:
294 return 'StepFinished received for non-existing step: %r' % step_info['name'], 400 330​ return jsonify({'message': "id %r could not be converted to int" % id}), 400
331​ try:
332​ step = job.steps[id]
333​ except IndexError:
334​ return jsonify({'message': "Step not found"}), 404
295 335​
296 step.finish() 336​ if args['outcome'] and args['outcome'] not in EXEC_OUTCOME:
337​ return jsonify({'message': "outcome %r not one of %r" % (args['outcome'], EXEC_OUTCOME,)}), 400
297 338​
298 step.status = 'OK' 339​ if args['outcome']:
299 # results key is only present for non-ok results 340​ if step.status != 'FINISHED':
300 if 'results' in step_info.keys(): 341​ step.finish(args['outcome'])
301 step.status = 'NOT OK' 342​ step.outcome = args['outcome']
302 step.data = json.dumps(data['payload']) # FIXME - store sensible subset of data ​
303 343​
304 db.session.add(step) 344​ if args['duration'] is not None:
305 db.session.commit() 345​ if step.status != 'FINISHED':
306# app.logger.debug("%s - Step Finished - %s - written to db" % (uuid, step_info['name'])) 346​ step.finish()
347​ step.t_finished = datetime.datetime.utcnow()
348​ step.t_started = step.t_finished - datetime.timedelta(seconds=args['duration'])
307 349​
308 elif event == 'buildFinished' and job.current_state == 'Running': 350​ db.session.add(step)
309 job.finish() 351​ db.session.add(job)
310 db.session.add(job) 352​ db.session.commit()
311 db.session.commit() ​
312# app.logger.debug("%s - Build Finished " % uuid) ​
313 353​
354​ return jsonify(SERIALIZE(step))
314 355​
315@main.route('/buildbottest', methods=['POST']) 356​
316def bb_push(): 357​# =============================================================================
317 """ 358​# FRONTEND
318 Receives the post-push notifications from buildbot and fills in 359​# =============================================================================
319 the steps for the job. 360​
320 """ 361​RP['index'] = reqparse.RequestParser()
321 # data are embedded in form field 'packets' 362​RP['index'].add_argument('page', default=0, type=int, location='args')
322 data = request.form 363​RP['index'].add_argument('limit', default=QUERY_LIMIT, type=int, location='args')
364​
365​@main.route('/')
366​@main.route('/index')
367​@main.route('/jobs', methods=['GET'])
368​def index():
323 try: 369​ try:
324 data = request.form['packets'] 370​ args = RP['index'].parse_args()
325 except werkzeug.exceptions.BadRequestKeyError: 371​ except JSONBadRequest as error:
326 return 'Field `packets` missing in request form.', 400 372​ return jsonify({"message": "Bad Request"}), error.code
327 data = json.loads(data) 373​ except HTTPException as error:
328 374​ return jsonify(error.data), error.code
329 # app.logger.debug(pformat(data)) 375​
330 376​ query = db.session.query(Job).order_by(desc(Job.t_triggered))
331 # multiple messages may be present in one 'packet' 377​
332 for entry in data: 378​ query = pagination(query, args['page'], args['limit'])
333 process_event(entry) 379​ data, prev, next = prev_next_urls(query.all(), args['limit'])
334# app.logger.debug("%s %s, %s", entry['id'], entry['event'], process_event(entry)) 380​
335 381​ jobs = [SERIALIZE(d) for d in data]
336 # plain 200 code needs to be returned - otherwise buildbot is 382​
337 # endlessly trying to re-send the message. 383​ return render_template('index.html',
338 # FIXME - add logging for non-200 responses 384​ jobs=jobs,
339 return '', 200 385​ prev=prev,
386​ next=next)
387​
388​
389​@main.route('/jobs/<uuid>', methods=['GET'])
390​def show_job(uuid):
391​ try:
392​ job = db.session.query(Job).filter(Job.uuid == uuid).one()
393​ except orm_exc.NoResultFound:
394​ return 'UUID not found', 404
395​
396​ job = SERIALIZE(job)
397​
398​ return render_template('show_job.html',
399​ job=job,
400​ resultsdb_url=RESULTSDB_URL,
401​ artifacts_base_url=app.config['ARTIFACTS_BASE_URL'])
402​
View Options

execdb/models/job.py

Show All 12 Lines
13# You should have received a copy of the GNU General Public License along 13​# You should have received a copy of the GNU General Public License along
14# with this program; if not, write to the Free Software Foundation, Inc., 14​# with this program; if not, write to the Free Software Foundation, Inc.,
15# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 15​# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
16# 16​#
17# Authors: 17​# Authors:
18# Josef Skladanka <jskladan@redhat.com> 18​# Josef Skladanka <jskladan@redhat.com>
19 19​
20from execdb import db, app 20​from execdb import db, app
21​from execdb.serializers import DBSerialize
21 22​
22import datetime 23​import datetime
23import uuid 24​import uuid
24 25​
25 26​
26class Job(db.Model): 27​EXEC_OUTCOME = ('COMPLETED', 'FAILED', 'ABORTED', 'CRASHED')
28​
29​class Job(db.Model, DBSerialize):
27 id = db.Column(db.Integer, primary_key=True) 30​ id = db.Column(db.Integer, primary_key=True)
28 uuid = db.Column(db.String(36), unique=True) 31​ uuid = db.Column(db.String(36), unique=True)
29 32​
30 t_triggered = db.Column(db.DateTime, default=datetime.datetime.utcnow) 33​ t_triggered = db.Column(db.DateTime, default=datetime.datetime.utcnow)
31 t_build_started = db.Column(db.DateTime) 34​ t_started = db.Column(db.DateTime)
32 t_build_ended = db.Column(db.DateTime) 35​ t_finished = db.Column(db.DateTime)
33 ​
34 fedmsg_data = db.Column(db.Text) ​
35 36​
36 taskname = db.Column(db.String(20)) 37​ taskname = db.Column(db.Text)
37 item = db.Column(db.Text) 38​ item = db.Column(db.Text)
38 item_type = db.Column(db.String(20)) 39​ item_type = db.Column(db.Text)
39 arch = db.Column(db.String(10)) ​
40 ​
41 slavename = db.Column(db.String(20)) ​
42 # slave_env_details = db.Column(db.Text) # ??? ​
43 ​
44 # task_git_hash = db.Column(db.String(40)) # ??? ​
45 40​
46 # link_machine_log = db.Column(db.Text) # ??? 41​ outcome = db.Column(db.Enum(*EXEC_OUTCOME, name='execoutcome'))
47 link_build_log = db.Column(db.Text) # $buildbot_url/builders/$builder_name/builds/$build_id 42​ ref_url = db.Column(db.Text)
48 # link_resultsdb = db.Column(db.Text) # Should be UUID-based ​
49 43​
50 build_steps = db.relation('BuildStep', backref='job', order_by="BuildStep.id") 44​ steps = db.relation('BuildStep', backref='job', order_by="BuildStep.id")
51 45​
52 __table_args__ = ( 46​ __table_args__ = (
53 db.Index('job_idx_item', 'item', 47​ db.Index('job_idx_item', 'item',
54 postgresql_ops={'item': 'text_pattern_ops'}, 48​ postgresql_ops={'item': 'text_pattern_ops'},
55 ), 49​ ),
50​ db.Index('job_idx_type', 'item_type',
51​ postgresql_ops={'item_type': 'text_pattern_ops'},
52​ ),
56 db.Index('job_idx_taskname', 'taskname', 53​ db.Index('job_idx_taskname', 'taskname',
57 postgresql_ops={'taskname': 'text_pattern_ops'}, 54​ postgresql_ops={'taskname': 'text_pattern_ops'},
58 ), 55​ ),
59 ) 56​ )
60 57​
61 def __init__(self, fedmsg_data=None): 58​ def __init__(self, taskname=None, item=None, item_type=None, ref_url=None):
62 self.uuid = str(uuid.uuid1()) 59​ self.uuid = str(uuid.uuid1())
63 self.fedmsg_data = fedmsg_data 60​ self.taskname = taskname
61​ self.item = item
62​ self.item_type = item_type
63​ self.ref_url = ref_url
64 64​
65 def start(self): 65​ def start(self):
66 self.t_build_started = datetime.datetime.utcnow() 66​ self.t_started = datetime.datetime.utcnow()
67 67​
68 def finish(self): 68​ def finish(self, outcome="COMPLETE"):
69 self.t_build_ended = datetime.datetime.utcnow() 69​ self.t_finished = datetime.datetime.utcnow()
70​ if outcome == '_COMPUTED_':
71​ outcome = 'COMPLETED'
72​ for step in self.steps:
73​ if EXEC_OUTCOME.index(step.outcome) > EXEC_OUTCOME.index(outcome):
74​ outcome = step.outcome
75​ self.outcome = outcome
70 76​
71 @property 77​ @property
72 def current_state(self): 78​ def status(self):
73 if self.t_build_started and not self.t_build_ended: 79​ if self.t_started and not self.t_finished:
74 return "Running" 80​ return "RUNNING"
75 if self.t_build_ended: 81​ if self.t_finished:
76 return "Finished" 82​ return "FINISHED"
77 return "Triggered" 83​ return "SCHEDULED"
78 84​
79 @property 85​ @property
80 def started_after(self): 86​ def started_after(self):
81 try: 87​ try:
82 return self.t_build_started - self.t_triggered 88​ return self.t_started - self.t_triggered
83 except TypeError: 89​ except TypeError:
84 return None 90​ return None
85 91​
86 @property 92​ @property
87 def build_took(self): 93​ def duration(self):
88 try: 94​ try:
89 return self.t_build_ended - self.t_build_started 95​ return self.t_finished - self.t_started
90 except TypeError: 96​ except TypeError:
91 return None 97​ try:
92 98​ return datetime.datetime.utcnow() - self.t_started
93 def get_build_step(self, name): 99​ except TypeError:
94 for step in self.build_steps: 100​ return datetime.timedelta()
95 if step.name == name: ​
96 return step ​
97 raise KeyError("Step %r not found" % name) ​
98 101​
99 102​
100class BuildStep(db.Model): 103​class BuildStep(db.Model, DBSerialize):
101 id = db.Column(db.Integer, primary_key=True) 104​ id = db.Column(db.Integer, primary_key=True)
102 job_id = db.Column(db.Integer, db.ForeignKey('job.id')) 105​ job_id = db.Column(db.Integer, db.ForeignKey('job.id'))
103 name = db.Column(db.String(20)) 106​ name = db.Column(db.Text)
104 status = db.Column(db.String(10)) 107​ description = db.Column(db.Text)
105 started_at = db.Column(db.DateTime) 108​ outcome = db.Column(db.Enum(*EXEC_OUTCOME, name='execoutcome'))
106 finished_at = db.Column(db.DateTime) 109​ t_started = db.Column(db.DateTime, default=datetime.datetime.utcnow)
107 data = db.Column(db.Text) 110​ t_finished = db.Column(db.DateTime)
108 111​
109 __table_args__ = ( 112​ __table_args__ = (
110 db.Index('buildstep_idx_status', 'status', ​
111 postgresql_ops={'status': 'text_pattern_ops'}, ​
112 ), ​
113 db.Index('buildstep_fk_job_id', 'job_id'), 113​ db.Index('buildstep_fk_job_id', 'job_id'),
114 ) 114​ )
115 115​
116 def __repr__(self): 116​ def __repr__(self):
117 return "(%s, %s)" % (self.name, self.status) 117​ return "(%s, %s, %s)" % (self.name, self.description, self.status)
118 118​
119 def __init__(self, name): 119​ def __init__(self, description=None, name=None, outcome=None):
120 self.name = name 120​ self.name = name
121​ self.description = description
122​ self.outcome = outcome
121 123​
122 def start(self): 124​ def start(self):
123 self.started_at = datetime.datetime.utcnow() 125​ self.t_started = datetime.datetime.utcnow()
124 126​
125 def finish(self): 127​ def finish(self, outcome="COMPLETED"):
126 self.finished_at = datetime.datetime.utcnow() 128​ self.t_finished = datetime.datetime.utcnow()
129​ self.outcome = outcome
127 130​
128 @property 131​ @property
129 def step_took(self): 132​ def status(self):
133​ if self.t_started and not self.t_finished:
134​ return "RUNNING"
135​ if self.t_finished:
136​ return "FINISHED"
137​ return "SCHEDULED"
138​
139​ @property
140​ def duration(self):
130 try: 141​ try:
131 return self.finished_at - self.started_at 142​ return self.t_finished - self.t_started
132 except TypeError: 143​ except TypeError:
133 try: 144​ try:
134 return datetime.datetime.utcnow() - self.started_at 145​ return datetime.datetime.utcnow() - self.t_started
135 except TypeError: 146​ except TypeError:
136 return datetime.timedelta() 147​ return datetime.timedelta()
137 ​
138 @property ​
139 def log_url(self): ​
140 return "%s/steps/%s" % (self.job.link_build_log, self.name) ​
View Options

execdb/serializers/__init__.py

  • This file was added.
1​# Copyright 2013, Red Hat, Inc
2​#
3​# This program is free software; you can redistribute it and/or modify
4​# it under the terms of the GNU General Public License as published by
5​# the Free Software Foundation; either version 2 of the License, or
6​# (at your option) any later version.
7​#
8​# This program is distributed in the hope that it will be useful,
9​# but WITHOUT ANY WARRANTY; without even the implied warranty of
10​# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11​# GNU General Public License for more details.
12​#
13​# You should have received a copy of the GNU General Public License along
14​# with this program; if not, write to the Free Software Foundation, Inc.,
15​# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
16​#
17​# Authors:
18​# Josef Skladanka <jskladan@redhat.com>
19​
20​from datetime import date, datetime, timedelta
21​
22​
23​class DBSerialize(object):
24​ pass
25​
26​
27​class BaseSerializer(object):
28​
29​ def serialize(self, value, **kwargs):
30​ # serialize the database objects
31​ # the specific serializer needs to implement serialize_CLASSNAME methods
32​ if DBSerialize in value.__class__.__bases__:
33​ return getattr(self, '_serialize_%s' % value.__class__.__name__)(value, **kwargs)
34​
35​ if type(value) == timedelta:
36​ ret = []
37​ s = value.seconds
38​ hours = s // 3600
39​ s -= hours*3600
40​ if hours:
41​ ret.append('%s hours' % hours)
42​
43​ minutes = s // 60
44​ s -= minutes*60
45​ if minutes:
46​ ret.append('%s minutes' % minutes)
47​
48​ ret.append('%s second%s' % (s, '' if s == 1 else 's'))
49​ return ' '.join(ret)
50​
51​
52​
53​ # convert datetimes to the right format
54​ if type(value) in (datetime, date):
55​ return value.isoformat()
56​
57​ if isinstance(value, dict):
58​ ret = {}
59​ for k, v in value.iteritems():
60​ ret[k] = self.serialize(v, **kwargs)
61​ return ret
62​
63​ # convert iterables to list of serialized stuff
64​ if hasattr(value, '__iter__'):
65​ ret = []
66​ for v in value:
67​ ret.append(self.serialize(v, **kwargs))
68​ return ret
69​
70​ return value
View Options

execdb/serializers/api_v1.py

  • This file was added.
1​# Copyright 2013, Red Hat, Inc
2​#
3​# This program is free software; you can redistribute it and/or modify
4​# it under the terms of the GNU General Public License as published by
5​# the Free Software Foundation; either version 2 of the License, or
6​# (at your option) any later version.
7​#
8​# This program is distributed in the hope that it will be useful,
9​# but WITHOUT ANY WARRANTY; without even the implied warranty of
10​# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11​# GNU General Public License for more details.
12​#
13​# You should have received a copy of the GNU General Public License along
14​# with this program; if not, write to the Free Software Foundation, Inc.,
15​# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
16​#
17​# Authors:
18​# Josef Skladanka <jskladan@redhat.com>
19​
20​from flask import url_for
21​from execdb.serializers import BaseSerializer
22​
23​
24​class Serializer(BaseSerializer):
25​
26​ def _serialize_Job(self, o, **kwargs):
27​ rv = dict(
28​ uuid=o.uuid,
29​ t_triggered=o.t_triggered,
30​ t_started=o.t_started,
31​ t_finished=o.t_finished,
32​ taskname=o.taskname,
33​ item=o.item,
34​ item_type=o.item_type,
35​ outcome=o.outcome,
36​ ref_url=o.ref_url,
37​ status=o.status,
38​ duration=o.duration,
39​ steps=o.steps,
40​
41​ href=url_for('main.get_job', uuid=o.uuid, _external=True),
42​ )
43​
44​ return {key: self.serialize(value) for key, value in rv.iteritems()}
45​
46​ def _serialize_BuildStep(self, o, **kwargs):
47​ rv = dict(
48​ id=o.job.steps.index(o),
49​ name=o.name,
50​ description=o.description,
51​ outcome=o.outcome,
52​ t_started=o.t_started,
53​ t_finished=o.t_finished,
54​ status=o.status,
55​ duration=o.duration,
56​ )
57​
58​ return {key: self.serialize(value) for key, value in rv.iteritems()}
59​
60​
View Options

execdb/templates/index.html

1{% extends "layout.html" %} 1​{% extends "layout.html" %}
2 2​
3{% block body %} 3​{% block body %}
4<br /> 4​<br />
5<table class="table table-bordered table-striped"> 5​<table class="table table-bordered table-striped">
6 <thead> 6​ <thead>
7 <th>Job</th> 7​ <th>Job</th>
8 <th>Item</th> 8​ <th>Item</th>
9 <th>State</th> 9​ <th>State</th>
10 <th>Build Steps</th> 10​ <th>Progress</th>
11 <th>Moar</th> 11​ <th>Moar</th>
12 </thead> 12​ </thead>
13 <tbody> 13​ <tbody>
14{% for job in jobs -%} 14​{% for job in jobs -%}
15<tr> 15​<tr>
16 <td> 16​ <td>
17 {{job.taskname}} 17​ {{job.taskname}}
18 </td> 18​ </td>
19 <td>{{job.item|truncate(50, True)}}</td> 19​ <td>{{job.item|truncate(50, True)}}</td>
20 <td>{{job.current_state}}</td> 20​ <td>{{job.status}}</td>
21 <td> 21​ <td>
22 {% for step in job.build_steps %} 22​ {% for step in job.steps %}
23 {# <a href="{{buildbot_url}}{{step.log_url}}"> #} 23​ <span title="{{step.description}}"
24 <span title="{{step.name}}" 24​ {% if step.status == 'RUNNING' %}
25 {% if step.status == 'INPROGRESS' %} ​
26 class="glyphicon glyphicon-cog" 25​ class="glyphicon glyphicon-cog"
27 {% elif step.status == 'OK' %} 26​ {% elif step.outcome == 'COMPLETED' %}
28 class="glyphicon glyphicon-ok" 27​ class="glyphicon glyphicon-ok"
29 {% elif step.status == 'NOT OK' %} 28​ {% elif step.outcome in ['FAILED', 'CRASHED', 'ABORTED'] %}
30 style="color:#B80000" class="glyphicon glyphicon-remove" 29​ style="color:#B80000" class="glyphicon glyphicon-remove"
31 {% else %} 30​ {% else %}
32 class="glyphicon glyphicon-minus" 31​ class="glyphicon glyphicon-minus"
33 {% endif %} 32​ {% endif %}
34 ></span> 33​ ></span>
35 {# </a> #} 34​ {# </a> #}
36 {% endfor %} 35​ {% endfor %}
37 <a style="float:right" title="Build Logs" href="{{buildbot_url}}{{job.link_build_log}}"> 36​ <a style="float:right" title="Execution Logs" href="{{job.ref_url}}">
38 <span class="glyphicon glyphicon-new-window"></span> 37​ <span class="glyphicon glyphicon-new-window"></span>
39 </a> 38​ </a>
40 </td> 39​ </td>
41 <td> 40​ <td>
42 <a href="{{ url_for('main.show_job', uuid=job.uuid)}}">Detail</a> 41​ <a href="{{ url_for('main.show_job', uuid=job.uuid)}}">Detail</a>
43 </td> 42​ </td>
44</tr> 43​</tr>
45{% endfor -%} 44​{% endfor -%}
Show All 13 Lines
View Options

execdb/templates/show_job.html

1{% extends "layout.html" %} 1​{% extends "layout.html" %}
2 2​
3{% block scripts %} 3​{% block scripts %}
4<script> 4​<script>
5function toggle_text_visibility(e){ 5​function toggle_text_visibility(e){
6 $(e).find('#description').toggle(); 6​ $(e).find('#description').toggle();
7} 7​}
8 8​
9function show_data(){ 9​function show_data(){
10 $.get("/jobs/{{job.uuid}}/steps", function(data){ 10​ $.get("/api/v1/jobs/{{job.uuid}}", function(data){
11 console.log(data); 11​ console.log(data);
12 $('#job_progress').empty(); 12​ $('#job_progress').empty();
13 var computed_job_status = "OK"; ​
14 data.steps.forEach(function(value){ 13​ data.steps.forEach(function(value){
15 var step_url = '<a href="'+data.buildbot_url+value.log_url+'"><i class="fa fa-external-link"></i></a>'; 14​ var step_url = '<a href="'+data.buildbot_url+value.log_url+'"><i class="fa fa-external-link"></i></a>';
16 var icon = ''; 15​ var icon = '';
17 var row_bgcolor = 'none'; 16​ var row_bgcolor = 'none';
18 if (value.status == 'INPROGRESS'){ 17​ if (value.status == 'RUNNING'){
19 icon = '<i class="fa fa-spinner fa-pulse fa-fw"></i><span class="sr-only">In progress</span>'; 18​ icon = '<i class="fa fa-spinner fa-pulse fa-fw"></i><span class="sr-only">In progress</span>';
20 row_bgcolor = '#577591'; 19​ row_bgcolor = '#577591';
21 } 20​ }
22 else if (value.status == 'OK'){ 21​ else if (value.outcome == 'COMPLETED'){
23 icon = '<i class="fa fa-check fa-fw"></i><span class="sr-only">OK</span>'; 22​ icon = '<i class="fa fa-check fa-fw"></i><span class="sr-only">OK</span>';
24 row_bgcolor = '#639a81'; 23​ row_bgcolor = '#639a81';
25 } 24​ }
26 else if (value.status == 'NOT OK'){ 25​ else {
27 icon = '<i class="fa fa-times fa-fw"></i><span class="sr-only">Not OK</span>'; 26​ icon = '<i class="fa fa-times fa-fw"></i><span class="sr-only">Not OK</span>';
28 computed_job_status = "NOT OK"; ​
29 row_bgcolor = '#cc6168'; 27​ row_bgcolor = '#cc6168';
30 } 28​ }
31 var field = '<div class="row status" style="padding:15px 0px 15px 0px;background-color: '+row_bgcolor+';"><div class="col-sm-8 text-left">'+icon+'&nbsp;'+value.description+'</div><div class="col-sm-4 text-right">'+value.duration+'</div></div>'; 29​ var field = '<div class="row status" style="padding:15px 0px 15px 0px;background-color: '+row_bgcolor+';"><div class="col-sm-8 text-left">'+icon+'&nbsp;'+value.description+'</div><div class="col-sm-4 text-right">'+value.duration+'</div></div>';
32 $('#job_progress').append(field); 30​ $('#job_progress').append(field);
33 }); 31​ });
34 var job_icon = ''; 32​ var job_icon = '';
35 var job_duration = ''+data.job_duration; 33​ var job_duration = data.duration;
36 if (data.job_status == 'Triggered'){ 34​ if (data.status == 'SCHEDULED'){
37 job_duration = 'Scheduled to run soon'; 35​ job_duration = 'Scheduled to run soon';
38 job_icon = '<i class="fa fa-calendar fa-fw"></i>'; 36​ job_icon = '<i class="fa fa-calendar fa-fw"></i>';
39 $('#data_envelope').css({'background-color':'#535452'}); 37​ $('#data_envelope').css({'background-color':'#535452'});
40 } 38​ }
41 if (data.job_status == 'Running'){ 39​ if (data.status == 'RUNNING'){
42 job_duration = 'The job is running'; 40​ job_duration = data.duration;
43 job_icon = '<i class="fa fa-spinner fa-pulse fa-fw"></i>'; 41​ job_icon = '<i class="fa fa-spinner fa-pulse fa-fw"></i>';
44 $('#data_envelope').css({'background-color':'#577591'}); 42​ $('#data_envelope').css({'background-color':'#577591'});
45 } 43​ }
46 if (data.job_status == 'Finished'){ 44​ if (data.status == 'FINISHED'){
47 if (computed_job_status == 'OK'){ 45​ if (data.outcome == 'COMPLETED'){
48 //job_icon = '<i class="fa fa-check fa-fw text-green"></i>'; 46​ //job_icon = '<i class="fa fa-check fa-fw text-green"></i>';
49 job_icon = '<i class="fa fa-check fa-fw"></i>'; 47​ job_icon = '<i class="fa fa-check fa-fw"></i>';
50 $('#data_envelope').css({'background-color':'#639a81'}); 48​ $('#data_envelope').css({'background-color':'#639a81'});
51 } 49​ }
52 else{ 50​ else{
53 //job_icon = '<i class="fa fa-times fa-fw text-red"></i>'; 51​ //job_icon = '<i class="fa fa-times fa-fw text-red"></i>';
54 job_icon = '<i class="fa fa-times fa-fw"></i>'; 52​ job_icon = '<i class="fa fa-times fa-fw"></i>';
55 $('#data_envelope').css({'background-color':'#cc6168'}); 53​ $('#data_envelope').css({'background-color':'#cc6168'});
56 } 54​ }
57 } 55​ }
58 $('#job_status').html(job_icon); 56​ $('#job_status').html(job_icon);
59 $('#job_duration').text(job_duration); 57​ $('#job_duration').text(job_duration);
60 58​
61 if (data.job_status != 'Finished'){ 59​ if (data.status != 'FINISHED'){
62 setTimeout(show_data, 5000); 60​ setTimeout(show_data, 1000);
63 } 61​ }
64 else 62​ else
65 { 63​ {
66 console.log("FINISHED!"); 64​ console.log("FINISHED!");
67 }; 65​ };
68 }); 66​ });
69}; 67​};
70 68​
Show All 23 Lines90 <div class="row">
94 <div class="col-sm-6">{{job.item}}</div> 92​ <div class="col-sm-6">{{job.item}}</div>
95 <div class="col-sm-1 text-right"><i title="Triggered at" class="fa fa-calendar" aria-hidden="true"></i></div> 93​ <div class="col-sm-1 text-right"><i title="Triggered at" class="fa fa-calendar" aria-hidden="true"></i></div>
96 <div class="col-sm-4 text-left">{{job.t_triggered}}</div> 94​ <div class="col-sm-4 text-left">{{job.t_triggered}}</div>
97 </div> 95​ </div>
98 <div class="row"> 96​ <div class="row">
99 <div class="col-sm-1">Type</div> 97​ <div class="col-sm-1">Type</div>
100 <div class="col-sm-6">{{job.item_type}}</div> 98​ <div class="col-sm-6">{{job.item_type}}</div>
101 <div class="col-sm-1 text-right"><i title="Run time" class="fa fa-clock-o" aria-hidden="true"></i></div> 99​ <div class="col-sm-1 text-right"><i title="Run time" class="fa fa-clock-o" aria-hidden="true"></i></div>
102 <div class="col-sm-4 text-left" id="job_duration"></div> 100​ <div class="col-sm-4 text-left" id="job_duration">{{job.duration}}</div>
103 </div> ​
104 <div class="row"> ​
105 <div class="col-sm-1">Arch</div> ​
106 <div class="col-sm-6">{{job.arch}}</div> ​
107 <div class="col-sm-1 text-right"></div> ​
108 <div class="col-sm-4 text-left"></div> ​
109 </div> 101​ </div>
110 </div> 102​ </div>
111 </div> 103​ </div>
112 <div class="row" style="padding-top: 2em"> 104​ <div class="row" style="padding-top: 2em">
113 <div class="col-sm-3 text-center">Progress</div> 105​ <div class="col-sm-3 text-center">Progress</div>
114 <div class="col-sm-3 text-center">Logs</div> 106​ <div class="col-sm-3 text-center">Logs</div>
115 <div class="col-sm-3 text-center">Artifacts</div> 107​ <div class="col-sm-3 text-center">Artifacts</div>
116 <div class="col-sm-3 text-center">ResultsDB</div> 108​ <div class="col-sm-3 text-center">ResultsDB</div>
117 </div> 109​ </div>
118</div> 110​</div>
119<br /> 111​<br />
120 112​
121<div class="container" id="job_progress" style="color: #e8e8e8;"> 113​<div class="container" id="job_progress" style="color: #e8e8e8;">
122</div> 114​</div>
123 115​
124{# ​
125 ​
126 ​
127 ​
128 ​
129 ​
130 ​
131 ​
132 ​
133 ​
134 ​
135<br /> ​
136<br /> ​
137<br /> ​
138<br /> ​
139<br /> ​
140<br /> ​
141<br /> ​
142 ​
143 ​
144 ​
145 ​
146 ​
147 ​
148 ​
149 ​
150 ​
151 ​
152 ​
153 ​
154 ​
155 ​
156 ​
157 ​
158 ​
159 ​
160 ​
161 ​
162<h3>The job is <strong id="job_status">{{job.current_state}}</strong></h3> ​
163<div class="container"> ​
164 <div class="row"> ​
165 <div class="status_block" id="job_progress"></div> ​
166 </div> ​
167</div> ​
168 ​
169<h3>Details</h3> ​
170 ​
171<div class="container"> ​
172 ​
173<div class="row"> ​
174 <div class="col-sm-2"><strong>Taskname</strong></div> ​
175 <div class="col-sm-9">{{job.taskname}}</div> ​
176</div> ​
177 ​
178<div class="row"> ​
179 <div class="col-sm-2"><strong>Item</strong></div> ​
180 <div class="col-sm-9">{{job.item}}</div> ​
181</div> ​
182 ​
183<div class="row"> ​
184 <div class="col-sm-2"><strong>Item type</strong></div> ​
185 <div class="col-sm-9">{{job.item_type}}</div> ​
186</div> ​
187 ​
188<div class="row"> ​
189 <div class="col-sm-2"><strong>Arch</strong></div> ​
190 <div class="col-sm-9">{{job.arch}}</div> ​
191</div> ​
192 ​
193<br /> ​
194 ​
195<div class="row"> ​
196 <div class="col-sm-2"><strong>Triggered at</strong></div> ​
197 <div class="col-sm-9">{{job.t_triggered}}</div> ​
198</div> ​
199 ​
200<div class="row"> ​
201 <div class="col-sm-2"><strong>Started after</strong></div> ​
202 <div class="col-sm-9">{{job.started_after}}</div> ​
203</div> ​
204 ​
205<div class="row"> ​
206 <div class="col-sm-2"><strong>Finished in</strong></div> ​
207 <div class="col-sm-9">{{job.build_took}}</div> ​
208</div> ​
209 ​
210<br /> ​
211 ​
212<div class="row"> ​
213 <div class="col-sm-2"><strong>FedMsg data</strong></div> ​
214 <div class="col-sm-9">{{job.fedmsg_data}}</div> ​
215</div> ​
216<div class="row"> ​
217 <div class="col-sm-2"><strong>Slave</strong></div> ​
218 <div class="col-sm-9"> ​
219 <a href="{{buildbot_url}}/buildslaves/{{job.slavename}}"> ​
220 {{job.slavename}} ​
221 <span class="glyphicon glyphicon-new-window"></span> ​
222 </a> ​
223 </div> ​
224</div> ​
225 ​
226<br /> ​
227 ​
228<div class="row"> ​
229 <div class="col-sm-2"><strong>Logs</strong></div> ​
230 <div class="col-sm-9"><a href="{{buildbot_url}}{{job.link_build_log}}"><span class="glyphicon glyphicon-new-window"></span></a></div> ​
231</div> ​
232 ​
233<div class="row"> ​
234 <div class="col-sm-2"><strong>Artifacts</strong></div> ​
235 <div class="col-sm-9"><a href="{{artifacts_base_url}}/{{job.uuid}}"><span class="glyphicon glyphicon-new-window"></span></a></div> ​
236</div> ​
237 ​
238<br /> ​
239 ​
240<div class="row"> ​
241 <div class="col-sm-2"><strong>ResultsDB</strong></div> ​
242 <div class="col-sm-9"><a href="{{resultsdb_url}}/groups/{{job.uuid}}"><span class="glyphicon glyphicon-new-window"></span></a></div> ​
243</div> ​
244 ​
245</div> ​
246#} ​
247{% endblock %} 116​{% endblock %}
View Options

test_api.py

  • This file was added.
1​# Copyright 2013, Red Hat, Inc.
2​#
3​# This program is free software; you can redistribute it and/or modify
4​# it under the terms of the GNU General Public License as published by
5​# the Free Software Foundation; either version 2 of the License, or
6​# (at your option) any later version.
7​#
8​# This program is distributed in the hope that it will be useful,
9​# but WITHOUT ANY WARRANTY; without even the implied warranty of
10​# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11​# GNU General Public License for more details.
12​#
13​# You should have received a copy of the GNU General Public License along
14​# with this program; if not, write to the Free Software Foundation, Inc.,
15​# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
16​#
17​# Author: Josef Skladanka <jskladan@redhat.com>
18​
19​import requests
20​import json
21​import inspect
22​import simplejson
23​import logging
24​
25​logger = logging.getLogger('execdb_api')
26​logger.addHandler(logging.NullHandler())
27​
28​_KEEP = object()
29​
30​
31​def _fparams(expand_kwargs=True):
32​ """Gets the parameters of the function, from which _fparams is called
33​ and returns the list of params, minus `self`"""
34​ frame = inspect.currentframe().f_back
35​ args, varargs, keywords, values = inspect.getargvalues(frame)
36​
37​ params = {}
38​
39​ for key in args:
40​ if key == 'self':
41​ continue
42​ params[key] = values[key]
43​
44​ if keywords:
45​ if expand_kwargs:
46​ for key, value in values[keywords].iteritems():
47​ params[key] = value
48​ else:
49​ params[keywords] = values[keywords]
50​
51​ # RequestParser fucks up None for strings, and converts it to u'None'...
52​ cleaned_up = {}
53​ for key, value in params.iteritems():
54​ if value is None:
55​ continue
56​ cleaned_up[key] = value
57​
58​ return cleaned_up
59​
60​
61​class ExecDBapiException(Exception):
62​
63​ def __init__(self, message='', response=None):
64​ ''':param response: :class:`requests.Response` object'''
65​ self.message = message
66​ self.response = response
67​
68​ def __str__(self):
69​ return repr(self.message)
70​
71​
72​class ExecDBapi(object):
73​
74​ def __init__(self, api_url):
75​ # remove trailing slash(es), so we don't generate
76​ # urls with a double slash which breaks werkzeug
77​ # https://github.com/mitsuhiko/werkzeug/issues/491
78​ self.url = api_url.rstrip('/')
79​
80​ def __raise_on_error(self, r):
81​ if r.ok:
82​ return
83​
84​ try:
85​ logger.warn('Received HTTP failure status code %s for request: %s',
86​ r.status_code, r.url)
87​ raise ExecDBapiException(
88​ '%s (HTTP %s)' % (r.json()['message'], r.status_code), r)
89​ except simplejson.JSONDecodeError as e:
90​ logger.debug('Received invalid JSON data: %s\n%s', e, r.text)
91​ raise ExecDBapiException(
92​ 'Invalid JSON (HTTP %s): %s' % (r.status_code, e), r)
93​ except KeyError:
94​ raise ExecDBapiException('HTTP %s Error' % r.status_code, r)
95​
96​ def __prepare_params(self, params_all):
97​ params = {}
98​ for key, value in params_all.iteritems():
99​ if value is None:
100​ continue
101​ if key == 'raw_params':
102​ continue
103​
104​ # if a param's name ends with _like, we treat it as if :like filter should be applied
105​ # for the rare case, where it really is supposed to be the name, user should provide
106​ # it in the 'raw_params' dict
107​ if key.endswith('_like'):
108​ key = "%s:like" % key[:-len('_like')]
109​
110​ if type(value) in (list, tuple):
111​ params[key] = ','.join([unicode(v) for v in value])
112​ else:
113​ params[key] = unicode(value)
114​
115​ if 'raw_params' in params_all.keys() and params_all['raw_params']:
116​ raw_params = {
117​ key: unicode(value) for key, value in params_all['raw_params'].iteritems()}
118​ params.update(raw_params)
119​ return params
120​
121​ def create_job(self, taskname, item, item_type=None, ref_url=None):
122​ url = "%s/jobs" % self.url
123​ headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
124​ r = requests.post(url, data=json.dumps(_fparams()), headers=headers)
125​ self.__raise_on_error(r)
126​
127​ return r.json()
128​
129​ def update_job(self, uuid, start=None, ref_url=None, outcome=None, duration=None):
130​ url = "%s/jobs/%s" % (self.url, uuid)
131​ headers = {'content-type': 'application/json', 'accept': 'text/plain'}
132​ params = _fparams()
133​ del(params['uuid'])
134​ r = requests.post(url, data=json.dumps(params), headers=headers)
135​ self.__raise_on_error(r)
136​
137​ return r.json()
138​
139​ def get_job(self, uuid):
140​ url = "%s/jobs/%s" % (self.url, uuid)
141​ r = requests.get(url)
142​ self.__raise_on_error(r)
143​
144​ return r.json()
145​
146​ def get_jobs(self, page=None, limit=None):
147​ url = "%s/jobs" % self.url
148​ r = requests.get(url, params=self.__prepare_params(_fparams()))
149​ self.__raise_on_error(r)
150​
151​ return r.json()
152​
153​ def create_step(self, job_uuid, description, name=None, outcome=None, duration=None):
154​ url = "%s/jobs/%s/steps" % (self.url, job_uuid)
155​ data = _fparams()
156​ del(data['job_uuid'])
157​ headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
158​ r = requests.post(url, data=json.dumps(data), headers=headers)
159​ self.__raise_on_error(r)
160​
161​ return r.json()
162​
163​ def update_step(self, job_uuid, step_id='last', outcome=None, duration=None):
164​ url = "%s/jobs/%s/steps/%s" % (self.url, job_uuid, step_id)
165​ headers = {'content-type': 'application/json', 'accept': 'text/plain'}
166​ data = _fparams()
167​ del(data['job_uuid'])
168​ del(data['step_id'])
169​ r = requests.post(url, data=json.dumps(data), headers=headers)
170​ self.__raise_on_error(r)
171​
172​ return r.json()
173​
174​
175​if __name__ == '__main__':
176​ import time
177​ import datetime
178​ import random
179​
180​ dsc_map = {
181​ "rm_tmp": "Cleaning-up temporary files",
182​ "rm_log": "Cleaning-up log files",
183​ "git": "Cloning the task repository",
184​ "runtask": "Running the testsuite",
185​ "MasterShellCommand": "Creating artifacts directory on master",
186​ "upload": "Uploading the artifacts to master",
187​ "MasterShellCommand_1": "Compressing artifacts",
188​ "MasterShellCommand_2": "Publishing the artifacts",
189​ }
190​
191​ api = ExecDBapi('http://localhost:5003/api/v1')
192​
193​ # Create job
194​ job = api.create_job('rpmgrill', 'libtaskotron-0.4.18-1.fc25', 'koji_build')
195​ time.sleep(5)
196​ # Start job
197​ api.update_job(job['uuid'], start=True, ref_url="http://example.com")
198​ time.sleep(3)
199​
200​ # Test step autofinishing
201​ for step_name, sleep_time in (('rm_tmp', 2), ('git', 8)):
202​ step = api.create_step(job['uuid'], dsc_map[step_name], step_name)
203​ time.sleep(sleep_time)
204​ # Test manual finish
205​ api.update_step(job['uuid'], step['id'], outcome="FAILED")
206​
207​ # Test duration
208​ step_name = 'runtask'
209​ time.sleep(17)
210​ step = api.create_step(job['uuid'], dsc_map[step_name], step_name, duration=17, outcome="COMPLETED")
211​
212​ step_name = 'MasterShellCommand'
213​ step = api.create_step(job['uuid'], dsc_map[step_name], step_name)
214​ time.sleep(3)
215​ api.update_step(job['uuid'], step['id'], duration=1)
216​
217​ # Test 'last' as step id
218​ step_name = 'upload'
219​ step = api.create_step(job['uuid'], dsc_map[step_name], step_name)
220​ time.sleep(6)
221​ api.update_step(job['uuid'], 'last', outcome='ABORTED')
222​
223​ # Do the ret of the steps
224​ for step_name, sleep_time in (('MasterShellCommand_1', 3), ('MasterShellCommand_2', 5)):
225​ step = api.create_step(job['uuid'], dsc_map[step_name], step_name)
226​ time.sleep(sleep_time)
227​
228​ # Finish Job
229​ api.update_job(job['uuid'], outcome='_COMPUTED_')
230​