Differential D263

With all JSON queries, return more pagination metadata.
ClosedPublic
Actions

Authored by • ralph2 on Nov 11 2014, 4:57 PM.

Details

Reviewers
tflink
jskladan
Commits
rRSDBbbe42e01516e: With all JSON queries, return more pagination metadata.
Summary

This now includes:

  • total - the total number of entities that can be retrieved.
  • pages - the total number of pages that can be retrieved

Without these, if you have code that is paging through resultsdb
results, there is no way to know when you are going to be done -- and no
way to jump to the last page and iterate through them in reverse.

Test Plan

Run the test suite.

Diff Detail

Repository
rRSDB resultsdb
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
• ralph2 retitled this revision from to With all JSON queries, return more pagination metadata..Nov 11 2014, 4:57 PM
• ralph2 updated this object.
• ralph2 edited the test plan for this revision. (Show Details)
• ralph2 added reviewers: tflink, jskladan.
tflink accepted this revision.Nov 11 2014, 11:17 PM

LGTM

This revision is now accepted and ready to land.Nov 11 2014, 11:17 PM
Closed by commit rRSDBbbe42e01516e: With all JSON queries, return more pagination metadata. (authored by Ralph Bean <rbean@redhat.com>). · Explain WhyJul 29 2015, 5:09 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathPackages
Mresultsdb/controllers/api_v1.py (48 lines)
Mtesting/functest_api.py (21 lines)

Diff 1263

View Options

resultsdb/controllers/api_v1.py

1# Copyright 2013, Red Hat, Inc 1​# Copyright 2013-2014, Red Hat, Inc
2# 2​#
3# This program is free software; you can redistribute it and/or modify 3​# This program is free software; you can redistribute it and/or modify
4# it under the terms of the GNU General Public License as published by 4​# it under the terms of the GNU General Public License as published by
5# the Free Software Foundation; either version 2 of the License, or 5​# the Free Software Foundation; either version 2 of the License, or
6# (at your option) any later version. 6​# (at your option) any later version.
7# 7​#
8# This program is distributed in the hope that it will be useful, 8​# This program is distributed in the hope that it will be useful,
9# but WITHOUT ANY WARRANTY; without even the implied warranty of 9​# but WITHOUT ANY WARRANTY; without even the implied warranty of
10# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 10​# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11# GNU General Public License for more details. 11​# GNU General Public License for more details.
12# 12​#
13# You should have received a copy of the GNU General Public License along 13​# You should have received a copy of the GNU General Public License along
14# with this program; if not, write to the Free Software Foundation, Inc., 14​# with this program; if not, write to the Free Software Foundation, Inc.,
15# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 15​# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
16# 16​#
17# Authors: 17​# Authors:
18# Josef Skladanka <jskladan@redhat.com> 18​# Josef Skladanka <jskladan@redhat.com>
19​# Ralph Bean <rbean@redhat.com>
19 20​
20import datetime 21​import datetime
22​import math
21import re 23​import re
22from functools import partial 24​from functools import partial
23 25​
24from flask import Blueprint, jsonify, request, url_for, abort 26​from flask import Blueprint, jsonify, request, url_for, abort
25from flask.ext.restful import reqparse 27​from flask.ext.restful import reqparse
26 28​
27from sqlalchemy.orm import exc as orm_exc 29​from sqlalchemy.orm import exc as orm_exc
28from sqlalchemy import exc as sqlalchemy_exc 30​from sqlalchemy import exc as sqlalchemy_exc
▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Line(s)
78__serializer = Serializer(__get_uri) 80​__serializer = Serializer(__get_uri)
79SERIALIZE = __serializer.serialize 81​SERIALIZE = __serializer.serialize
80 82​
81# ============================================================================= 83​# =============================================================================
82# GLOBAL METHODS 84​# GLOBAL METHODS
83# ============================================================================= 85​# =============================================================================
84 86​
85def pagination(q, page, limit): 87​def pagination(q, page, limit):
88​
89​ total = q.count()
90​ pages = int(math.ceil(total / float(limit)))
91​
86 # pagination offset 92​ # pagination offset
87 try: 93​ try:
88 page = int(page) 94​ page = int(page)
89 if page > 0: 95​ if page > 0:
90 offset = page * limit 96​ offset = page * limit
91 q = q.offset(offset) 97​ q = q.offset(offset)
92 except (TypeError, ValueError): 98​ except (TypeError, ValueError):
93 pass 99​ pass
94 100​
95 # apply the query limit 101​ # apply the query limit
96 try: 102​ try:
97 limit = int(limit) 103​ limit = int(limit)
98 except (ValueError, TypeError): 104​ except (ValueError, TypeError):
99 limit = QUERY_LIMIT 105​ limit = QUERY_LIMIT
100 106​
101 q = q.limit(limit) 107​ q = q.limit(limit)
102 return q 108​ return total, pages, q
103 109​
104#TODO: find a better way to do this 110​#TODO: find a better way to do this
105def prev_next_urls(): 111​def prev_next_urls():
106 global RE_PAGE 112​ global RE_PAGE
107 try: 113​ try:
108 match = RE_PAGE.findall(request.url) 114​ match = RE_PAGE.findall(request.url)
109 flag, page = match[0][0], int(match[0][1]) 115​ flag, page = match[0][0], int(match[0][1])
110 except IndexError: #page not found 116​ except IndexError: #page not found
▲ Show 20 Lines • Show All 169 Lines • ▼ Show 20 Line(s)285 try:
280 s, e = parse_since(args['since']) 286​ s, e = parse_since(args['since'])
281 except iso8601.iso8601.ParseError: 287​ except iso8601.iso8601.ParseError:
282 return jsonify({"message": "'since' parameter not in ISO8601 format"}), 400 288​ return jsonify({"message": "'since' parameter not in ISO8601 format"}), 400
283 289​
284 if args['status'] is not None and args['status'] not in JOB_STATUS: 290​ if args['status'] is not None and args['status'] not in JOB_STATUS:
285 return jsonify({'message': "status must be one of %r" % (JOB_STATUS,) }), 400 291​ return jsonify({'message': "status must be one of %r" % (JOB_STATUS,) }), 400
286 292​
287 q = select_jobs(since_start = s, since_end = e, status = args['status'], name = args['name']) 293​ q = select_jobs(since_start = s, since_end = e, status = args['status'], name = args['name'])
288 q = pagination(q, args['page'], args['limit']) ​
289 294​
295​ total, pages, q = pagination(q, args['page'], args['limit'])
290 prev, next = prev_next_urls() 296​ prev, next = prev_next_urls()
291 297​
292 return jsonify(dict(href = request.url, prev = prev, next = next, data = [SERIALIZE(o, job_load_results = args['load_results']) for o in q.all()])) 298​ return jsonify(dict(
299​ href=request.url,
300​ prev=prev,
301​ next=next,
302​ total=total,
303​ pages=pages,
304​ data=[
305​ SERIALIZE(o, job_load_results=args['load_results'])
306​ for o in q.all()
307​ ],
308​ ))
293 309​
294@api.route('/v1.0/jobs/<job_id>', methods = ['GET']) 310​@api.route('/v1.0/jobs/<job_id>', methods = ['GET'])
295def get_job(job_id): 311​def get_job(job_id):
296 try: 312​ try:
297 job_id = int(job_id) 313​ job_id = int(job_id)
298 except ValueError: # uuid can not be parsed to int 314​ except ValueError: # uuid can not be parsed to int
299 q = Job.query.filter_by(uuid = job_id) 315​ q = Job.query.filter_by(uuid = job_id)
300 else: # id was int -> filter by id 316​ else: # id was int -> filter by id
▲ Show 20 Lines • Show All 171 Lines • ▼ Show 20 Line(s)487 q = select_results(
472 since_start = args['since']['start'], 488​ since_start = args['since']['start'],
473 since_end = args['since']['end'], 489​ since_end = args['since']['end'],
474 since_source = args['since_source'], 490​ since_source = args['since_source'],
475 outcome = args['outcome'], 491​ outcome = args['outcome'],
476 result_data = p['result_data'], 492​ result_data = p['result_data'],
477 job_id = j_id, 493​ job_id = j_id,
478 testcase_name = t_nm, 494​ testcase_name = t_nm,
479 ) 495​ )
480 q = pagination(q, args['page'], args['limit']) ​
481 496​
497​ total, pages, q = pagination(q, args['page'], args['limit'])
482 prev, next = prev_next_urls() 498​ prev, next = prev_next_urls()
483 499​
484 return jsonify(dict(href = request.url, prev = prev, next = next, data = [SERIALIZE(o) for o in q.all()])) 500​ return jsonify(dict(
501​ href=request.url,
502​ prev=prev,
503​ next=next,
504​ total=total,
505​ pages=pages,
506​ data=[SERIALIZE(o) for o in q.all()],
507​ ))
485 508​
486@api.route('/v1.0/jobs/<job_id>/results', methods = ['GET']) 509​@api.route('/v1.0/jobs/<job_id>/results', methods = ['GET'])
487@api.route('/v1.0/testcases/<testcase_name>/results', methods = ['GET']) 510​@api.route('/v1.0/testcases/<testcase_name>/results', methods = ['GET'])
488def get_results_by_job_testcase(job_id = None, testcase_name = None): 511​def get_results_by_job_testcase(job_id = None, testcase_name = None):
489 # check whether the job/testcase exists. If not, throw 404 512​ # check whether the job/testcase exists. If not, throw 404
490 if job_id is not None: 513​ if job_id is not None:
491 try: 514​ try:
492 job_id = int(job_id) 515​ job_id = int(job_id)
▲ Show 20 Lines • Show All 135 Lines • ▼ Show 20 Line(s)650 except JSONBadRequest as error:
628 return jsonify({"message": "Malformed JSON data"}), error.code 651​ return jsonify({"message": "Malformed JSON data"}), error.code
629 except HTTPException as error: 652​ except HTTPException as error:
630 return jsonify(error.data), error.code 653​ return jsonify(error.data), error.code
631 654​
632 655​
633 q = db.session.query(Testcase) 656​ q = db.session.query(Testcase)
634 q.order_by(db.asc(Testcase.name)) 657​ q.order_by(db.asc(Testcase.name))
635 658​
636 q = pagination(q, args['page'], args['limit']) 659​ total, pages, q = pagination(q, args['page'], args['limit'])
637 ​
638 prev, next = prev_next_urls() 660​ prev, next = prev_next_urls()
639 return jsonify(dict(href = request.url, prev = prev, next = next, data = [SERIALIZE(o) for o in q.all()])) 661​
662​ return jsonify(dict(
663​ href=request.url,
664​ prev=prev,
665​ next=next,
666​ total=total,
667​ pages=pages,
668​ data=[SERIALIZE(o) for o in q.all()],
669​ ))
640 670​
641@api.route('/v1.0/testcases/<testcase_name>', methods = ['GET']) 671​@api.route('/v1.0/testcases/<testcase_name>', methods = ['GET'])
642def get_testcase(testcase_name): 672​def get_testcase(testcase_name):
643 673​
644 try: 674​ try:
645 testcase = Testcase.query.filter_by(name = testcase_name).one() 675​ testcase = Testcase.query.filter_by(name = testcase_name).one()
646 except orm_exc.NoResultFound: 676​ except orm_exc.NoResultFound:
647 return jsonify({'message': "Testcase not found" }), 404 677​ return jsonify({'message': "Testcase not found" }), 404
▲ Show 20 Lines • Show All 58 Lines • Show Last 20 Lines
View Options

testing/functest_api.py

Show First 20 Lines • Show All 46 Lines • ▼ Show 20 Line(s)39 def setup_method(self, method):
47 self.ref_job_url = "http://fedoraqa.fedoraproject.org" 47​ self.ref_job_url = "http://fedoraqa.fedoraproject.org"
48 self.ref_job_name = "F20 Virtualization Testday" 48​ self.ref_job_name = "F20 Virtualization Testday"
49 self.ref_status = "SCHEDULED" 49​ self.ref_status = "SCHEDULED"
50 self.ref_outcome = "PASSED" 50​ self.ref_outcome = "PASSED"
51 self.ref_result_id = 1 51​ self.ref_result_id = 1
52 self.ref_result_data = {'data': 'fakedata', 'data1': ['fakedata1'], 'data2': 1} 52​ self.ref_result_data = {'data': 'fakedata', 'data1': ['fakedata1'], 'data2': 1}
53 self.ref_result_summary = "1 PASSED, 0 FAILED" 53​ self.ref_result_summary = "1 PASSED, 0 FAILED"
54 self.ref_result_log_url = "http://fedoraqa.fedoraproject.org/logs" 54​ self.ref_result_log_url = "http://fedoraqa.fedoraproject.org/logs"
55​ self.prefix = 'http://localhost/api/v1.0/'
55 56​
56 def test_create_testcase(self): 57​ def test_create_testcase(self):
57 ref_data = json.dumps({'name': self.ref_testcase_name, 'url': self.ref_testcase_url}) 58​ ref_data = json.dumps({'name': self.ref_testcase_name, 'url': self.ref_testcase_url})
58 59​
59 r = self.app.post('/api/v1.0/testcases', data=ref_data, content_type='application/json') 60​ r = self.app.post('/api/v1.0/testcases', data=ref_data, content_type='application/json')
60 61​
61 data = json.loads(r.data) 62​ data = json.loads(r.data)
62 63​
▲ Show 20 Lines • Show All 345 Lines • ▼ Show 20 Line(s)406 def test_get_results(self):
408 r = self.app.get('/api/v1.0/results') 409​ r = self.app.get('/api/v1.0/results')
409 410​
410 data = json.loads(r.data) 411​ data = json.loads(r.data)
411 412​
412 assert r.status_code == 200 413​ assert r.status_code == 200
413 assert data['data'][0]['outcome'] == self.ref_outcome 414​ assert data['data'][0]['outcome'] == self.ref_outcome
414 assert data['data'][0]['id'] == self.ref_job_id 415​ assert data['data'][0]['id'] == self.ref_job_id
415 assert data['data'][0]['testcase']['name'] == self.ref_testcase_name 416​ assert data['data'][0]['testcase']['name'] == self.ref_testcase_name
417​ assert data['href'] == self.prefix + 'results'
418​ assert data['next'] == self.prefix + 'results?page=1'
419​ assert data['prev'] == None
420​ assert data['pages'] == 1
421​ assert data['total'] == 1
416 422​
417 def test_get_empty_results(self): 423​ def test_get_empty_results(self):
418 r = self.app.get('/api/v1.0/results') 424​ r = self.app.get('/api/v1.0/results')
419 425​
420 data = json.loads(r.data) 426​ data = json.loads(r.data)
421 427​
422 assert r.status_code == 200 428​ assert r.status_code == 200
423 assert data['data'] == [] 429​ assert data['data'] == []
424 430​
425 def test_get_testcases_results(self): 431​ def test_get_testcases_results(self):
426 self.test_create_result() 432​ self.test_create_result()
427 433​
428 r = self.app.get('/api/v1.0/testcases/%s/results' % self.ref_testcase_name) 434​ r = self.app.get('/api/v1.0/testcases/%s/results' % self.ref_testcase_name)
429 435​
430 data = json.loads(r.data) 436​ data = json.loads(r.data)
431 437​
432 assert r.status_code == 200 438​ assert r.status_code == 200
433 assert data['data'][0]['outcome'] == self.ref_outcome 439​ assert data['data'][0]['outcome'] == self.ref_outcome
434 assert data['data'][0]['id'] == self.ref_job_id 440​ assert data['data'][0]['id'] == self.ref_job_id
435 assert data['data'][0]['testcase']['name'] == self.ref_testcase_name 441​ assert data['data'][0]['testcase']['name'] == self.ref_testcase_name
442​ assert data['href'] == self.prefix + 'testcases/testcase/results'
443​ assert data['next'] == self.prefix + 'testcases/testcase/results?page=1'
444​ assert data['prev'] == None
445​ assert data['pages'] == 1
446​ assert data['total'] == 1
436 447​
437 def test_get_testcases_empty_results(self): 448​ def test_get_testcases_empty_results(self):
438 r = self.app.get('/api/v1.0/testcases/%s/results' % self.ref_testcase_name) 449​ r = self.app.get('/api/v1.0/testcases/%s/results' % self.ref_testcase_name)
439 450​
440 data = json.loads(r.data) 451​ data = json.loads(r.data)
441 452​
442 assert r.status_code == 404 453​ assert r.status_code == 404
443 assert data['message'] == "Testcase not found" 454​ assert data['message'] == "Testcase not found"
444 455​
445 def test_get_jobs_results(self): 456​ def test_get_jobs_results(self):
446 self.test_create_result() 457​ self.test_create_result()
447 458​
448 r = self.app.get('/api/v1.0/testcases/%s/results' % self.ref_testcase_name) 459​ r = self.app.get('/api/v1.0/testcases/%s/results' % self.ref_testcase_name)
449 460​
450 data = json.loads(r.data) 461​ data = json.loads(r.data)
451 462​
452 assert r.status_code == 200 463​ assert r.status_code == 200
453 assert data['data'][0]['outcome'] == self.ref_outcome 464​ assert data['data'][0]['outcome'] == self.ref_outcome
454 assert data['data'][0]['id'] == self.ref_job_id 465​ assert data['data'][0]['id'] == self.ref_job_id
455 assert data['data'][0]['testcase']['name'] == self.ref_testcase_name 466​ assert data['data'][0]['testcase']['name'] == self.ref_testcase_name
467​ assert data['href'] == self.prefix + 'testcases/testcase/results'
468​ assert data['next'] == self.prefix + 'testcases/testcase/results?page=1'
469​ assert data['prev'] == None
470​ assert data['pages'] == 1
471​ assert data['total'] == 1
456 472​
457 def test_get_jobs_empty_results(self): 473​ def test_get_jobs_empty_results(self):
458 r = self.app.get('/api/v1.0/jobs/%s/results' % self.ref_job_id) 474​ r = self.app.get('/api/v1.0/jobs/%s/results' % self.ref_job_id)
459 475​
460 data = json.loads(r.data) 476​ data = json.loads(r.data)
461 477​
462 assert r.status_code == 404 478​ assert r.status_code == 404
463 assert data['message'] == "Job not found" 479​ assert data['message'] == "Job not found"
464 480​
465 def test_get_jobs_results_with_jsonp(self): 481​ def test_get_jobs_results_with_jsonp(self):
466 self.test_create_result() 482​ self.test_create_result()
467 483​
468 r = self.app.get('/api/v1.0/testcases/%s/results?callback=wat' % self.ref_testcase_name) 484​ r = self.app.get('/api/v1.0/testcases/%s/results?callback=wat' % self.ref_testcase_name)
469 485​
470 assert r.data.startswith('wat(') 486​ assert r.data.startswith('wat(')
471 data = json.loads(r.data[4:-2]) 487​ data = json.loads(r.data[4:-2])
472 488​
473 assert r.status_code == 200 489​ assert r.status_code == 200
474 assert data['data'][0]['outcome'] == self.ref_outcome 490​ assert data['data'][0]['outcome'] == self.ref_outcome
475 assert data['data'][0]['id'] == self.ref_job_id 491​ assert data['data'][0]['id'] == self.ref_job_id
476 assert data['data'][0]['testcase']['name'] == self.ref_testcase_name 492​ assert data['data'][0]['testcase']['name'] == self.ref_testcase_name
493​ assert data['href'] == self.prefix + 'testcases/testcase/results?callback=wat'
494​ assert data['next'] == self.prefix + 'testcases/testcase/results?callback=wat&page=1'
495​ assert data['prev'] == None
496​ assert data['pages'] == 1
497​ assert data['total'] == 1