Add API functional tests
ClosedPublic
Actions

Authored by mkrizek on Apr 18 2014, 1:38 PM.

Details

Reviewers

kparal
tflink
jskladan

Maniphest Tasks

T137: Add API functional tests

Commits

rRSDB5b3ad1fc1007: Add API functional tests

Summary

It doesn't cover the whole thing (about 71%, says pytest-cov) but should suffice I guess.

Fixes T137

Test Plan

Itself :)

Diff Detail

Lint

Lint Skipped

Unit

Unit Tests Skipped

resultsdb/models/results.py
45 ↗	(On Diff #167)	This is from D61 and shouldn't be here. I did arc diff --create feature/T134schemafix and added dependency here but it still shows up in diff here, not sure why...

mkrizek added a reviewer: kparal.Apr 18 2014, 2:01 PM

Overall, the content looks good to me. However, I'd like to see all the monkeypatch statements go away. If you use setup_class() instead of setup_method(), the code is run for the class once instead of before every method.

@classmethod
def setup_class(cls):
    resultsdb.db.db_url = 'sqlite://'
    # other setup

That way, you can get rid of the monkeypatches, use an in-memory database for speed and not have to deal with all the tempfiles. The in-memory thing isn't as important as getting rid of the monkeypatches, though.

testing/functest_api.py
32	you can create a sqlite in-memory database using the url `sqlite://`. Using an in-memory database should make the tests go faster

Use in-memory db in functest

Remove unused imports

Looks good to me

Closed by commit rRSDB5b3ad1fc1007 (authored by @mkrizek).

Revision Contents

		Path
M		.gitignore (2 lines)
A	M	pytest.ini (4 lines)
M		requirements.txt (2 lines)
A	M	testing/__init__.py
A	M	testing/conftest.py (27 lines)
A	M	testing/functest_api.py (399 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	167	2477266		Apr 18 2014, 1:38 PM	★	★
Diff 2	168	2477266	- Use in-memory db in functest	Apr 22 2014, 7:11 AM	★	★
Diff 3	169	2477266	- Remove unused imports	Apr 22 2014, 8:36 AM	★	★
Diff 4	261	65cfdf9	Commit rRSDB5b3ad1fc100767a06efbadee32d081571dc7d826	May 12 2014, 9:05 AM	★	★

Commit	Tree	Parents	Author	Summary	Date
7938bca679b9	fd210a14437e	9ea6eabdd10e	Martin Krizek	Remove unused imports	Apr 22 2014, 8:35 AM
9ea6eabdd10e	69ad32ef9abe	1a2101f2894b	Martin Krizek	Use in-memory db in functest	Apr 22 2014, 7:09 AM
1a2101f2894b	efd2ce61907b	ecc43f4a719b	Martin Krizek	Add API functional tests (Show More…)	Apr 18 2014, 1:25 PM
ecc43f4a719b	bcb7b3e924c0	2477266fdddc	Martin Krizek	Fix db schema so it works on sqlite and postgres (Show More…)	Apr 18 2014, 1:22 PM

	Status	Author	Revision
	Closed	mkrizek	D62 Add API functional tests
	Closed	mkrizek	D61 Fix db schema so it works on sqlite and postgres

Diff 261

View Options

.gitignore

1	*.pyc	1	*.pyc
2	*.pyo	2	*.pyo
3	*.swp	3	*.swp
4	__pycache__	4	__pycache__
5	.ropeproject/	5	.ropeproject/
6	conf/settings.py	6	conf/settings.py
7	*.sqlite	7	*.sqlite
8	*.sass-cache	8	*.sass-cache
9	*build/	9	*build/
10	*dist/	10	*dist/
11	.egg	11	.egg
12	/env	12	/env
		13	.coverage
		14	htmlcov/

View Options

pytest.ini

This file was added.

		1		[pytest]
		2		minversion = 2.0
		3		python_functions=test should
		4		python_files=test_* functest_*

View Options

requirements.txt

1	Flask>=0.9	1	Flask>=0.9
2	Flask-SQLAlchemy>=0.16	2	Flask-SQLAlchemy>=0.16
3	SQLAlchemy>= 0.7	3	SQLAlchemy>= 0.7
4	WTForms>1.0	4	WTForms>1.0
5	Flask-WTF==0.8	5	Flask-WTF==0.8
6	#FIXME: Flask-WTF > 0.8 introduced API change for imports	6	#FIXME: Flask-WTF > 0.8 introduced API change for imports
7	Flask-Login>=0.2.2	7	Flask-Login>=0.2.2
8	Flask-RESTful >= 0.2.5	8	Flask-RESTful >= 0.2.5
		9	pytest>=2.4.2
		10	pytest-cov>=1.6
9	# although this is a Flask-Restful dependency, it was not installed, adding manualy	11	# although this is a Flask-Restful dependency, it was not installed, adding manualy
10	six	12	six
11	iso8601 >= 0.1.4	13	iso8601 >= 0.1.4

View Options

testing/init.py

This file was added.

This is an empty file.

View Options

testing/conftest.py

This file was added.

		1		def pytest_addoption(parser):
		2		"""
		3		Add an option to the py.test parser to detect when the functional tests
		4		should be detected and run
		5		"""
		6
		7		parser.addoption('-F', '--functional', action='store_true', default=False,
		8		help='Add functional tests')
		9
		10
		11		def pytest_ignore_collect(path, config):
		12		"""Prevents collection of any files named functest* to speed up non
		13		integration tests"""
		14		if path.fnmatch('functest'):
		15		try:
		16		is_functional = config.getvalue('functional')
		17		except KeyError:
		18		return True
		19
		20		return not is_functional
		21
		22
		23		def pytest_configure(config):
		24		"""Called after command line options have been parsed and all plugins and
		25		initial conftest files been loaded."""
		26
		27		pass

View Options

testing/functest_api.py

This file was added.

1		# Copyright 2014, Red Hat, Inc.
2		#
3		# This program is free software; you can redistribute it and/or modify
4		# it under the terms of the GNU General Public License as published by
5		# the Free Software Foundation; either version 2 of the License, or
6		# (at your option) any later version.
7		#
8		# This program is distributed in the hope that it will be useful,
9		# but WITHOUT ANY WARRANTY; without even the implied warranty of
10		# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11		# GNU General Public License for more details.
12		#
13		# You should have received a copy of the GNU General Public License along
14		# with this program; if not, write to the Free Software Foundation, Inc.,
15		# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
16		#
17		# Authors:
18		# Martin Krizek <mkrizek@redhat.com>
19
20		import json
21		import datetime
22
23		import resultsdb
24		import resultsdb.cli
25
26		class TestFuncApi():
27		@classmethod
28		def setup_class(cls):
29		resultsdb.app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite://'
30
31		def setup_method(self, method):
32		self.app = resultsdb.app.test_client()
	tflinkUnsubmitted Not Done you can create a sqlite in-memory database using the url `sqlite://`. Using an in-memory database should make the tests go faster
33		resultsdb.cli.initialize_db()
34
35		self.ref_testcase_name = "testcase"
36		self.ref_testcase_url = "http://fedoraqa.fedoraproject.org/%s" % self.ref_testcase_name
37		self.ref_job_id = 1
38		self.ref_job_url = "http://fedoraqa.fedoraproject.org"
39		self.ref_job_name = "F20 Virtualization Testday"
40		self.ref_status = "SCHEDULED"
41		self.ref_outcome = "PASSED"
42		self.ref_result_id = 1
43		self.ref_result_data = {'data': 'fakedata', 'data1': ['fakedata1'], 'data2': 1}
44		self.ref_result_summary = "1 PASSED, 0 FAILED"
45		self.ref_result_log_url = "http://fedoraqa.fedoraproject.org/logs"
46
47		def test_create_testcase(self):
48		ref_data = json.dumps({'name': self.ref_testcase_name, 'url': self.ref_testcase_url})
49
50		r = self.app.post('/api/v1.0/testcases', data=ref_data, content_type='application/json')
51
52		data = json.loads(r.data)
53
54		assert r.status_code == 201
55		assert data['name'] == self.ref_testcase_name
56		assert data['url'] == self.ref_testcase_url
57
58		def test_create_duplicate_testcase(self):
59		self.test_create_testcase()
60
61		ref_data = json.dumps({'name': self.ref_testcase_name, 'url': self.ref_testcase_url})
62
63
64		r = self.app.post('/api/v1.0/testcases', data=ref_data, content_type='application/json')
65
66		data = json.loads(r.data)
67
68		assert r.status_code == 400
69		assert data['message'] == "Testcase with this name already exists"
70
71		def test_update_testcase(self):
72		self.test_create_testcase()
73
74		ref_data = json.dumps({'url': self.ref_testcase_url})
75
76		r = self.app.put('/api/v1.0/testcases/%s' % self.ref_testcase_name, data=ref_data, content_type='application/json')
77
78		data = json.loads(r.data)
79
80		assert r.status_code == 200
81		assert data['name'] == self.ref_testcase_name
82		assert data['url'] == self.ref_testcase_url
83
84		def test_update_invalid_testcase(self):
85		ref_data = json.dumps({'url': self.ref_testcase_url})
86
87		r = self.app.put('/api/v1.0/testcases/%s' % self.ref_testcase_name, data=ref_data, content_type='application/json')
88
89		data = json.loads(r.data)
90
91		assert r.status_code == 404
92		assert data['message'] == "Testcase not found"
93
94		def test_get_testcase(self):
95		self.test_create_testcase()
96
97		r = self.app.get('/api/v1.0/testcases/%s' % self.ref_testcase_name, content_type='application/json')
98
99		data = json.loads(r.data)
100
101		assert r.status_code == 200
102		assert data['name'] == self.ref_testcase_name
103		assert data['url'] == self.ref_testcase_url
104
105		def test_get_invalid_testcase(self):
106		r = self.app.get('/api/v1.0/testcases/%s' % self.ref_testcase_name, content_type='application/json')
107
108		data = json.loads(r.data)
109
110		assert r.status_code == 404
111		assert data['message'] == "Testcase not found"
112
113		def test_get_testcases(self):
114		self.test_create_testcase()
115
116		r = self.app.get('/api/v1.0/testcases', content_type='application/json')
117
118		data = json.loads(r.data)
119
120		assert r.status_code == 200
121		assert data['data'][0]['name'] == self.ref_testcase_name
122		assert data['data'][0]['url'] == self.ref_testcase_url
123
124		def test_get_empty_testcases(self):
125		r = self.app.get('/api/v1.0/testcases', content_type='application/json')
126
127		data = json.loads(r.data)
128
129		assert r.status_code == 200
130		assert data['data'] == []
131
132		def test_create_job(self):
133		ref_data = json.dumps({'ref_url': self.ref_job_url, 'status': self.ref_status, 'name': self.ref_job_name})
134
135		r = self.app.post('/api/v1.0/jobs', data=ref_data, content_type='application/json')
136
137		data = json.loads(r.data)
138
139		assert r.status_code == 201
140		assert data['ref_url'] == self.ref_job_url
141		assert data['status'] == self.ref_status
142
143		def test_create_invalid_job(self):
144		ref_data = json.dumps({'ref_url': self.ref_job_url, 'status': 'INVALIDFAKE'})
145
146		r = self.app.post('/api/v1.0/jobs', data=ref_data, content_type='application/json')
147
148		data = json.loads(r.data)
149
150		assert r.status_code == 400
151		assert data['message'].startswith("status must be one of")
152
153		def test_update_job(self):
154		self.test_create_job()
155
156		ref_status = "RUNNING"
157		ref_data = json.dumps({'status': ref_status})
158
159		r = self.app.put('/api/v1.0/jobs/%d' % self.ref_job_id, data=ref_data, content_type='application/json')
160
161		data = json.loads(r.data)
162
163		assert r.status_code == 200
164		assert data['status'] == ref_status
165		assert data['start_time'] is not None
166
167		def test_update_job_to_completed(self):
168		self.test_create_job()
169
170		ref_status = "COMPLETED"
171		ref_data = json.dumps({'status': ref_status})
172
173		r = self.app.put('/api/v1.0/jobs/%d' % self.ref_job_id, data=ref_data, content_type='application/json')
174
175		data = json.loads(r.data)
176
177		assert r.status_code == 200
178		assert data['status'] == ref_status
179		assert data['start_time'] is not None
180		assert data['end_time'] is not None
181
182		def test_update_invalid_job(self):
183		ref_status = "RUNNING"
184		ref_data = json.dumps({'status': ref_status})
185
186		r = self.app.put('/api/v1.0/jobs/%d' % self.ref_job_id, data=ref_data, content_type='application/json')
187
188		data = json.loads(r.data)
189
190		assert r.status_code == 404
191		assert data['message'] == "Job not found"
192
193		def test_update_invalid_status_job(self):
194		self.test_create_job()
195
196		ref_status = "INVALIDFAKE"
197		ref_data = json.dumps({'status': ref_status})
198
199		r = self.app.put('/api/v1.0/jobs/%d' % self.ref_job_id, data=ref_data, content_type='application/json')
200
201		data = json.loads(r.data)
202
203		assert r.status_code == 400
204		assert data['message'].startswith("status must be one of")
205
206		def test_get_job(self):
207		self.test_create_job()
208
209		r = self.app.get('/api/v1.0/jobs/%d' % self.ref_job_id, content_type='application/json')
210
211		data = json.loads(r.data)
212
213		assert r.status_code == 200
214		assert data['id'] == self.ref_job_id
215		assert data['status'] == self.ref_status
216
217		def test_get_invalid_job(self):
218		r = self.app.get('/api/v1.0/jobs/%d' % self.ref_job_id, content_type='application/json')
219
220		data = json.loads(r.data)
221
222		assert r.status_code == 404
223		assert data['message'] == "Job not found"
224
225		def test_get_jobs(self):
226		self.test_create_job()
227
228		r = self.app.get('/api/v1.0/jobs', content_type='application/json')
229
230		data = json.loads(r.data)
231
232		assert r.status_code == 200
233		assert data['data'][0]['id'] == self.ref_job_id
234		assert data['data'][0]['status'] == self.ref_status
235
236		def test_get_jobs_params(self):
237		self.test_create_job()
238
239		ref_data = json.dumps({'page': 0,
240		'limit': 10,
241		'status': self.ref_status,
242		'since': datetime.datetime(1970, 1, 1).isoformat(' ')})
243
244		r = self.app.get('/api/v1.0/jobs', data=ref_data, content_type='application/json')
245
246		data = json.loads(r.data)
247
248		assert r.status_code == 200
249		assert data['data'][0]['id'] == self.ref_job_id
250		assert data['data'][0]['status'] == self.ref_status
251
252		def test_get_empty_jobs(self):
253		r = self.app.get('/api/v1.0/jobs', content_type='application/json')
254
255		data = json.loads(r.data)
256
257		assert r.status_code == 200
258		assert data['data'] == []
259
260		def test_create_result(self):
261		self.test_create_testcase()
262		self.test_create_job()
263		self.test_update_job()
264
265		ref_data = json.dumps({'outcome': self.ref_outcome,
266		'job_id': self.ref_job_id,
267		'testcase_name': self.ref_testcase_name,
268		'result_data': self.ref_result_data,
269		'summary': self.ref_result_summary,
270		'log_url': self.ref_result_log_url})
271
272		r = self.app.post('/api/v1.0/results', data=ref_data, content_type='application/json')
273
274		data = json.loads(r.data)
275
276		assert r.status_code == 201
277		assert data['outcome'] == self.ref_outcome
278		assert data['id'] == self.ref_job_id
279		assert data['testcase']['name'] == self.ref_testcase_name
280
281		def test_create_invalid_job_result(self):
282		self.test_create_testcase()
283		# not creating any jobs
284
285		ref_data = json.dumps({'outcome': self.ref_outcome, 'job_id': self.ref_job_id, 'testcase_name': self.ref_testcase_name})
286
287		r = self.app.post('/api/v1.0/results', data=ref_data, content_type='application/json')
288
289		data = json.loads(r.data)
290
291		assert r.status_code == 404
292		assert data['message'] == "Job not found"
293
294		def test_create_invalid_outcome_result(self):
295		self.test_create_testcase()
296		self.test_create_job()
297		self.test_update_job()
298
299		ref_data = json.dumps({'outcome': 'FAKEOUTCOME', 'job_id': self.ref_job_id, 'testcase_name': self.ref_testcase_name})
300
301		r = self.app.post('/api/v1.0/results', data=ref_data, content_type='application/json')
302
303		data = json.loads(r.data)
304
305		assert r.status_code == 400
306		assert data['message'].startswith("outcome must be one of")
307
308		def test_create_invalid_result(self):
309		self.test_create_testcase()
310		self.test_create_job() # create SCHEDULED job
311
312		ref_data = json.dumps({'outcome': self.ref_outcome, 'job_id': self.ref_job_id, 'testcase_name': self.ref_testcase_name})
313
314		r = self.app.post('/api/v1.0/results', data=ref_data, content_type='application/json')
315
316		data = json.loads(r.data)
317
318		assert r.status_code == 400
319		assert data['message'] == "Job not running"
320
321		def test_get_result(self):
322		self.test_create_result()
323
324		r = self.app.get('/api/v1.0/results/%d' % self.ref_result_id, content_type='application/json')
325
326		data = json.loads(r.data)
327
328		assert r.status_code == 200
329		assert data['outcome'] == self.ref_outcome
330		assert data['id'] == self.ref_job_id
331		assert data['testcase']['name'] == self.ref_testcase_name
332
333		def test_get_invalid_result(self):
334		r = self.app.get('/api/v1.0/results/%d' % self.ref_result_id, content_type='application/json')
335
336		data = json.loads(r.data)
337
338		assert r.status_code == 404
339		assert data['message'] == "Result not found"
340
341		def test_get_results(self):
342		self.test_create_result()
343
344		r = self.app.get('/api/v1.0/results', content_type='application/json')
345
346		data = json.loads(r.data)
347
348		assert r.status_code == 200
349		assert data['data'][0]['outcome'] == self.ref_outcome
350		assert data['data'][0]['id'] == self.ref_job_id
351		assert data['data'][0]['testcase']['name'] == self.ref_testcase_name
352
353		def test_get_empty_results(self):
354		r = self.app.get('/api/v1.0/results', content_type='application/json')
355
356		data = json.loads(r.data)
357
358		assert r.status_code == 200
359		assert data['data'] == []
360
361		def test_get_testcases_results(self):
362		self.test_create_result()
363
364		r = self.app.get('/api/v1.0/testcases/%s/results' % self.ref_testcase_name, content_type='application/json')
365
366		data = json.loads(r.data)
367
368		assert r.status_code == 200
369		assert data['data'][0]['outcome'] == self.ref_outcome
370		assert data['data'][0]['id'] == self.ref_job_id
371		assert data['data'][0]['testcase']['name'] == self.ref_testcase_name
372
373		def test_get_testcases_empty_results(self):
374		r = self.app.get('/api/v1.0/testcases/%s/results' % self.ref_testcase_name, content_type='application/json')
375
376		data = json.loads(r.data)
377
378		assert r.status_code == 404
379		assert data['message'] == "Testcase not found"
380
381		def test_get_jobs_results(self):
382		self.test_create_result()
383
384		r = self.app.get('/api/v1.0/testcases/%s/results' % self.ref_testcase_name, content_type='application/json')
385
386		data = json.loads(r.data)
387
388		assert r.status_code == 200
389		assert data['data'][0]['outcome'] == self.ref_outcome
390		assert data['data'][0]['id'] == self.ref_job_id
391		assert data['data'][0]['testcase']['name'] == self.ref_testcase_name
392
393		def test_get_jobs_empty_results(self):
394		r = self.app.get('/api/v1.0/jobs/%s/results' % self.ref_job_id, content_type='application/json')
395
396		data = json.loads(r.data)
397
398		assert r.status_code == 404
399		assert data['message'] == "Job not found"

Add API functional testsClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 261

.gitignore

pytest.ini

requirements.txt

testing/__init__.py

testing/conftest.py

testing/functest_api.py

Add API functional tests
ClosedPublic
Actions

testing/init.py