Differential D774

fix pytest-cov instructions and running test suite multiple times
ClosedPublic
Actions

Authored by kparal on Mar 10 2016, 1:45 PM.

Details

Reviewers
lbrabec
mkrizek
jskladan
Group Reviewers
libtaskotron
Maniphest Tasks
T737: libtaskotron cov testing fails due to logs not getting cleared.
Commits
rLTRNdb49c66fde96: fix pytest-cov instructions and running test suite multiple times
Summary

Coverage instructions were not correctly using long option names (equal
sign needed). Also functest_logger.py could not have been run multiple
times in a row, because mem_handler was not erased. Bump pytest and
pytest-cov deps to avoid yet another weird error in pytest when
running test suite multiple times.

This fixes T737.

Test Plan

try

py.test -F testing/ testing/

before and after this patch. Now it should pass OK.
Then test

py.test --functional --cov-report term-missing --cov libtaskotron testing/
py.test --functional --cov-report=term-missing --cov=libtaskotron testing/

The first one should run the test suite twice, the second one only once.

Be sure to update your virtualenv from requirements.txt when trying the patched version.

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
kparal retitled this revision from to fix pytest-cov instructions and running test suite multiple times.Mar 10 2016, 1:45 PM
kparal updated this object.
kparal edited the test plan for this revision. (Show Details)
kparal added a reviewer: libtaskotron.
mkrizek accepted this revision.Mar 10 2016, 1:52 PM
mkrizek added a reviewer: mkrizek.
This revision is now accepted and ready to land.Mar 10 2016, 1:52 PM
jskladan accepted this revision.Mar 10 2016, 2:07 PM
jskladan added a reviewer: jskladan.
lbrabec accepted this revision.Mar 10 2016, 2:07 PM
lbrabec added a reviewer: lbrabec.
kparal added a comment.Mar 10 2016, 2:45 PM

Oh btw, kudos to @lbrabec for finding the mem_handler issue here.

tflink added a comment.Mar 10 2016, 4:32 PM

Is the fix to have newer pytest/pytest-cov or resetting the memory buffer or both?

kparal added a comment.Mar 11 2016, 7:58 AM
In D774#14778, @tflink wrote:

Is the fix to have newer pytest/pytest-cov or resetting the memory buffer or both?

Both. Resetting memory buffer fixes our bug, newer pytest fixes some other pytest bug which also occurred on duplicate execution. I bumped pytest-cov as well, simply because I tested the latest version as my first step and it worked well, so I concluded we can update it also. If you want to avoid bumping pytest-cov, I can test properly even with the old version.

tflink added a comment.Mar 11 2016, 1:25 PM

Both. Resetting memory buffer fixes our bug, newer pytest fixes some other pytest bug which also occurred on duplicate execution. I bumped pytest-cov as well, simply because I tested the latest version as my first step and it worked well, so I concluded we can update it also. If you want to avoid bumping pytest-cov, I can test properly even with the old version.

Nah, that isn't needed. Mostly wondering what fixed the issue both of those versions are in f24 so I'm not too worried about it.

Closed by commit rLTRNdb49c66fde96: fix pytest-cov instructions and running test suite multiple times (authored by kparal). · Explain WhyMar 11 2016, 2:54 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathPackages
Mlibtaskotron.spec (2 lines)
Mreadme.rst (4 lines)
Mrequirements.txt (4 lines)
Mtesting/functest_logger.py (2 lines)

Diff 1983

View Options

libtaskotron.spec

Show All 26 Lines
27 27​
28Requires: libtaskotron-config = %{version}-%{release} 28​Requires: libtaskotron-config = %{version}-%{release}
29Requires: python-jinja2 >= 2.8 29​Requires: python-jinja2 >= 2.8
30Requires: python-progressbar >= 2.3 30​Requires: python-progressbar >= 2.3
31Requires: python-requests >= 2.7.0 31​Requires: python-requests >= 2.7.0
32Requires: python-setuptools 32​Requires: python-setuptools
33Requires: PyYAML >= 3.11 33​Requires: PyYAML >= 3.11
34Requires: resultsdb_api >= 1.2.1 34​Requires: resultsdb_api >= 1.2.1
35BuildRequires: pytest >= 2.6.4 35​BuildRequires: pytest >= 2.7.3
36BuildRequires: python-devel 36​BuildRequires: python-devel
37BuildRequires: python-dingus >= 0.3.4 37​BuildRequires: python-dingus >= 0.3.4
38BuildRequires: python-mock >= 1.0.1 38​BuildRequires: python-mock >= 1.0.1
39BuildRequires: python-progressbar >= 2.3 39​BuildRequires: python-progressbar >= 2.3
40BuildRequires: python-requests >= 2.7.0 40​BuildRequires: python-requests >= 2.7.0
41BuildRequires: python-setuptools 41​BuildRequires: python-setuptools
42BuildRequires: PyYAML >= 3.11 42​BuildRequires: PyYAML >= 3.11
43BuildRequires: resultsdb_api >= 1.2.1 43​BuildRequires: resultsdb_api >= 1.2.1
▲ Show 20 Lines • Show All 298 Lines • Show Last 20 Lines
View Options

readme.rst

Show First 20 Lines • Show All 152 Lines • ▼ Show 20 Line(s)151to run unit tests, or::
153 py.test --functional testing/ 153​ py.test --functional testing/
154 154​
155to run both unit and functional tests (recommended). Make sure to use the 155​to run both unit and functional tests (recommended). Make sure to use the
156virtualenv-provided version of py.test, not the system one, otherwise you'll 156​virtualenv-provided version of py.test, not the system one, otherwise you'll
157encounter import errors. 157​encounter import errors.
158 158​
159You can also see a test coverage of the code, just run:: 159​You can also see a test coverage of the code, just run::
160 160​
161 py.test --functional --cov-report term-missing --cov libtaskotron testing/ 161​ py.test --functional --cov-report=term-missing --cov=libtaskotron testing/
162 162​
163A nice HTML-based representation is available if you use ``--cov-report html`` 163​A nice HTML-based representation is available if you use ``--cov-report=html``
164parameter instead. 164​parameter instead.
165 165​
166If you write new tests, be sure to run this to see whether the code is 166​If you write new tests, be sure to run this to see whether the code is
167sufficiently covered by your tests. 167​sufficiently covered by your tests.
168 168​
169 169​
170Building Documentation 170​Building Documentation
171====================== 171​======================
Show All 39 Lines
View Options

requirements.txt

Show All 9 Lines
10# lifecycle). If the Fedora libraries span multiple versions across releases, 10​# lifecycle). If the Fedora libraries span multiple versions across releases,
11# you can define a dependency range here instead of an exact version. 11​# you can define a dependency range here instead of an exact version.
12 12​
13dingus == 0.3.4 13​dingus == 0.3.4
14Jinja2 == 2.8 14​Jinja2 == 2.8
15munch == 2.0.2 15​munch == 2.0.2
16mock == 1.0.1 16​mock == 1.0.1
17paramiko == 1.15.1 17​paramiko == 1.15.1
18pytest == 2.6.4 18​pytest == 2.7.3
19pytest-cov == 1.8.1 19​pytest-cov == 2.2.1
20python-fedora == 0.5.5 20​python-fedora == 0.5.5
21progressbar == 2.3 21​progressbar == 2.3
22requests == 2.7.0 22​requests == 2.7.0
23PyYAML == 3.11 23​PyYAML == 3.11
24resultsdb_api == 1.2.1 24​resultsdb_api == 1.2.1
25Sphinx == 1.2.3 25​Sphinx == 1.2.3
View Options

testing/functest_logger.py

Show All 26 Lines15class TestLogger():
27 27​
28 def teardown_method(self, method): 28​ def teardown_method(self, method):
29 '''Run after every method''' 29​ '''Run after every method'''
30 # reset the list of root handlers 30​ # reset the list of root handlers
31 logging.getLogger().handlers = self.root_handlers 31​ logging.getLogger().handlers = self.root_handlers
32 # reset the stream handler level 32​ # reset the stream handler level
33 if logger.stream_handler: 33​ if logger.stream_handler:
34 logger.stream_handler.level = self.stream_level 34​ logger.stream_handler.level = self.stream_level
35​ # reset memory buffer in case we run this test several times in a row
36​ logger.mem_handler.buffer = []
35 37​
36 def test_logfile(self, tmpdir): 38​ def test_logfile(self, tmpdir):
37 '''Messages should be logged to file when enabled''' 39​ '''Messages should be logged to file when enabled'''
38 log_path = tmpdir.join('test.log').strpath 40​ log_path = tmpdir.join('test.log').strpath
39 41​
40 logger.init() 42​ logger.init()
41 fh = logger.add_filehandler(filelog_path=log_path) 43​ fh = logger.add_filehandler(filelog_path=log_path)
42 msg = 'This should appear in the log file' 44​ msg = 'This should appear in the log file'
▲ Show 20 Lines • Show All 45 Lines • Show Last 20 Lines