Differential D1149

Update testing: log packages in update and installed packages
ClosedPublic
Actions

Authored by adamwill on Feb 22 2017, 11:36 PM.

Details

Reviewers
jsedlak
Commits
rOPENQATESTS461f3a613259: Update testing: log packages in update and installed packages
Summary

This adds some logging related to the update testing workflow,
so we have some idea what we actually tested. We log precisely
which packages were actually downloaded from the update - this
is important as updates can be edited and when examining results
we'll want to know which packages actually got used. We also
add a new module which runs at the end of postinstall and tries
to figure out which packages from the update were installed in
the course of the test. This still isn't a guarantee the test
actually *tested them* in any way, but it at least means they
got installed successfully and didn't interfere with the test.

Test Plan

Run the update test workflow, check the logs get
uploaded and seem accurate (sometimes some RPM garbage messages
wind up in the package log, I'm not too worried about that at
present). Run the compose test workflow and check it didn't
break.

Diff Detail

Repository
rOPENQATESTS os-autoinst-distri-fedora
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
adamwill retitled this revision from to Update testing: log packages in update and installed packages.Feb 22 2017, 11:36 PM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added a reviewer: jsedlak.
jsedlak accepted this revision.Feb 23 2017, 12:39 PM

It works without problem.

As a side-note maybe we could come up with basic tests for critpath packages so that we could test something more than "yup, it still boots even with this update installed".

Anyway, great work!

This revision is now accepted and ready to land.Feb 23 2017, 12:39 PM
adamwill added a comment.Feb 23 2017, 4:14 PM

well, the tests we run cover *most* of the critpath functionality. I dunno how far I wanna go adding tests on this path, especially covering ground we could cover with Taskotron tests, but hey, why not. :) We *could* add a test that just does "dnf install *.rpm", I was thinking about that.

I'm working on the scheduler side ATM, didn't have enough bits ready to send a diff yesterday though.

Closed by commit rOPENQATESTS461f3a613259: Update testing: log packages in update and installed packages (authored by Adam Williamson <awilliam@redhat.com>). · Explain WhyFeb 23 2017, 10:51 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathPackages
Mlib/utils.pm (8 lines)
Mmain.pm (6 lines)
AMtests/_advisory_post.pm (28 lines)
Mtests/_advisory_update.pm (4 lines)

Diff 2930

View Options

lib/utils.pm

Show First 20 Lines • Show All 319 Lines • ▼ Show 20 Line(s)303sub _repo_setup_updates {
320 # download the packages 320​ # download the packages
321 my $version = lc(get_var("VERSION")); 321​ my $version = lc(get_var("VERSION"));
322 if ($version eq 'rawhide' || $version > 25) { 322​ if ($version eq 'rawhide' || $version > 25) {
323 # bodhi client 2.x 323​ # bodhi client 2.x
324 assert_script_run "bodhi updates download --updateid " . get_var("ADVISORY"), 300; 324​ assert_script_run "bodhi updates download --updateid " . get_var("ADVISORY"), 300;
325 } 325​ }
326 else { 326​ else {
327 # bodhi client 0.9 327​ # bodhi client 0.9
328 # latest git python-fedora fixes bug which makes bodhi -D UPDATE_ID fail 328​ # use git python-fedora for
329​ # https://github.com/fedora-infra/python-fedora/pull/192
330​ # until packages with that fix are pushed stable
329 assert_script_run "git clone https://github.com/fedora-infra/python-fedora.git"; 331​ assert_script_run "git clone https://github.com/fedora-infra/python-fedora.git";
330 assert_script_run "PYTHONPATH=python-fedora/ bodhi -D " . get_var("ADVISORY"), 300; 332​ assert_script_run "PYTHONPATH=python-fedora/ bodhi -D " . get_var("ADVISORY"), 300;
331 } 333​ }
334​ # log the exact packages in the update at test time, with their
335​ # source packages and epochs. log is uploaded by _advisory_update
336​ # and used for later comparison by _advisory_post
337​ assert_script_run 'rpm -qp *.rpm --qf "%{SOURCERPM} %{EPOCH} %{NAME}-%{VERSION}-%{RELEASE}\n" | sort -u > /var/log/updatepkgs.txt';
332 # create the repo metadata 338​ # create the repo metadata
333 assert_script_run "createrepo ."; 339​ assert_script_run "createrepo .";
334 # write a repo config file 340​ # write a repo config file
335 assert_script_run 'printf "[advisory]\nname=Advisory repo\nbaseurl=file:///opt/update_repo\nenabled=1\nmetadata_expire=3600\ngpgcheck=0" > /etc/yum.repos.d/advisory.repo'; 341​ assert_script_run 'printf "[advisory]\nname=Advisory repo\nbaseurl=file:///opt/update_repo\nenabled=1\nmetadata_expire=3600\ngpgcheck=0" > /etc/yum.repos.d/advisory.repo';
336 # run an update now 342​ # run an update now
337 script_run "dnf -y update", 300; 343​ script_run "dnf -y update", 300;
338} 344​}
339 345​
340sub repo_setup { 346​sub repo_setup {
341 # Run the appropriate sub-function for the job 347​ # Run the appropriate sub-function for the job
342 get_var("ADVISORY") ? _repo_setup_updates : _repo_setup_compose; 348​ get_var("ADVISORY") ? _repo_setup_updates : _repo_setup_compose;
343} 349​}
344 350​
View Options

main.pm

Show First 20 Lines • Show All 251 Lines • ▼ Show 20 Line(s)198sub load_postinstall_tests() {
252 # generic post-install test load 252​ # generic post-install test load
253 if (get_var("POSTINSTALL")) { 253​ if (get_var("POSTINSTALL")) {
254 my @pis = split(/ /, get_var("POSTINSTALL")); 254​ my @pis = split(/ /, get_var("POSTINSTALL"));
255 foreach my $pi (@pis) { 255​ foreach my $pi (@pis) {
256 autotest::loadtest "tests/${pi}.pm"; 256​ autotest::loadtest "tests/${pi}.pm";
257 } 257​ }
258 } 258​ }
259 259​
260​ # load the ADVISORY post-install test - this records which update
261​ # packages were actually installed during the test
262​ if (get_var("ADVISORY")) {
263​ autotest::loadtest "tests/_advisory_post.pm";
264​ }
265​
260 # we should shut down before uploading disk images 266​ # we should shut down before uploading disk images
261 if (get_var("STORE_HDD_1") || get_var("PUBLISH_HDD_1")) { 267​ if (get_var("STORE_HDD_1") || get_var("PUBLISH_HDD_1")) {
262 autotest::loadtest "tests/_console_shutdown.pm"; 268​ autotest::loadtest "tests/_console_shutdown.pm";
263 } 269​ }
264} 270​}
265 271​
266## LOADING STARTS HERE 272​## LOADING STARTS HERE
267 273​
Show All 26 Lines
View Options

tests/_advisory_post.pm

  • This file was added.
1​use base "installedtest";
2​use strict;
3​use testapi;
4​use utils;
5​
6​sub run {
7​ my $self = shift;
8​ # figure out which packages from the update actually got installed
9​ # (if any) as part of this test
10​ $self->root_console(tty=>3);
11​ assert_script_run 'rpm -qa --qf "%{SOURCERPM} %{EPOCH} %{NAME}-%{VERSION}-%{RELEASE}\n" | sort -u > /tmp/allpkgs.txt';
12​ # this finds lines which appear in both files
13​ # http://www.unix.com/unix-for-dummies-questions-and-answers/34549-find-matching-lines-between-2-files.html
14​ assert_script_run 'comm -12 /tmp/allpkgs.txt /var/log/updatepkgs.txt > /var/log/testedpkgs.txt';
15​ upload_logs "/var/log/testedpkgs.txt";
16​}
17​
18​sub test_flags {
19​ # without anything - rollback to 'lastgood' snapshot if failed
20​ # 'fatal' - whole test suite is in danger if this fails
21​ # 'milestone' - after this test succeeds, update 'lastgood'
22​ # 'important' - if this fails, set the overall state to 'fail'
23​ return { fatal => 1 };
24​}
25​
26​1;
27​
28​# vim: set sw=4 et:
View Options

tests/_advisory_update.pm

1use base "installedtest"; 1​use base "installedtest";
2use strict; 2​use strict;
3use testapi; 3​use testapi;
4use utils; 4​use utils;
5 5​
6sub run { 6​sub run {
7 my $self = shift; 7​ my $self = shift;
8 # do the 'repo setup' steps, which set up a repo containing the 8​ # do the 'repo setup' steps, which set up a repo containing the
9 # update packages and run 'dnf update' 9​ # update packages and run 'dnf update'
10 $self->root_console(tty=>3); 10​ $self->root_console(tty=>3);
11 repo_setup; 11​ repo_setup;
12​ # upload the log of installed packages which repo_setup created
13​ # we do this here and not in repo_setup because this is the best
14​ # place to make sure it happens once and only once per job
15​ upload_logs "/var/log/updatepkgs.txt";
12 # reboot, in case any of the updates need a reboot to apply 16​ # reboot, in case any of the updates need a reboot to apply
13 script_run "reboot", 0; 17​ script_run "reboot", 0;
14} 18​}
15 19​
16sub test_flags { 20​sub test_flags {
17 # without anything - rollback to 'lastgood' snapshot if failed 21​ # without anything - rollback to 'lastgood' snapshot if failed
18 # 'fatal' - whole test suite is in danger if this fails 22​ # 'fatal' - whole test suite is in danger if this fails
19 # 'milestone' - after this test succeeds, update 'lastgood' 23​ # 'milestone' - after this test succeeds, update 'lastgood'
20 # 'important' - if this fails, set the overall state to 'fail' 24​ # 'important' - if this fails, set the overall state to 'fail'
21 return { fatal => 1 }; 25​ return { fatal => 1 };
22} 26​}
23 27​
241; 28​1;
25 29​
26# vim: set sw=4 et: 30​# vim: set sw=4 et: