Differential D911

createhdds: work around #1351352, fix support_server fails(?)
AbandonedPublic
Actions

Authored by adamwill on Jun 29 2016, 9:43 PM.

Details

Reviewers
None
Commits
rOPENQA5a5542b44d39: createhdds: work around #1351352, fix support_server fails(?)
rOPENQA42e9f00ed94b: createhdds: work around #1351352, fix support_server fails(?)
rOPENQA4adb2187575b: createhdds: work around #1351352, fix support_server fails(?)
Summary

First of all, this works around #1351352 for all virt-builder
images. As described in the bug, it seems that if you build
an updated f24 image with an selinux-relabel at present, it
gets stuck in a boot loop; the workaround is to set SELinux
to permissive for the first boot by editing /etc/selinux/config
then restore it with a firstboot-command (which will run when
createhdds lets the system reboot after the relabel, logs in,
and shuts down, so when openQA boots the image, SELinux should
be in enforcing mode).

Secondly, we patch some kernel params out of grub.cfg which
virt-builder puts in there (again, with a firstboot-command)
in support.commands. The support_server jobs are failing
sometimes in prod, and from the video, it looks like the login
prompt shows briefly in one video mode, then the console
switches modes and the prompt disappears; if openQA doesn't
run a needle match while the prompt is briefly visible, it fails
to log in. I think dropping these params (especially console=
tty0) should avoid that.

Test Plan

Check that creation of all virt-builder images works
and that support-server test runs with the new support server
image (I didn't bump the imgver because I don't feel like doing
a PR for openqa_fedora too...). The failure is kinda intermittent
though.

Diff Detail

Repository
rOPENQA fedora_openqa
Branch
relabel-permissive
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 694
Build 694: arc lint + arc unit
adamwill retitled this revision from to createhdds: work around #1351352, fix support_server fails(?).Jun 29 2016, 9:43 PM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added a reviewer: garretraziel.
adamwill added a comment.Jun 29 2016, 9:54 PM

staging is on this branch of the tools ATM, and I've actually ninja'd a rebuild of the support-server disk image with these changes onto prod too (though its git checkout is still develop), because I want to see if the fix actually works and the bug only seems to happen often on prod (probably a speed thing).

Here are some cases of the support-server fail:

https://openqa.fedoraproject.org/tests/23985
https://openqa.fedoraproject.org/tests/23884
https://openqa.fedoraproject.org/tests/23790
https://openqa.fedoraproject.org/tests/23950

if you watch the videos carefully, you can see the login prompt appear briefly around 3 secs into boot then some kind of mode switch occurs very shortly afterwards and the prompt is lost.

The other fix is the important one here, though - without that, all virt-builder image builds fail, so when the upgrade images age out in a few days they'll fail to rebuild.

adamwill added a comment.EditedJun 29 2016, 10:38 PM

Interestingly the same issue actually affects the minimal image, but we're shielded from it there (at least through upgrade_preinstall) because in the test where we use that image, we never actually check for a login prompt on tty1; we use $self->boot_to_login_screen(), which winds up just doing a wait_still_screen, then switch to tty3. The support_server test uses _console_wait_login, which *does* login on tty1. The minimal upgrade test does use _console_wait_login after the upgrade, though, I wonder if we've ever failed there...

adamwill updated this revision to Diff 2326.Jun 30 2016, 6:00 AM

replace virt-builder kernel args with 'rhgb quiet'

garretraziel added a comment.Jul 1 2016, 8:07 AM

Is this really necessary if we're switching to virt-install in D917?

adamwill added a comment.Jul 1 2016, 3:41 PM

Right, as I said in D917, it makes this unnecessary. But if you want to take a bit of time to review D917 we should merge this in the mean time.

D917 seems to be working fine except a change in ansible's git behaviour caused us to lose all the goddamn images it had built on stg :( Now rebuilding them.

garretraziel resigned from this revision.Jul 8 2016, 12:03 PM
garretraziel removed a reviewer: garretraziel.

This is no longer needed.

adamwill abandoned this revision.Aug 3 2016, 8:34 PM

Indeed, no longer needed.

Revision Contents

PathPackages
Mtools/desktop.commands (11 lines)
Mtools/kde.commands (11 lines)
Mtools/minimal.commands (11 lines)
Mtools/server.commands (11 lines)
Mtools/support.commands (11 lines)

Diff 2326

View Options

tools/desktop.commands

1root-password password:weakpassword 1​root-password password:weakpassword
2update 2​update
3selinux-relabel 3​selinux-relabel
4install @workstation-product-environment 4​install @workstation-product-environment
5​# set permissive mode for first boot (where relabel happens) to ensure
6​# label issues don't break the relabel - RHBZ #1351352
7​edit /etc/selinux/config:s/SELINUX=.*/SELINUX=permissive/
5firstboot-command useradd -m -p '' test 8​firstboot-command useradd -m -p '' test
6firstboot-command echo 'test:weakpassword' | chpasswd 9​firstboot-command echo 'test:weakpassword' | chpasswd
7# we do this on firstboot (i.e. when createhdds boots to do the selinux 10​# we do this on firstboot (i.e. when createhdds boots to do the selinux
8# relabel) so the selinux label on the symlink is correct; see RHBZ #1349586 11​# relabel) so the selinux label on the symlink is correct; see RHBZ #1349586
9firstboot-command systemctl set-default graphical.target 12​firstboot-command systemctl set-default graphical.target
10# workaround RHBZ #1349664 13​# workaround RHBZ #1349664
11firstboot-command systemctl enable gdm.service 14​firstboot-command systemctl enable gdm.service
15​# make cmdline params more like a real install (virt-builder sets
16​# this stuff), avoids login prompt disappearing and makes sure we
17​# catch plymouth bugs in upgrade
18​firstboot-command sed -i -e 's/console=tty0//' /boot/grub2/grub.cfg
19​firstboot-command sed -i -e 's/console=ttyS0,115200//' /boot/grub2/grub.cfg
20​firstboot-command sed -i -e 's/rd_NO_PLYMOUTH/rhgb quiet/' /boot/grub2/grub.cfg
21​# reset selinux to enforcing for openQA boot
22​firstboot-command sed -i -e 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config
View Options

tools/kde.commands

1root-password password:weakpassword 1​root-password password:weakpassword
2update 2​update
3selinux-relabel 3​selinux-relabel
4# the imsettings-qt is to work around RHBZ #1349743 4​# the imsettings-qt is to work around RHBZ #1349743
5install @kde-desktop-environment,imsettings-qt 5​install @kde-desktop-environment,imsettings-qt
6delete /etc/systemd/system/multi-user.target.wants/initial-setup-text.service 6​delete /etc/systemd/system/multi-user.target.wants/initial-setup-text.service
7delete /etc/systemd/system/graphical.target.wants/initial-setup-graphical.service 7​delete /etc/systemd/system/graphical.target.wants/initial-setup-graphical.service
8​# set permissive mode for first boot (where relabel happens) to ensure
9​# label issues don't break the relabel - RHBZ #1351352
10​edit /etc/selinux/config:s/SELINUX=.*/SELINUX=permissive/
8firstboot-command useradd -m -p '' test 11​firstboot-command useradd -m -p '' test
9firstboot-command echo 'test:weakpassword' | chpasswd 12​firstboot-command echo 'test:weakpassword' | chpasswd
10# we do this on firstboot (i.e. when createhdds boots to do the selinux 13​# we do this on firstboot (i.e. when createhdds boots to do the selinux
11# relabel) so the selinux label on the symlink is correct; see RHBZ #1349586 14​# relabel) so the selinux label on the symlink is correct; see RHBZ #1349586
12firstboot-command systemctl set-default graphical.target 15​firstboot-command systemctl set-default graphical.target
13# workaround RHBZ #1349664 16​# workaround RHBZ #1349664
14firstboot-command systemctl enable sddm.service 17​firstboot-command systemctl enable sddm.service
18​# make cmdline params more like a real install (virt-builder sets
19​# this stuff), avoids login prompt disappearing and makes sure we
20​# catch plymouth bugs in upgrade
21​firstboot-command sed -i -e 's/console=tty0//' /boot/grub2/grub.cfg
22​firstboot-command sed -i -e 's/console=ttyS0,115200//' /boot/grub2/grub.cfg
23​firstboot-command sed -i -e 's/rd_NO_PLYMOUTH/rhgb quiet/' /boot/grub2/grub.cfg
24​# reset selinux to enforcing for openQA boot
25​firstboot-command sed -i -e 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config
View Options

tools/minimal.commands

1root-password password:weakpassword 1​root-password password:weakpassword
2update 2​update
3selinux-relabel 3​selinux-relabel
4​# set permissive mode for first boot (where relabel happens) to ensure
5​# label issues don't break the relabel - RHBZ #1351352
6​edit /etc/selinux/config:s/SELINUX=.*/SELINUX=permissive/
7​# make cmdline params more like a real install (virt-builder sets
8​# this stuff), avoids login prompt disappearing and makes sure we
9​# catch plymouth bugs in upgrade
10​firstboot-command sed -i -e 's/console=tty0//' /boot/grub2/grub.cfg
11​firstboot-command sed -i -e 's/console=ttyS0,115200//' /boot/grub2/grub.cfg
12​firstboot-command sed -i -e 's/rd_NO_PLYMOUTH/rhgb quiet/' /boot/grub2/grub.cfg
13​# reset selinux to enforcing for openQA boot
14​firstboot-command sed -i -e 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config
View Options

tools/server.commands

1root-password password:weakpassword 1​root-password password:weakpassword
2update 2​update
3selinux-relabel 3​selinux-relabel
4install @server-product-environment 4​install @server-product-environment
5​# set permissive mode for first boot (where relabel happens) to ensure
6​# label issues don't break the relabel - RHBZ #1351352
7​edit /etc/selinux/config:s/SELINUX=.*/SELINUX=permissive/
5firstboot-command useradd -m -p '' test 8​firstboot-command useradd -m -p '' test
6firstboot-command echo 'test:weakpassword' | chpasswd 9​firstboot-command echo 'test:weakpassword' | chpasswd
10​# make cmdline params more like a real install (virt-builder sets
11​# this stuff), avoids login prompt disappearing and makes sure we
12​# catch plymouth bugs in upgrade
13​firstboot-command sed -i -e 's/console=tty0//' /boot/grub2/grub.cfg
14​firstboot-command sed -i -e 's/console=ttyS0,115200//' /boot/grub2/grub.cfg
15​firstboot-command sed -i -e 's/rd_NO_PLYMOUTH/rhgb quiet/' /boot/grub2/grub.cfg
16​# reset selinux to enforcing for openQA boot
17​firstboot-command sed -i -e 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config
View Options

tools/support.commands

1root-password password:weakpassword 1​root-password password:weakpassword
2update 2​update
3selinux-relabel 3​selinux-relabel
4install scsi-target-utils,nfs-utils,dnsmasq 4​install scsi-target-utils,nfs-utils,dnsmasq
5​# set permissive mode for first boot (where relabel happens) to ensure
6​# label issues don't break the relabel - RHBZ #1351352
7​edit /etc/selinux/config:s/SELINUX=.*/SELINUX=permissive/
8​# make cmdline params more like a real install (virt-builder sets
9​# this stuff), avoids login prompt disappearing and makes sure we
10​# catch plymouth bugs in upgrade
11​firstboot-command sed -i -e 's/console=tty0//' /boot/grub2/grub.cfg
12​firstboot-command sed -i -e 's/console=ttyS0,115200//' /boot/grub2/grub.cfg
13​firstboot-command sed -i -e 's/rd_NO_PLYMOUTH/rhgb quiet/' /boot/grub2/grub.cfg
14​# reset selinux to enforcing for openQA boot
15​firstboot-command sed -i -e 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config