add ARM support to openqa_fedora_tools
AbandonedPublic

Authored by garretraziel on Jan 21 2016, 2:17 PM.

Details

Reviewers
adamwill
Summary

I hate ARM and I want it to die horrible death.

openQA 4.3 (with KERNEL, INITRD and APPEND support) needed for this.
This adds possibility to test our ARM images with openQA. For some reason,
only available images (that I know) are disk images with Fedora preinstalled.
That means that we have to download XZed disk image, uncompress it, copy
kernel and initramfs out of it and then use direct kernel boot with QEMU
to boot it. There is some black magic with openQA's request (unfortunatelly,
we have to conditionally check whether we are planning ARM testcases or not,
because we are passing it completely different set of arguments).
download_image now passes whole path to iso/disk file (because we need full
path for uncompressing) and it uses basename before POST.
backports.lzma Python library is needed for uncompressing.

I will follow this DR with patches with actual ARM tests (that will also do
some variables rejiggling).

See wiki on some
notes on ARM with openQA.

Test Plan

Create ARM machine (see wiki), bind some random test to it and
use fedora-openqa-schedule to plan some test on it. Alternativelly, wait
until I will do some test for ARM.

Diff Detail

Repository
rOPENQA fedora_openqa
Branch
arcpatch-D729
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 433
Build 433: arc lint + arc unit
garretraziel retitled this revision from to add ARM support to openqa_fedora_tools.Jan 21 2016, 2:17 PM
garretraziel updated this object.
garretraziel edited the test plan for this revision. (Show Details)
garretraziel added reviewers: adamwill, jskladan.

Thanks for figuring that out! Good work. There are only two things about this that concern me:

  1. It ties scheduling pretty heavily to the openQA server box again, when I was trying to make them separable (I never got around to changing the scheduler to use the openQA ISO download function, but it's a trivial change).
  1. I don't think anything will clean up the extracted images, and they're going to eat quite a bit of disk space, aren't they?

The required kernel and initramfs are actually available for download, though fedfind doesn't give those locations at present it fairly easily could. See for e.g. https://kojipkgs.fedoraproject.org/mash/rawhide-20160121/rawhide/armhfp/os/images/pxeboot/ . Downloading both the disk image and the kernel and initramfs files is a bit less efficient in terms of transfer, I guess, but should make the code a lot less complex. We'd still be tied to the scheduler, but it'd be at least potentially easier to unpick (I could implement something like HDDURL, KERNELURL and INITRDURL for openQA along the same lines of ISOURL, I intentionally left the design such that that sort of thing should be relatively easy to do). WDYT?

I guess I also worry a bit that ARM tests would take forever using non-accelerated virtualization of ARM on an x86 host. Is there some plan in place to get us some arm or (better) aarch64 hardware so we can have a worker host that does accelerated ARM virt?

  1. It ties scheduling pretty heavily to the openQA server box again, when I was trying to make them separable (I never got around to changing the scheduler to use the openQA ISO download function, but it's a trivial change).

I guess that when we'll want to use openQA to download images for us, we can also change this. Hardest thing was to figure out how to actually get ARM to run on QEMU, this code isn't that long to be missed when we will download things through openQA and delete all this :-).

Question is whether we want to do it now.

  1. I don't think anything will clean up the extracted images, and they're going to eat quite a bit of disk space, aren't they?

I was kinda hoping that we could set GRU to clean up things for us :-).

The required kernel and initramfs are actually available for download, though fedfind doesn't give those locations at present it fairly easily could. See for e.g. https://kojipkgs.fedoraproject.org/mash/rawhide-20160121/rawhide/armhfp/os/images/pxeboot/ . Downloading both the disk image and the kernel and initramfs files is a bit less efficient in terms of transfer, I guess, but should make the code a lot less complex. We'd still be tied to the scheduler, but it'd be at least potentially easier to unpick (I could implement something like HDDURL, KERNELURL and INITRDURL for openQA along the same lines of ISOURL, I intentionally left the design such that that sort of thing should be relatively easy to do). WDYT?

I agree, it would be much better. Kernel + initramfs aren't that big anyway.

I guess I also worry a bit that ARM tests would take forever using non-accelerated virtualization of ARM on an x86 host. Is there some plan in place to get us some arm or (better) aarch64 hardware so we can have a worker host that does accelerated ARM virt?

I would say that virtualizing ARM on x86 is ~5 times slower. I think that @lbrabec got some AArch64 machines in RH Beaker to test Fedora on, but I don't know whether we would be able to use them. openSUSE guys are using PPC64 machine to virtualize AArch64 on and they are using TIMEOUT_SCALE=4 for those tests (I think that it means that all waiting in tests is four times longer).

Since we don't have installation images (I still don't know whether we should have them - how we should test all those Installation tests on ARM?), only things we will test are from Base matrix and they will not take long (compared to many Fedora installations on x86). Together with functionality where we report test as soon as they are finished (instead of waiting till the end), I don't thing we will have any problems with it, at least from start.

  1. It ties scheduling pretty heavily to the openQA server box again, when I was trying to make them separable (I never got around to changing the scheduler to use the openQA ISO download function, but it's a trivial change).

I guess that when we'll want to use openQA to download images for us, we can also change this. Hardest thing was to figure out how to actually get ARM to run on QEMU, this code isn't that long to be missed when we will download things through openQA and delete all this :-).

Question is whether we want to do it now.

I'm fine with not actually switching to openQA downloads yet, I just wanted to consider whether we should get tied into a design which would make switching to them really difficult. i.e. if we just switch to downloading the kernel/initramfs instead of extracting them from the disk image, that's totally fine for me, because it's easy to switch over if I do bother to implement downloads of those things for openQA.

  1. I don't think anything will clean up the extracted images, and they're going to eat quite a bit of disk space, aren't they?

I was kinda hoping that we could set GRU to clean up things for us :-).

GRU is not really configurable in this way, the code that decides what files it will actually clean up is part of openQA itself. So to make GRU clean up after us here, we'd have to get a patch merged upstream. I think we'd have a reasonable shot at merging a patch that cleaned up these style of disk images (if they're not already covered), kernels and initramfs'es; I don't think, if we stick with the 'extract kernel/initramfs from the image' approach, we'd be able to sell upstream on cleaning up the directory where we do the extraction, and that could be a bit of a problem, as I don't see how we could entirely reliably clean it up once openQA was done (we *could* adjust the current code to wipe all the files except the kernel and initramfs it needs, though, which would mitigate the issue).

One thing I'd suggest we do is take a look and see if SUSE are doing any direct kernel boot tests, and if so where they put the kernel/initramfs files and if there's anything in place to clean them up.

The required kernel and initramfs are actually available for download, though fedfind doesn't give those locations at present it fairly easily could. See for e.g. https://kojipkgs.fedoraproject.org/mash/rawhide-20160121/rawhide/armhfp/os/images/pxeboot/ . Downloading both the disk image and the kernel and initramfs files is a bit less efficient in terms of transfer, I guess, but should make the code a lot less complex. .. WDYT?

I agree, it would be much better. Kernel + initramfs aren't that big anyway.

OK, I'll add a method or something to fedfind to help us find kernel/initramfs for an arch, shouldn't be too hard.

I guess I also worry a bit that ARM tests would take forever using non-accelerated virtualization of ARM on an x86 host. Is there some plan in place to get us some arm or (better) aarch64 hardware so we can have a worker host that does accelerated ARM virt?

I would say that virtualizing ARM on x86 is ~5 times slower. I think that @lbrabec got some AArch64 machines in RH Beaker to test Fedora on, but I don't know whether we would be able to use them. openSUSE guys are using PPC64 machine to virtualize AArch64 on and they are using TIMEOUT_SCALE=4 for those tests (I think that it means that all waiting in tests is four times longer).

Yeah, that's what it means indeed, they have a bit of code in their distribution class IIRC which lets them scale all the timeout values.

Since we don't have installation images (I still don't know whether we should have them - how we should test all those Installation tests on ARM?),

There's a stage2 image for ARM (install.img) in any install tree. You can boot the installer by doing a direct kernel boot and passing in that stage2, I think. pwhalen would have all the details I expect, since he does these tests manually.

only things we will test are from Base matrix and they will not take long (compared to many Fedora installations on x86). Together with functionality where we report test as soon as they are finished (instead of waiting till the end), I don't thing we will have any problems with it, at least from start.

Fair point, I guess, it would be useful to hit the base tests indeed.

adamwill added a comment.EditedFeb 2 2016, 11:51 AM

So a couple of notes here: I have the pungi4 rewrite in progress and working, but we are planning to have taskotron doing the scheduling, which means all scheduling stuff needs to work with openQA doing the image downloading. So we're gonna have to add openQA download support for disk images, I guess. That should be relatively simple, I can work on it or I can help someone else with it if you want to do some upstream familiarization.

Other note is probably useless if we have to do remote downloading, but just for information - virt-builder actually has a function for extracting kernel and initramfs from a disk image, virt-builder --get-kernel IMAGE. See http://libguestfs.org/virt-builder.1.html .

Some of the bits of the diff won't apply to the new scheduler at all, but the main meat of it - what params to pass to the POST request - should be the same.

Other note is probably useless if we have to do remote downloading, but just for information - virt-builder actually has a function for extracting kernel and initramfs from a disk image, virt-builder --get-kernel IMAGE. See http://libguestfs.org/virt-builder.1.html .

I'm already using it in this code :-).

Well shoot, I clearly didn't remember the code properly :)

Good news from openSUSE folks: they're totally fine with us extending the download stuff to cover disk images, kernels and initramfs'es (in fact we're gonna make it generic so you can add _URL to *any* expected setting name and it'll run through the 'download' logic for it). So we can do that :)

garretraziel updated this revision to Diff 1893.Feb 3 2016, 8:52 AM

rebasing to develop branch

Just to note - we are waiting for Pungi 4 and openQA download stuff and I will rebase it to it once it will be released.

Here is the PR for kernel / initrd / hdd downloading:

https://github.com/os-autoinst/openQA/pull/535

The generic asset download code is now merged upstream and is in openqa-4.3-17, which is in updates-testing at present. stg is running that openqa (prod is still on 4.3-12 for now). You can pass KERNEL_URL and INITRD_URL to download kernels and initrds. We'll probably also want to set KERNEL and INITRD to give them unique file names.

garretraziel abandoned this revision.Apr 1 2016, 12:38 PM