Bug 1102592

The heat openstack gate job uses the fedora could image in heat jobs.
The tests running on VMs without nested guest support, and booting up the f20 could image as L2 guest, so the virt_type is qemu, not kvm.

The Linux kernel in every other vitalization case would automatically disable the the timer check (including kvm), but it does not do it with qemu, because linux does not detects it as virtual system.

We saw the no_timer_check kernel parameter solves the issue with the cirros image. It should be included to the f20 cloud image.

openstack gate jobs failing approximately 22 times / per week because of this issue, so we wasting 100 server hour per weak because of this.  

Can you publish an x86_64 qcow2 f20 image which includes 'no_timer_check' as kernel parameter ?


Approximately half of the failing cases are visible here:

http://logstash.openstack.org/#eyJmaWVsZHMiOltdLCJzZWFyY2giOiJtZXNzYWdlOlwiTVAtQklPUyBidWdcIiBBTkQgdGFnczpcImNvbnNvbGVcIiBBTkQgYnVpbGRfbmFtZTpcImNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWhlYXQtc2xvd1wiIiwidGltZWZyYW1lIjoiYWxsIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJvZmZzZXQiOjAsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIiwic3RhbXAiOjE0MDMwNzk5MzA5MTN9

message:"MP-BIOS bug" AND tags:"console" AND build_name:"check-tempest-dsvm-neutron-heat-slow"

A related old QEMU bug which has some more context about no_timer_check

https://bugzilla.redhat.com/show_bug.cgi?id=502058#c15

I think you should always add no_timer_check to cloud images
(it is implied automatically when the image is booted on KVM, but
not on software QEMU).

Does this make sense when using Xen (and, for example, in EC2)? (We currently use a kludge to provide different boot parameters for ttys already...)

(In reply to Matthew Miller from comment #4)
> Does this make sense when using Xen (and, for example, in EC2)? (We
> currently use a kludge to provide different boot parameters for ttys
> already...)

This is a guest kernel option which tells the kernel not to try to
test for an obscure IO-APIC bug which almost certainly only affects
ancient baremetal.  The precise comment is (arch/x86/kernel/apic/io_apic.c):

/*
 * There is a nasty bug in some older SMP boards, their mptable lies
 * about the timer IRQ. We do the following to work around the situation:
 *
 *      - timer IRQ defaults to IO-APIC IRQ
 *      - if this function detects that timer IRQs are defunct, then we fall
 *        back to ISA timer IRQs
 */

This test fails under virtualization (like all timing loops) because
the VM might be rescheduled during the loop, completely breaking
the assumptions of the test.

(IIRC on older kernels, no_timer_check also disabled some other tests
for clock stability.  However it looks like that code has now finally
been removed.  Yay!  In any case what old kernels did doesn't apply to
Fedora.)

Looking at the kernel sources, only KVM and Hyper-V are currently
being detected and setting no_timer_check implicitly.  Not Xen AFAICT.

Therefore: I think you should add this flag unconditionally on all
cloud images.

Can someone tell me more about

"
Additional info:
The notsc and a proper lpj=n kernel parameter is also recommended for soft-emu,
usage.
"

?

(In reply to Matthew Miller from comment #6)
> Can someone tell me more about
> 
> "
> Additional info:
> The notsc and a proper lpj=n kernel parameter is also recommended for
> soft-emu,
> usage.
> "
> 
> ?

Don't know about notsc.

However lpj= is an ongoing problem.  Basically in order to avoid
the guest kernel attempting to calibrate loops_per_jiffy (in a timing
loop, hence all the problems that involves) it is possible to force
it to use a particular setting on the command line using lpj=NNNN

Two problems remain:

(1) You have to set this per host.  It should be set to the same
value as the host loops_per_jiffy.  It's not a setting that you
can or should include in a cloud image which is to be distributed
to everyone.

(2) In the case where you might be able to set lpj from the host,
you cannot always read the correct setting from the host.  Libguestfs
does this:

https://github.com/libguestfs/libguestfs/blob/master/src/lpj.c

I proposed a patch upstream which would export host loops_per_jiffy
in /proc/cpuinfo but it was naturally ignored:

https://lkml.org/lkml/2013/3/1/308

Note you cannot calculate host loops_per_jiffy from HZ.  It seems
like you should be able to, but you cannot.

I should say that we have never needed to use notsc in libguestfs,
even though we frequently use QEMU and encounter (and fix)
problems with it.

The notsc and the lpj was recommended for RHEL 6.0 hosts without HW visualization support.

With the f20 guest image I never seen very badly calibrated lpj, or any issue which could be solved nor workaround-ed by the notsc parameter. 

Linux uses a very large sample for lpj calculation, the host really needs to be overloaded to cause big mistake. 

The no_timer_check parameter alone is good for solving all seen gate issues (with 1 vcpu).

F20 images with no_timer_check:
http://dl.fedoraproject.org/pub/alt/openstack/20/x86_64/

Closing Per comment #10, and examining one of the F20 OpenStack cloud image:


---------------
$ guestfish --ro -i -a Fedora-x86_64-20-20140618-sda.qcow2 
[. . .]
><fs> cat /etc/grub.conf 
default=0
timeout=0


title Fedora (3.11.10-301.fc20.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-3.11.10-301.fc20.x86_64 ro root=UUID=314b4a27-3885-49e8-9415-af098db4fd2a no_timer_check console=hvc0 LANG=en_US.UTF-8
        initrd /boot/initramfs-3.11.10-301.fc20.x86_64.img

><fs> 
---------------

Also, for completeness' sake: Fedora 21 and above OpenStack cloud images also have the 'no_timer_check' parameter already present.