Bug 1102592 - F20 could image affected by timing issues when using with soft-qemu
Summary: F20 could image affected by timing issues when using with soft-qemu
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: distribution
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Václav Pavlín
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-05-29 09:25 UTC by Attila Fazekas
Modified: 2015-02-05 16:44 UTC (History)
6 users (show)

Fixed In Version: Fedora-x86_64-20-20140618
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-02-05 16:43:31 UTC


Attachments (Terms of Use)

Description Attila Fazekas 2014-05-29 09:25:00 UTC
Description of problem:

The Linux kernel does not disables automatically the timer check, when the image booted in soft-qemu (TCG, no kvm acceleration, no paravirtulaized clock).

The timer check can fail when the qemu process is preempted by host system,
what leads to the following fatal error:

http://logs.openstack.org/10/82410/4/check/check-tempest-dsvm-neutron-heat-slow/094a340/console.html.gz#_2014-03-25_14_28_40_090

[    0.202000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
[    0.202000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.202000] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[    0.202000] ...trying to set up timer (IRQ0) through the 8259A ...
[    0.202000] ..... (found apic 0 pin 2) ...
[    0.202000] ....... failed.
[    0.202000] ...trying to set up timer as Virtual Wire IRQ...
Version-Release number of selected component (if applicable):
Fedora-x86_64-20-20131211.1-sda ..

The timer check must be disabled in all virtualized/emulated environment.
AFAIK it is safe to disable even in a not virtulized env.

Expected results:
no_timer_check is part of the default kernel parameters.

Additional info:
The notsc and a proper lpj=n kernel parameter is also recommended for soft-emu,
usage.

The cloud image uses the kernel parameters from this file:
/boot/extlinux/extlinux.conf

Comment 1 Attila Fazekas 2014-06-18 09:10:37 UTC
The heat openstack gate job uses the fedora could image in heat jobs.
The tests running on VMs without nested guest support, and booting up the f20 could image as L2 guest, so the virt_type is qemu, not kvm.

The Linux kernel in every other vitalization case would automatically disable the the timer check (including kvm), but it does not do it with qemu, because linux does not detects it as virtual system.

We saw the no_timer_check kernel parameter solves the issue with the cirros image. It should be included to the f20 cloud image.

openstack gate jobs failing approximately 22 times / per week because of this issue, so we wasting 100 server hour per weak because of this.  

Can you publish an x86_64 qcow2 f20 image which includes 'no_timer_check' as kernel parameter ?


Approximately half of the failing cases are visible here:

http://logstash.openstack.org/#eyJmaWVsZHMiOltdLCJzZWFyY2giOiJtZXNzYWdlOlwiTVAtQklPUyBidWdcIiBBTkQgdGFnczpcImNvbnNvbGVcIiBBTkQgYnVpbGRfbmFtZTpcImNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWhlYXQtc2xvd1wiIiwidGltZWZyYW1lIjoiYWxsIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJvZmZzZXQiOjAsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIiwic3RhbXAiOjE0MDMwNzk5MzA5MTN9

message:"MP-BIOS bug" AND tags:"console" AND build_name:"check-tempest-dsvm-neutron-heat-slow"

Comment 2 Kashyap Chamarthy 2014-06-18 10:51:32 UTC
A related old QEMU bug which has some more context about no_timer_check

https://bugzilla.redhat.com/show_bug.cgi?id=502058#c15

Comment 3 Richard W.M. Jones 2014-06-18 11:19:30 UTC
I think you should always add no_timer_check to cloud images
(it is implied automatically when the image is booted on KVM, but
not on software QEMU).

Comment 4 Matthew Miller 2014-06-18 14:10:34 UTC
Does this make sense when using Xen (and, for example, in EC2)? (We currently use a kludge to provide different boot parameters for ttys already...)

Comment 5 Richard W.M. Jones 2014-06-18 14:23:06 UTC
(In reply to Matthew Miller from comment #4)
> Does this make sense when using Xen (and, for example, in EC2)? (We
> currently use a kludge to provide different boot parameters for ttys
> already...)

This is a guest kernel option which tells the kernel not to try to
test for an obscure IO-APIC bug which almost certainly only affects
ancient baremetal.  The precise comment is (arch/x86/kernel/apic/io_apic.c):

/*
 * There is a nasty bug in some older SMP boards, their mptable lies
 * about the timer IRQ. We do the following to work around the situation:
 *
 *      - timer IRQ defaults to IO-APIC IRQ
 *      - if this function detects that timer IRQs are defunct, then we fall
 *        back to ISA timer IRQs
 */

This test fails under virtualization (like all timing loops) because
the VM might be rescheduled during the loop, completely breaking
the assumptions of the test.

(IIRC on older kernels, no_timer_check also disabled some other tests
for clock stability.  However it looks like that code has now finally
been removed.  Yay!  In any case what old kernels did doesn't apply to
Fedora.)

Looking at the kernel sources, only KVM and Hyper-V are currently
being detected and setting no_timer_check implicitly.  Not Xen AFAICT.

Therefore: I think you should add this flag unconditionally on all
cloud images.

Comment 6 Matthew Miller 2014-06-18 14:53:55 UTC
Can someone tell me more about

"
Additional info:
The notsc and a proper lpj=n kernel parameter is also recommended for soft-emu,
usage.
"

?

Comment 7 Richard W.M. Jones 2014-06-18 14:59:41 UTC
(In reply to Matthew Miller from comment #6)
> Can someone tell me more about
> 
> "
> Additional info:
> The notsc and a proper lpj=n kernel parameter is also recommended for
> soft-emu,
> usage.
> "
> 
> ?

Don't know about notsc.

However lpj= is an ongoing problem.  Basically in order to avoid
the guest kernel attempting to calibrate loops_per_jiffy (in a timing
loop, hence all the problems that involves) it is possible to force
it to use a particular setting on the command line using lpj=NNNN

Two problems remain:

(1) You have to set this per host.  It should be set to the same
value as the host loops_per_jiffy.  It's not a setting that you
can or should include in a cloud image which is to be distributed
to everyone.

(2) In the case where you might be able to set lpj from the host,
you cannot always read the correct setting from the host.  Libguestfs
does this:

https://github.com/libguestfs/libguestfs/blob/master/src/lpj.c

I proposed a patch upstream which would export host loops_per_jiffy
in /proc/cpuinfo but it was naturally ignored:

https://lkml.org/lkml/2013/3/1/308

Note you cannot calculate host loops_per_jiffy from HZ.  It seems
like you should be able to, but you cannot.

Comment 8 Richard W.M. Jones 2014-06-18 15:04:16 UTC
I should say that we have never needed to use notsc in libguestfs,
even though we frequently use QEMU and encounter (and fix)
problems with it.

Comment 9 Attila Fazekas 2014-06-19 08:16:00 UTC
The notsc and the lpj was recommended for RHEL 6.0 hosts without HW visualization support.

With the f20 guest image I never seen very badly calibrated lpj, or any issue which could be solved nor workaround-ed by the notsc parameter. 

Linux uses a very large sample for lpj calculation, the host really needs to be overloaded to cause big mistake. 

The no_timer_check parameter alone is good for solving all seen gate issues (with 1 vcpu).

Comment 10 Attila Fazekas 2014-10-13 10:36:06 UTC
F20 images with no_timer_check:
http://dl.fedoraproject.org/pub/alt/openstack/20/x86_64/

Comment 11 Kashyap Chamarthy 2015-02-05 16:43:31 UTC
Closing Per comment #10, and examining one of the F20 OpenStack cloud image:


---------------
$ guestfish --ro -i -a Fedora-x86_64-20-20140618-sda.qcow2 
[. . .]
><fs> cat /etc/grub.conf 
default=0
timeout=0


title Fedora (3.11.10-301.fc20.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-3.11.10-301.fc20.x86_64 ro root=UUID=314b4a27-3885-49e8-9415-af098db4fd2a no_timer_check console=hvc0 LANG=en_US.UTF-8
        initrd /boot/initramfs-3.11.10-301.fc20.x86_64.img

><fs> 
---------------

Also, for completeness' sake: Fedora 21 and above OpenStack cloud images also have the 'no_timer_check' parameter already present.


Note You need to log in before you can comment on or make changes to this bug.