Bug 854448

Summary: add pmtimer support
Product: Red Hat Enterprise Linux 6 Reporter: Gerd Hoffmann <kraxel>
Component: seabiosAssignee: Gerd Hoffmann <kraxel>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.3CC: areis, bcao, flang, juzhang, knoel, lagarcia, qzhou, tburke, virt-maint, ypu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: seabios-0.6.1.2-23.el6 Doc Type: Bug Fix
Doc Text:
Cause: seabios tries to use the tsc for timekeeping, with a simple calibration loop. Consequence: on a busy host the calibration can be wrong by orders of magnitude, resulting in timeouts being way too short, leading to boot failures. Fix: use (fixed frequency) pmtimer for timekeeping. Result: timeouts are working correctly under all circumstances.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 08:16:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
upstream patch
none
log for results 1
none
seabios.log format txt
none
updated bios binary
none
seabios log none

Description Gerd Hoffmann 2012-09-05 05:36:14 UTC
Description of problem:
seabios uses the tsc for timekeeping (mostly timeouts).  tsc calibration in a virtual machine is fragile, the measured tsc frequency is *way* off at times.  

Better use a fixed-frequency time source such as the acpi pmtimer to avoid calibration issues.

How reproducible:
now and then.

Steps to Reproduce:
One visible effect is that booting from usb may fail due to (bogous) timeouts,
especially on a loaded host.

Comment 1 Gerd Hoffmann 2012-09-05 05:37:18 UTC
Created attachment 609860 [details]
upstream patch

Comment 4 langfang 2012-09-18 11:07:57 UTC
Created attachment 613974 [details]
log for results 1

Comment 5 langfang 2012-09-18 11:14:10 UTC
test this bug as follow version:
# uname -r
2.6.32-305.el6.x86_64
# rpm -qa |grep qemu-kvm
qemu-kvm-rhev-debuginfo-0.12.1.2-2.295.el6_3.1.x86_64
qemu-kvm-rhev-0.12.1.2-2.295.el6_3.1.x86_64
qemu-kvm-rhev-tools-0.12.1.2-2.295.el6_3.1.x86_64


# rpm -q seabios
seabios-0.6.1.2-22.el6.x86_64

1.boot guest 
/usr/libexec/qemu-kvm -m 2G -smp 2 -cpu Penryn -rtc base=localtime,clock=host,driftfix=slew -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,mac=04:53:19:29:28:33,bus=pci.0,addr=0x4,id=net0 -boot d -uuid ff820d25-034a-4547-9f67-ad057662a6c2 -rtc-td-hack -monitor stdio -name rhel6.4 -vnc :5 -device usb-ehci,id=ehci -drive file=/home/rhel6.4-copy1.qcow2,if=none,id=storage0,media=disk,cache=none,format=qcow2 -device usb-storage,drive=storage0 -chardev file,id=seabios,path=/home/seabios.log -device isa-debugcon,iobase=0x402,chardev=seabios

results 1)
How reproducible:
15%

(qemu) KVM internal error. Suberror: 1
rax 000000000000000e rbx 00000000c49fe8dc rcx 000000000000000e rdx 0000000000000000
rsi 000000000e00ffff rdi 00000000462a2c02 rsp 000000000000002b rbp 00000000c49f3d00
r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
rip 00000000000006f9 rflags 00000012
cs a700 (000a7000/0000ffff p 1 dpl 0 db 0 s 1 type b l 0 g 0 avl 0)
ds 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
es 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
ss 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
fs 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
gs 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
tr 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 0 type b l 0 g 0 avl 0)
ldt 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 0 type 2 l 0 g 0 avl 0)
gdt 9180/27
idt 0/3ff
cr0 10 cr2 0 cr3 0 cr4 0 cr8 0 efer 0
emulation failure, check dmesg for details
(qemu) info status
VM status: paused (internal-error)

addinfo :attachment is seabios.log

results 2)
How reproducible:
18%

sometimes,in the booting precess ,guest not paused,but guest show dark screen.

Comment 6 langfang 2012-09-18 11:18:27 UTC
Created attachment 613980 [details]
seabios.log      format txt

Comment 7 Gerd Hoffmann 2012-09-18 12:07:53 UTC
Created attachment 613991 [details]
updated bios binary

Oops, no bios update, I through I did that already.

The backported patch hasn't been merged into the rhel6 package yet.
Attached is a bios binary with the patch applied for testing.  Pass
the binary to qemu.kvm using the -bios switch.

With the updated bios the 'black screen' failures should go away.

Comment 10 langfang 2012-09-28 01:09:55 UTC
test this bug on version:
seabios-0.6.1.2-23.el6.x86_64

# uname -r
2.6.32-307.el6.x86_64
#rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.314.el6.x86_64

the steps as same as comment5,test more than 10 times,not hit any problem.guest work well.

Comment 11 Gerd Hoffmann 2012-10-19 09:11:06 UTC
*** Bug 867214 has been marked as a duplicate of this bug. ***

Comment 12 Joy Pu 2012-11-06 08:49:58 UTC
Hit one very similar problem during the test. It is not very easy to trigger in normal tests. But after system_reset it is more easier to trigger. The guest is a rhel5.9 32 bit guest with rtl8139 nic and ide disk.

Error messgae:

 KVM internal error. Suberror: 1
 rax 0000000000000054 rbx 0000000008206510 rcx 0000000000020000 rdx 0000000000a89744
 rsi 0000000000a879b1 rdi 0000000000000054 rsp 00000000bf9c2a0c rbp 00000000bf9c2a18
 r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
 r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
 rip 00000000000b23e0 rflags 00013202
 cs 0073 (00000000/08204fff p 1 dpl 3 db 1 s 1 type b l 0 g 1 avl 0)
 ds 007b (00000000/ffffffff p 1 dpl 3 db 1 s 1 type 3 l 0 g 1 avl 0)
 es 007b (00000000/ffffffff p 1 dpl 3 db 1 s 1 type 3 l 0 g 1 avl 0)
 ss 007b (00000000/ffffffff p 1 dpl 3 db 1 s 1 type 3 l 0 g 1 avl 0)
 fs 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
 gs 0033 (b7fe06d0/ffffffff p 1 dpl 3 db 1 s 1 type 3 l 0 g 1 avl 1)
 tr 0080 (c300f400/00002073 p 1 dpl 0 db 0 s 0 type b l 0 g 0 avl 0)
 ldt 0088 (c0755020/00000027 p 1 dpl 0 db 0 s 0 type 2 l 0 g 0 avl 0)
 gdt c301e000/ff
 idt c0704000/7ff
 cr0 8005003b cr2 a86990 cr3 36641000 cr4 6d0 cr8 0 efer 0
 emulation failure, check dmesg for details

version:
# uname -r
2.6.32-337.el6.x86_64
# rpm -qa |grep seabios
seabios-0.6.1.2-25.el6.x86_64
# rpm -qa |grep qemu
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
qemu-img-0.12.1.2-2.333.el6.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.333.el6.x86_64
qemu-kvm-0.12.1.2-2.333.el6.x86_64
qemu-kvm-tools-0.12.1.2-2.333.el6.x86_64
qemu-guest-agent-0.12.1.2-2.333.el6.x86_64

Comment 13 Qingtang Zhou 2012-11-14 06:35:29 UTC
Created attachment 644589 [details]
seabios log

Hi, there, I met this problem too this week, when running a 32-bit RHEL5.9 guest.

KVM internal error. Suberror: 1
rax 000000000000000b rbx 0000000008206510 rcx 0000000000000000 rdx 0000000000000000
rsi 00000000bf86690c rdi 0000000000000054 rsp 00000000bf8668ec rbp 00000000bf866998
r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
rip 00000000000b3d00 rflags 00013246
cs 0073 (00000000/08204fff p 1 dpl 3 db 1 s 1 type b l 0 g 1 avl 0)
ds 007b (00000000/ffffffff p 1 dpl 3 db 1 s 1 type 3 l 0 g 1 avl 0)
es 007b (00000000/ffffffff p 1 dpl 3 db 1 s 1 type 3 l 0 g 1 avl 0)
ss 007b (00000000/ffffffff p 1 dpl 3 db 1 s 1 type 3 l 0 g 1 avl 0)
fs 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
gs 0033 (b7f876d0/ffffffff p 1 dpl 3 db 1 s 1 type 3 l 0 g 1 avl 1)
tr 0080 (c2010244/00002073 p 1 dpl 0 db 0 s 0 type b l 0 g 0 avl 0)
ldt 0088 (c0755020/00000027 p 1 dpl 0 db 0 s 0 type 2 l 0 g 0 avl 0)
gdt f7c9f000/ff
idt c0704000/7ff
cr0 8005003b cr2 54 cr3 3539c000 cr4 6d0 cr8 0 efer 0
emulation failure, check dmesg for details

attach seabios.log file.

Related packages list:
$ /bin/grep -E "^kernel|seabios|qemu" installed_packages.txt 
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
kernel-2.6.32-330.el6.x86_64
kernel-2.6.32-338.el6.x86_64
kernel-devel-2.6.32-330.el6.x86_64
kernel-firmware-2.6.32-338.el6.noarch
kernel-headers-2.6.32-330.el6.x86_64
qemu-guest-agent-0.12.1.2-2.334.el6.x86_64
qemu-img-0.12.1.2-2.334.el6.x86_64
qemu-kvm-0.12.1.2-2.334.el6.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.334.el6.x86_64
qemu-kvm-tools-0.12.1.2-2.334.el6.x86_64
seabios-0.6.1.2-25.el6.x86_64

Comment 14 Qingtang Zhou 2012-11-14 06:39:46 UTC
Attach qemu-kvm cli which hit this problem.

/usr/local/staf/test/RHEV/kvm/autotest-devel/client/tests/kvm/qemu 
    -S  
    -name 'vm1' 
    -nodefaults 
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20121112-144810-yePKecol,server,nowait 
    -mon chardev=qmp_id_qmpmonitor1,mode=control 
    -chardev socket,id=serial_id_serial1,path=/tmp/serial-serial1-20121112-144810-yePKecol,server,nowait 
    -device isa-serial,chardev=serial_id_serial1 
    -chardev socket,id=seabioslog_id_20121112-144810-yePKecol,path=/tmp/seabios-20121112-144810-yePKecol,server,nowait 
    -device isa-debugcon,chardev=seabioslog_id_20121112-144810-yePKecol,iobase=0x402 
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 
    -drive file='RHEL-Server-5.9-32-virtio.qcow2',if=none,id=drive-virtio-disk1,media=disk,cache=none,boot=off,snapshot=off,format=qcow2,aio=native 
    -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 
    -device virtio-net-pci,netdev=idaFPemd,mac=9a:49:4a:4b:4c:4d,bus=pci.0,addr=0x3,id='idY4fhnG' 
    -netdev tap,id=idaFPemd,vhost=on,fd=27 
    -m 2048 
    -smp 2,cores=1,threads=1,sockets=2 
    -cpu 'Penryn' 
    -M rhel6.4.0 
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 
    -vnc :0  
    -vga cirrus 
    -rtc base=utc,clock=host,driftfix=slew  
    -boot order=cdn,once=c,menu=off   
    -no-kvm-pit-reinjection 
    -enable-kvm

Comment 16 langfang 2012-12-20 11:05:43 UTC
Reproduce this bug as follow version:
# uname -r
2.6.32-348.el6.x86_64
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.346.el6.x86_64
# rpm -q seabios
seabios-0.6.1.2-26.el6.x86_64

Steps :
1.Install guest and repeadly reboot guest
 /usr/libexec/qemu-kvm -m 2G -smp 2 -cpu Penryn -rtc base=localtime,clock=host,driftfix=slew -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,mac=04:53:19:29:28:33,bus=pci.0,addr=0x4,id=net0 -boot c -uuid ff820d25-034a-4547-9f67-ad057662a6c2 -rtc-td-hack -monitor stdio -name rhel6.4 -vnc :5 -device usb-ehci,id=ehci -drive file=/home/rhel5.9-test.qcow2,if=none,id=storage0,media=disk,cache=none,format=qcow2 -device usb-storage,drive=storage0 -chardev file,id=seabios,path=/home/seabios.log -device isa-debugcon,iobase=0x402,chardev=seabios -drive file=/home/RHEL5.9-Server-20121114.2-i386-DVD.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -monitor unix:/tmp/monitor-unix,nowait,server

2.Repeatly reboot guest

# cat reboot-loop.sh 
#!/bin/bash
while true
do
echo "system_reset" | nc -U /tmp/monitor-unix
sleep 80
done

Results:
(qemu) KVM internal error. Suberror: 1
rax 00000000000a0000 rbx 000000000003fef8 rcx 0000000000006000 rdx 00000000000000b3
rsi 0000000000005d30 rdi 0000000000007010 rsp 0000000000001fe8 rbp 0000000000001ff0
r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
rip 0000000000004666 rflags 00010002
cs aded (000aded0/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
ds 6000 (00060000/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
es 0000 (00000000/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
ss 0000 (00000000/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
fs 0000 (00000000/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
gs 0000 (00000000/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
tr 0000 (feffd000/00002088 p 1 dpl 0 db 0 s 0 type b l 0 g 0 avl 0)
ldt 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 0 type 2 l 0 g 0 avl 0)
gdt 9040/27
idt 0/3ff
cr0 10 cr2 0 cr3 0 cr4 0 cr8 0 efer 0
emulation failure, check dmesg for details

(qemu) info status
VM status: paused


Verify this bug as follow version:
# uname -r
2.6.32-279.9.1.el6.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.295.el6_3.2.x86_64
# rpm -q seabios
seabios-0.6.1.2-19.el6.x86_64


The steps as same as reproduce

Results:
Tried about 1000 times.Guest boot up successfully,not hit any problem.

Addinfo: Very difficult to reproduce,need use  repeadly reboot guest.

According to above test,this bug have fixed.

Comment 17 langfang 2012-12-20 11:21:35 UTC
Please ignore comment16

Reproduce this bug as follow version:
# uname -r
2.6.32-279.9.1.el6.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.295.el6_3.2.x86_64
# rpm -q seabios
seabios-0.6.1.2-19.el6.x86_64

Steps :
1.Install guest and repeadly reboot guest
 /usr/libexec/qemu-kvm -m 2G -smp 2 -cpu Penryn -rtc base=localtime,clock=host,driftfix=slew -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,mac=04:53:19:29:28:33,bus=pci.0,addr=0x4,id=net0 -boot c -uuid ff820d25-034a-4547-9f67-ad057662a6c2 -rtc-td-hack -monitor stdio -name rhel5.9 -vnc :5 -device usb-ehci,id=ehci -drive file=/home/rhel5.9-test.qcow2,if=none,id=storage0,media=disk,cache=none,format=qcow2 -device usb-storage,drive=storage0 -chardev file,id=seabios,path=/home/seabios.log -device isa-debugcon,iobase=0x402,chardev=seabios -drive file=/home/RHEL5.9-Server-20121114.2-i386-DVD.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -monitor unix:/tmp/monitor-unix,nowait,server

2.Repeatly reboot guest

# cat reboot-loop.sh 
#!/bin/bash
while true
do
echo "system_reset" | nc -U /tmp/monitor-unix
sleep 80
done

Results:
(qemu) KVM internal error. Suberror: 1
rax 00000000000a0000 rbx 000000000003fef8 rcx 0000000000006000 rdx 00000000000000b3
rsi 0000000000005d30 rdi 0000000000007010 rsp 0000000000001fe8 rbp 0000000000001ff0
r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
rip 0000000000004666 rflags 00010002
cs aded (000aded0/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
ds 6000 (00060000/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
es 0000 (00000000/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
ss 0000 (00000000/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
fs 0000 (00000000/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
gs 0000 (00000000/0000ffff p 1 dpl 3 db 0 s 1 type 3 l 0 g 0 avl 0)
tr 0000 (feffd000/00002088 p 1 dpl 0 db 0 s 0 type b l 0 g 0 avl 0)
ldt 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 0 type 2 l 0 g 0 avl 0)
gdt 9040/27
idt 0/3ff
cr0 10 cr2 0 cr3 0 cr4 0 cr8 0 efer 0
emulation failure, check dmesg for details

(qemu) info status
VM status: paused

Verify this bug as follow version:

# uname -r
2.6.32-348.el6.x86_64
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.346.el6.x86_64
# rpm -q seabios
seabios-0.6.1.2-26.el6.x86_64


The steps as same as reproduce

Results:
Tried about 1000 times.Guest boot up successfully,not hit any problem.


Addinfo: Difficult to reproduce,need use repeadly reboot guest.

According to above test,this bug have fixed.

Addtional info:
1)Also tried repeadly install guest,not hit any problem.
2)As comment12,also test use NIC rtl8139,ide disk,rhel5.9-32 guest,not hit any problem
3)Host info:
[root@intel-q9400-8-2 ~]# cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Core(TM)2 Quad CPU    Q9400  @ 2.66GHz
stepping	: 10
cpu MHz		: 1998.000
cache size	: 3072 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm dts tpr_shadow vnmi flexpriority
bogomips	: 5333.33
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

Comment 20 errata-xmlrpc 2013-02-21 08:16:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0307.html