Bug 700553

Summary: F15 kvm performance problem (some virt-manager defaults may lead to suboptimal guest creation)
Product: [Fedora] Fedora Reporter: Reartes Guillermo <rtguille>
Component: qemuAssignee: Fedora Virtualization Maintainers <virt-maint>
Status: CLOSED DEFERRED QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 15CC: amit.shah, berrange, clalance, crobinso, dwmw2, ehabkost, extras-orphan, itamar, jaswinder, jforbes, knoel, notting, ondrejj, quintela, rhbugzilla, scottt.tw, tburke, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-05-28 19:21:38 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Attachments:
Description Flags
Messages from recent kvm issues none

Description Reartes Guillermo 2011-04-28 12:21:10 EDT
Description of problem:

I created 2 VMs, one F15 alpha and another F15 beta.
Both did boot ok, but the F15 beta VM froze after executing the installer.
The F15 alpha lasted a bit more, just to the keyboard menu, but it alzo froze.
The images are KDE live cd's (X86_64)

Version-Release number of selected component (if applicable):

qemu-kvm.x86_64                   2:0.14.0-7.fc15                   @fedora
kernel                            2.6.38.3-18.fc15.x86_64
libvirt.x86_64                        0.8.8-4.fc15                  @fedora
libvirt-client.x86_64                 0.8.8-4.fc15                  @fedora
libvirt-python.x86_64                 0.8.8-4.fc15                  @fedora

Steps to Reproduce:
above.
  
Actual results:
Apr 28 13:05:03 ulquiorra pulseaudio[2059]: ratelimit.c: 6 events suppressed
Apr 28 13:05:08 ulquiorra libvirtd: 13:05:08.437: 1483: info : libvirt version: 0.8.8, package: 4.fc15 (Fedora Project, 2011-04-05-19:58:51, x86-01.phx2.fedoraproject.org)
Apr 28 13:05:08 ulquiorra libvirtd: 13:05:08.437: 1483: warning : qemuSetupCgroup:319 : Memory cgroup is disabled in qemu configuration file: F15TST
Apr 28 13:05:08 ulquiorra kernel: [ 5622.797432] tun: Universal TUN/TAP device driver, 1.6
Apr 28 13:05:08 ulquiorra kernel: [ 5622.797440] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Apr 28 13:05:08 ulquiorra kernel: [ 5622.810043] device vnet0 entered promiscuous mode
Apr 28 13:05:08 ulquiorra kernel: [ 5622.810958] virbr0: topology change detected, propagating
Apr 28 13:05:08 ulquiorra kernel: [ 5622.810962] virbr0: port 1(vnet0) entering forwarding state
Apr 28 13:05:08 ulquiorra kernel: [ 5622.810964] virbr0: port 1(vnet0) entering forwarding state
Apr 28 13:05:08 ulquiorra lldpad[1400]: evb_ifdown:port vnet0 remove failed
Apr 28 13:05:08 ulquiorra lldpad[1400]: vdp_ifdown:vnet0 vdp data remove failed
Apr 28 13:05:08 ulquiorra lldpad[1400]: evb_ifdown:port vnet0 remove failed
Apr 28 13:05:08 ulquiorra lldpad[1400]: vdp_ifdown:vnet0 vdp data remove failed
Apr 28 13:05:08 ulquiorra lldpad[1400]: evb_ifdown:port vnet0 remove failed
Apr 28 13:05:08 ulquiorra lldpad[1400]: vdp_ifdown:vnet0 vdp data remove failed
Apr 28 13:05:08 ulquiorra dbus: [system] Activating service name='org.fedoraproject.Setroubleshootd' argv0='/lib64/dbus-1/dbus-daemon-launch-helper'
Apr 28 13:05:08 ulquiorra qemu-kvm: Could not find keytab file: /etc/qemu/krb5.tab: No such file or directory
Apr 28 13:05:09 ulquiorra dbus: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
Apr 28 13:05:11 ulquiorra setroubleshoot: SELinux is preventing /usr/bin/qemu-kvm from open access on the file /data0/software/iso/Fedora/15/Fedora-15-Beta-x86_64-Live-KDE.iso. For complete SELinux messages. run sealert -l 307596f3-2869-4593-bff8-6c90043323aa
Apr 28 13:05:26 ulquiorra kernel: [ 5640.382126] kvm: 4059: cpu0 unhandled rdmsr: 0xc0010001
Apr 28 13:05:48 ulquiorra dnsmasq-dhcp[1564]: DHCPDISCOVER(virbr0) 52:54:00:78:21:74 
Apr 28 13:05:48 ulquiorra dnsmasq-dhcp[1564]: DHCPOFFER(virbr0) 192.168.122.91 52:54:00:78:21:74 
Apr 28 13:05:48 ulquiorra dnsmasq-dhcp[1564]: DHCPREQUEST(virbr0) 192.168.122.91 52:54:00:78:21:74 
Apr 28 13:05:48 ulquiorra dnsmasq-dhcp[1564]: DHCPACK(virbr0) 192.168.122.91 52:54:00:78:21:74 
Apr 28 13:05:48 ulquiorra libvirtd: 13:05:48.616: 1484: warning : qemuSetupCgroup:319 : Memory cgroup is disabled in qemu configuration file: F15TSTa
Apr 28 13:05:48 ulquiorra kernel: [ 5662.955857] device vnet1 entered promiscuous mode
Apr 28 13:05:48 ulquiorra lldpad[1400]: evb_ifdown:port vnet1 remove failed
Apr 28 13:05:48 ulquiorra lldpad[1400]: vdp_ifdown:vnet1 vdp data remove failed
Apr 28 13:05:48 ulquiorra lldpad[1400]: evb_ifdown:port vnet1 remove failed
Apr 28 13:05:48 ulquiorra lldpad[1400]: vdp_ifdown:vnet1 vdp data remove failed
Apr 28 13:05:48 ulquiorra kernel: [ 5662.957735] virbr0: topology change detected, propagating
Apr 28 13:05:48 ulquiorra kernel: [ 5662.957739] virbr0: port 2(vnet1) entering forwarding state
Apr 28 13:05:48 ulquiorra kernel: [ 5662.957741] virbr0: port 2(vnet1) entering forwarding state
Apr 28 13:05:48 ulquiorra lldpad[1400]: evb_ifdown:port vnet1 remove failed
Apr 28 13:05:48 ulquiorra lldpad[1400]: vdp_ifdown:vnet1 vdp data remove failed
Apr 28 13:05:48 ulquiorra dbus: [system] Activating service name='org.fedoraproject.Setroubleshootd' argv0='/lib64/dbus-1/dbus-daemon-launch-helper'
Apr 28 13:05:48 ulquiorra qemu-kvm: Could not find keytab file: /etc/qemu/krb5.tab: No such file or directory
Apr 28 13:05:49 ulquiorra dbus: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
Apr 28 13:05:51 ulquiorra setroubleshoot: SELinux is preventing /usr/bin/qemu-kvm from open access on the file /data0/software/iso/Fedora/15/Fedora-15-Alpha-x86_64-Live-KDE.iso. For complete SELinux messages. run sealert -l bf6d2cf8-f933-488e-bf0e-bc1b1649bd0a
Apr 28 13:06:07 ulquiorra kernel: [ 5681.421272] kvm: 4198: cpu0 unhandled rdmsr: 0xc0010001
Apr 28 13:06:38 ulquiorra dnsmasq-dhcp[1564]: DHCPDISCOVER(virbr0) 52:54:00:ad:6f:09 
Apr 28 13:06:38 ulquiorra dnsmasq-dhcp[1564]: DHCPOFFER(virbr0) 192.168.122.150 52:54:00:ad:6f:09 
Apr 28 13:06:38 ulquiorra dnsmasq-dhcp[1564]: DHCPREQUEST(virbr0) 192.168.122.150 52:54:00:ad:6f:09 
Apr 28 13:06:38 ulquiorra dnsmasq-dhcp[1564]: DHCPACK(virbr0) 192.168.122.150 52:54:00:ad:6f:09 
Apr 28 13:06:40 ulquiorra dnsmasq-dhcp[1564]: DHCPREQUEST(virbr0) 192.168.122.150 52:54:00:ad:6f:09 
Apr 28 13:06:40 ulquiorra dnsmasq-dhcp[1564]: DHCPACK(virbr0) 192.168.122.150 52:54:00:ad:6f:09 
Apr 28 13:10:55 ulquiorra dbus: [system] Activating service name='net.reactivated.Fprint' argv0='/lib64/dbus-1/dbus-daemon-launch-helper'
Apr 28 13:10:55 ulquiorra dbus: [system] Successfully activated service 'net.reactivated.Fprint'
Apr 28 13:15:09 ulquiorra pulseaudio[2059]: ratelimit.c: 4 events suppressed

Expected results:
kvm

Additional info:
selinux is permissive.
Comment 1 Reartes Guillermo 2011-04-28 12:27:21 EDT
I powered the vm off and tried with F15 beta first. There is a 6 second lag between click and the action (in this case took 6 seconds to select spanish keyboard).
I says that the hypervisor is kvm... and the device is there.
Comment 2 Reartes Guillermo 2011-04-28 13:36:44 EDT
It is really slow, tried removing nic, tablet, serial. Changed the disk to ide. Change procesor to phenom ii, and even the vga to vmvga but it is very slow and it ates a lot of cpu.
Comment 3 Reartes Guillermo 2011-04-29 11:52:22 EDT
Updated the bug-report description to match the issue more accuratelly.

I have found a temporal workaround, i must create the VMs with the type generic linux 2.6.X and DO NOT use the default Linux Fedora 15 as a guest option.

I created 2 new VMs, one is Sackware64 and the other is a F15beta KDE x86_64 with type Generic Linux 2.6.X

IDLE: 18~30% during simultaneous install of both.

F15beta response times looked normal, no slowdown.

As soon as i launched a newly created default vm for F15 (with type linux, fedora 15) qemu-kvm started to eat cpu and the F15 box (not the newly created) started to malfunction, it did not freeze but no pointer and sometimes closing the window and opening again from
virt-manager showed changes in the screen.

I powered off the offendig F15 vm and qemu-kvm cpu usage returned to normal but the affected F15 did not recover and a vm poweroff was needed because in the end it froze.
The slackware vm was unaffected during this incident, continued installing itself with no problem.

There is some problem when using the guest type linux fedora 15
Comment 4 Reartes Guillermo 2011-05-07 10:11:09 EDT
It really looks like a performance problem.

Having one xp guest at partitioning step in setup (idle) consumes more that 100% as show with top, mpstat -P ALL show also high host cpu usage. 

Having two xp guests at partitioning step in setup (idle) consumes more that 100% cpu each qemu-kvm instance. mpstat -P ALL confirms very high host cpu usage.

A slackware 13.37 guest uses 60% host cpu when using kde.

The previously discused f15 guests uses 100% at plymouth screen ansd 80% after logout when loading kde.

In F15 host, X also uses 15% / 25% host cpu sometimes.

gpxe-roms-qemu.noarch              1.0.1-4.fc15                @fedora
qemu-common.x86_64                 2:0.14.0-7.fc15             @fedora
qemu-img.x86_64                    2:0.14.0-7.fc15             @fedora
qemu-kvm.x86_64                    2:0.14.0-7.fc15             @fedora
qemu-system-x86.x86_64             2:0.14.0-7.fc15             @fedora

libvirt.x86_64                      0.8.8-4.fc15               @fedora
libvirt-client.x86_64               0.8.8-4.fc15               @fedora
libvirt-python.x86_64               0.8.8-4.fc15               @fedora

virt-manager.noarch           0.8.7-4.fc15            @updates-testing

python-virtinst.noarch     0.500.6-2.fc15 @fedora 
spice-gtk-python.x86_64    0.5-6.fc15     @fedora

kernel 2.6.38.5-22.fc15.x86_64

---

Additional Info:
I started to have similar problems recently in F13, xp guests uses 60% host cpu at desktop only, and are really slow nowadays. Last update and cannot downgrade F13 due to a dependency error.
In F13, python also uses a lot of cpu, in sync with qemu-kvm process.
Comment 5 Reartes Guillermo 2011-05-20 15:49:22 EDT
Stopped lldpad, which was constanly stealing between 3 to 5 % cpu.

The issue is still there, sometimes i wish i could have a zetaflop procesor...

The below irq/sec reached to more than 4000 sometimes.
I started 2 F15 instances and 2 slackware instances.

 PerfTop:     440 irqs/sec  kernel:75.5%  exact:  0.0% [1000Hz cycles],  (all, 3 CPUs)
-----------------------------------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                                       DSO
             _______ _____ ______________________________________________ __________________________

             3544.00 20.7% svm_vcpu_run                                   [kvm_amd]                 
             2769.00 16.2% native_safe_halt                               [kernel.kallsyms]         
             2303.00 13.4% longest_match                                  /lib64/libz.so.1.2.5      
              867.00  5.1% _wordcopy_fwd_dest_aligned                     /lib64/libc-2.13.90.so    
              818.00  4.8% radeon_connector_analog_encoder_conflict_solve [radeon]                  
              688.00  4.0% g_atomic_pointer_get                           libglib-2.0.so.0.2800.6   
              243.00  1.4% __GI_memmove                                   /lib64/libc-2.13.90.so    
              190.00  1.1% avc_has_perm_noaudit                           [kernel.kallsyms]         
              185.00  1.1% find_vma                                       [kernel.kallsyms]         
              140.00  0.8% g_type_interface_peek                          libgobject-2.0.so.0.2800.6
              134.00  0.8% g_type_check_instance_cast                     libgobject-2.0.so.0.2800.6
              116.00  0.7% _int_malloc                                    /lib64/libc-2.13.90.so    
              110.00  0.6% deflate_slow                                   /lib64/libz.so.1.2.5      
               99.00  0.6% unix_poll                                      [kernel.kallsyms]         
               98.00  0.6% copy_user_generic_string                       [kernel.kallsyms]         
               96.00  0.6% fill_window                                    /lib64/libz.so.1.2.5      
               91.00  0.5% pthread_mutex_lock                             libpthread-2.13.90.so     
               89.00  0.5% g_atomic_int_get                               libglib-2.0.so.0.2800.6   
               88.00  0.5% _int_free                                      /lib64/libc-2.13.90.so    
               86.00  0.5% drm_mm_init_scan                               [drm]                     
               85.00  0.5% radeon_fence_process                           [radeon]                  
               84.00  0.5% __memset_sse2                                  /lib64/libc-2.13.90.so    
               66.00  0.4% do_select                                      [kernel.kallsyms]         

--------------

KVM in this moment is nearly useless on this cpu/motherboard. This problem did not exist in alpha (in wich i did not notice anything like this). I literally could see the guest redrawing when executed a find / in each. Even the host mouse pointer lagged . (radeon video issue???).
Comment 6 Reartes Guillermo 2011-06-03 16:43:09 EDT
I simple guest can barely use one youtube instance on this host. The host feels like an under-clocked old 386 swapping to tape over parallel port.

could it be a graphic performance problem? [radeon/xorg]
or a kvm / qemu-kvm problem?

the problem worsens when using virt-manager or virt-viewer to connect to the guest. if the guest is in the background, it is not so terrible.

without any vm guest running, xorg consumes between 5 ~ 10%, KDE DE on and slightly less with KDE DE off.

With on xp guest, xorg has peaks of +30% cpu usage.

All this is new, it started in the late beta for F15 and continued with
the released version.

Does anybody what this means/do?
  radeon_connector_analog_encoder_conflict_solve

I have seen it surpass 30% cpu usage running guest.

The system is usable for anything except virtualization. XVideo works ok even with h264 1080p.

but there is a noticeable slowness in screen update, for example i can se it update a firefox tab when i switch from one to other. There must be something wrong with radeon...

gpxe-roms-qemu.noarch            1.0.1-4.fc15              @fedora         
qemu-common.x86_64               2:0.14.0-7.fc15           @fedora         
qemu-img.x86_64                  2:0.14.0-7.fc15           @fedora         
qemu-kvm.x86_64                  2:0.14.0-7.fc15           @fedora         
qemu-system-x86.x86_64           2:0.14.0-7.fc15           @fedora

libvirt.x86_64                        0.8.8-4.fc15         @fedora         
libvirt-client.x86_64                 0.8.8-4.fc15         @fedora         
libvirt-python.x86_64                 0.8.8-4.fc15         @fedora 

virt-manager.noarch             0.8.7-4.fc15               @fedora

python-virtinst.noarch           0.500.6-2.fc15            @fedora

kernel 2.6.38.6-27.fc15.x86_64


The VGA Card is:  VGA compatible controller [0300]: ATI Technologies Inc Redwood [Radeon HD 5670] [1002:68d8]

I did not see any report of slowdown related to this, but i have the impression that there is a performance problem.

xorg-x11-drv-ati.x86_64   6.14.1-1.20110504gita6d2dba6.fc15   @koji-override-0/$releasever

This is after heavy usage (ake just boot an xp guest + yotube + power_off)
# cat /proc/interrupts 
           CPU0       CPU1       CPU2       
  0:        127          0         21   IO-APIC-edge      timer
  1:          0          0          4   IO-APIC-edge      i8042
  4:          0          0          1   IO-APIC-edge    
  7:          1          0          0   IO-APIC-edge    
  8:          0          0          1   IO-APIC-edge      rtc0
  9:          0          0          0   IO-APIC-fasteoi   acpi
 12:          0          0          7   IO-APIC-edge      i8042
 14:          0          0          0   IO-APIC-edge      pata_amd
 15:          0          0          0   IO-APIC-edge      pata_amd
 16:          0         38         21   IO-APIC-fasteoi   firewire_ohci
 17:          0          0         60   IO-APIC-fasteoi   snd_ca0106
 19:     261971          1         55   IO-APIC-fasteoi   eth1
 20:      64965          1         28   IO-APIC-fasteoi   ohci_hcd:usb3
 21:          7          0        107   IO-APIC-fasteoi   ehci_hcd:usb2
 22:          0          0         21   IO-APIC-fasteoi   ehci_hcd:usb1
 23:       1162          6         29   IO-APIC-fasteoi   ohci_hcd:usb4
 40:      97345          4       6644   PCI-MSI-edge      ahci
 41:       4256        321       1590   PCI-MSI-edge      radeon
 42:          0        144         31   PCI-MSI-edge      hda_intel
NMI:      37100      32576      23283   Non-maskable interrupts
LOC:    7402479    4680008    4335569   Local timer interrupts
SPU:          0          0          0   Spurious interrupts
PMI:      37100      32576      23283   Performance monitoring interrupts
IWI:         36         32         22   IRQ work interrupts
RES:    1650182    2087375    1009673   Rescheduling interrupts
CAL:      10533      13395      15502   Function call interrupts
TLB:       3398       2413       2609   TLB shootdowns
TRM:          0          0          0   Thermal event interrupts
THR:          0          0          0   Threshold APIC interrupts
MCE:          0          0          0   Machine check exceptions
MCP:         12         12         12   Machine check polls
ERR:          1
MIS:          0
Comment 7 Reartes Guillermo 2011-06-05 18:47:35 EDT
Tested again today, trying to reproduce the crash on newly created guests, and i was able to crash one F15 guest.

These are at the top of the list most of the time:  (# perf top)

svm_vcpu_run                                    /lib/modules/2.6.38.6-27.fc15.x86_64/kernel/arch/x86/kvm/kvm-amd.ko
_wordcopy_fwd_dest_aligned                      /lib64/libc-2.13.90.so
longest_match                                   /lib64/libz.so.1.2.5
g_atomic_pointer_get                            /lib64/libglib-2.0.so.0.2800.6 
radeon_connector_analog_encoder_conflict_solve  /lib/modules/2.6.38.6-27.fc15.x86_64/kernel/drivers/gpu/drm/radeon/radeon.ko

VM#0,1,2: F15-Gnome-LiveCD (32-BIT) (installed) (VM#0 was fully updated)
VM#3: WXP

This time VM#0 crashed with:

Jun  5 19:21:54 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 19:21:54 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 19:22:46 ulquiorra kernel: [23911.114435] qemu-kvm[5005]: segfault at 3f11000 ip 00000038a5a8b45b sp 00007fff7cf29398 error 4 in libc-2.13.90.so[38a5a00000+192000]
Jun  5 19:23:25 ulquiorra abrt[23424]: saved core dump of pid 5005 (/usr/bin/qemu-kvm) to /var/spool/abrt/ccpp-2011-06-05-19:22:46-5005.new/coredump (5524762624 bytes)
Jun  5 19:23:25 ulquiorra abrtd: Directory 'ccpp-2011-06-05-19:22:46-5005' creation detected
Jun  5 19:23:25 ulquiorra abrtd: Size of '/var/spool/abrt' >= 1000 MB, deleting 'ccpp-2011-06-05-01:52:21-9180'
Jun  5 19:23:25 ulquiorra abrt[23424]: /var/spool/abrt is 10692142582 bytes (more than 1562 MB), deleting 'ccpp-2011-06-05-01:52:21-9180'
Jun  5 19:23:25 ulquiorra abrtd: Lock file '/var/spool/abrt/ccpp-2011-06-05-01:52:21-9180/.lock' is locked by process 23424
Jun  5 19:23:25 ulquiorra abrtd: Lock file '/var/spool/abrt/ccpp-2011-06-05-01:52:21-9180/.lock' is locked by process 23424
Jun  5 19:23:26 ulquiorra abrtd: Lock file '/var/spool/abrt/ccpp-2011-06-05-01:52:21-9180/.lock' is locked by process 23424
Jun  5 19:23:26 ulquiorra abrtd: Lock file '/var/spool/abrt/ccpp-2011-06-05-01:52:21-9180/.lock' is locked by process 23424
Jun  5 19:23:27 ulquiorra kernel: [23951.553034] virbr0: port 1(vnet0) entering forwarding state
Jun  5 19:23:27 ulquiorra avahi-daemon[889]: Withdrawing address record for fe80::fc54:ff:fede:5ecc on vnet0.
Jun  5 19:23:27 ulquiorra kernel: [23951.563803] virbr0: port 1(vnet0) entering disabled state
Jun  5 19:23:27 ulquiorra avahi-daemon[889]: Withdrawing workstation service for vnet0.
Jun  5 19:23:27 ulquiorra kernel: [23951.564340] device vnet0 left promiscuous mode
Jun  5 19:23:27 ulquiorra kernel: [23951.564360] virbr0: port 1(vnet0) entering disabled state
Jun  5 19:23:27 ulquiorra abrtd: '/var/spool/abrt/ccpp-2011-06-05-01:52:21-9180' does not exist
Jun  5 19:23:27 ulquiorra libvirtd: 19:23:27.833: 1416: error : qemuMonitorJSONCommandWithFd:238 : cannot send monitor command '{"execute":"query-balloon"}': Connection reset by peer
Jun  5 19:23:31 ulquiorra abrtd: New dump directory /var/spool/abrt/ccpp-2011-06-05-19:22:46-5005, processing

Workload:
HOST: Firefox, Opera, Amarok, konsole (htop, perf top, iostat, mpstat, powertop, tail -f messages), virt-manager

VM#0: Firefox, youtube with two flash videos + 1 mplayer video
VM#1,2: Firefox, gnome-teminal at screen saver
VM#3: Opera, youtube with two flash videos

After varios hours, finally was able to crash one guest.
All VMs were installed today for the test.

Just one VM consumes most of the cpu.

Previously to the vm#0 crash i did this:
* tried with all guest windows (showing at leas partially) in the desktop, several times enabled/disabled composite (SHIFT+ALT+F12)
* issued in the host a (# find /&>/dev/null &) to consume io.
* went to VM#3 and tried to resize (as soon as flash starts the mouse lags, so it was difficult to do) and 
resized the video window for a minute. 

ABRT:
component = qemu 
reason = Process /usr/bin/qemu-kvm was killed by signal 11 (SIGSEGV)
cmdline = /usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name GNM3 -uuid a7ab8865-924a-3a93-ac46-e1071ae662d8 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/GNM3.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot order=c,menu=off -drive file=/var/lib/libvirt/images/GNM3.img,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/data0/software/iso/Fedora/15/Fedora-15-i686-Live-Desktop.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=21,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:de:5e:cc,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

Before starting testing i did disable these services that were incredibly enabled by default.
*lldpad, livesys, livesysd, iscsi, iscsid
Comment 8 Reartes Guillermo 2011-06-05 18:52:39 EDT
Clarification:

> Just one VM consumes most of the cpu.

When i say this, it means that just one guest is enough to consume all cpu time and crash (if ever crashes, not allways crashes).

Just tried with more guest to precipitate the crash and it worked.
Comment 9 Reartes Guillermo 2011-06-05 19:57:34 EDT
keep testing with the remaining guests, then VM#3 crashed 

Jun  5 19:25:18 ulquiorra pulseaudio[1915]: ratelimit.c: 2 events suppressed
Jun  5 19:25:52 ulquiorra pulseaudio[1915]: ratelimit.c: 2 events suppressed
Jun  5 19:30:00 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 19:30:00 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 19:30:27 ulquiorra pulseaudio[1915]: ratelimit.c: 2 events suppressed
Jun  5 19:33:53 ulquiorra pulseaudio[1915]: ratelimit.c: 4 events suppressed
Jun  5 19:35:22 ulquiorra dbus: [system] Activating service name='net.reactivated.Fprint' (using servicehelper)
Jun  5 19:35:22 ulquiorra dbus: [system] Successfully activated service 'net.reactivated.Fprint'
Jun  5 19:41:44 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.118 52:54:00:38:90:6e 
Jun  5 19:41:44 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.118 52:54:00:38:90:6e experien-012cf0
Jun  5 19:42:22 ulquiorra pulseaudio[1915]: ratelimit.c: 4 events suppressed
Jun  5 19:43:28 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 19:43:28 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 19:44:54 ulquiorra pulseaudio[1915]: ratelimit.c: 6 events suppressed
Jun  5 19:52:17 ulquiorra pulseaudio[1915]: ratelimit.c: 4 events suppressed
Jun  5 19:53:56 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 19:53:56 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 20:08:41 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.118 52:54:00:38:90:6e 
Jun  5 20:08:41 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.118 52:54:00:38:90:6e experien-012cf0
Jun  5 20:11:17 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 20:11:17 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 20:14:15 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 20:14:15 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 20:14:31 ulquiorra dbus: [system] Activating service name='net.reactivated.Fprint' (using servicehelper)
Jun  5 20:14:31 ulquiorra dbus: [system] Successfully activated service 'net.reactivated.Fprint'
Jun  5 20:21:41 ulquiorra dbus: [system] Activating service name='net.reactivated.Fprint' (using servicehelper)
Jun  5 20:21:41 ulquiorra dbus: [system] Successfully activated service 'net.reactivated.Fprint'
Jun  5 20:33:04 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 20:33:04 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 20:35:41 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 20:35:41 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 20:39:55 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.118 52:54:00:38:90:6e 
Jun  5 20:39:55 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.118 52:54:00:38:90:6e experien-012cf0
Jun  5 20:44:07 ulquiorra pulseaudio[1915]: ratelimit.c: 4 events suppressed
Jun  5 20:47:32 ulquiorra pulseaudio[1915]: ratelimit.c: 133 events suppressed
Jun  5 20:47:51 ulquiorra kernel: [29016.242185] qemu-kvm[11916]: segfault at 3749000 ip 00000038a5a8b2a8 sp 00007fff0fd580f8 error 4 in libc-2.13.90.so[38a5a00000+192000]
Jun  5 20:48:14 ulquiorra abrt[4393]: saved core dump of pid 11916 (/usr/bin/qemu-kvm) to /var/spool/abrt/ccpp-2011-06-05-20:47:51-11916.new/coredump (4888461312 bytes)
Jun  5 20:48:14 ulquiorra abrt[4393]: /var/spool/abrt is 10413282385 bytes (more than 1562 MB), deleting 'ccpp-2011-06-05-19:22:46-5005'
Jun  5 20:48:14 ulquiorra abrtd: Directory 'ccpp-2011-06-05-20:47:51-11916' creation detected
Jun  5 20:48:14 ulquiorra abrtd: Size of '/var/spool/abrt' >= 1000 MB, deleting 'ccpp-2011-06-05-19:22:46-5005'
Jun  5 20:48:14 ulquiorra abrtd: Lock file '/var/spool/abrt/ccpp-2011-06-05-19:22:46-5005/.lock' is locked by process 4393
Jun  5 20:48:14 ulquiorra libvirtd: 20:48:14.665: 1418: error : qemuMonitorJSONCommandWithFd:238 : cannot send monitor command '{"execute":"query-balloon"}': Connection reset by peer
Jun  5 20:48:14 ulquiorra kernel: [29038.999003] virbr0: port 4(vnet3) entering forwarding state
Jun  5 20:48:14 ulquiorra kernel: [29039.014194] virbr0: port 4(vnet3) entering disabled state
Jun  5 20:48:14 ulquiorra avahi-daemon[889]: Withdrawing address record for fe80::fc54:ff:fe38:906e on vnet3.
Jun  5 20:48:14 ulquiorra kernel: [29039.020141] device vnet3 left promiscuous mode
Jun  5 20:48:14 ulquiorra kernel: [29039.020182] virbr0: port 4(vnet3) entering disabled state
Jun  5 20:48:14 ulquiorra avahi-daemon[889]: Withdrawing workstation service for vnet3.
Jun  5 20:48:14 ulquiorra abrtd: '/var/spool/abrt/ccpp-2011-06-05-19:22:46-5005' does not exist
Jun  5 20:48:15 ulquiorra abrtd: New dump directory /var/spool/abrt/ccpp-2011-06-05-20:47:51-11916, processing

VM#3 (wXP) Workload: winscp, winrar, opera youtube + mplayer video. it keeps slowing down until freeze then crash.

ABRT:
component = qemu
reason = Process /usr/bin/qemu-kvm was killed by signal 11 (SIGSEGV)
cmdline = /usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name XPWa -uuid b110230b-526a-aeaf-77d9-7bbd16fc6a89 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/XPWa.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -boot c -drive file=/data1/vm/guille/virt-manager/XPWa.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/data0/software/iso/win/TinyXP-Rev09.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=21,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:38:90:6e,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:3 -vga std -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Comment 10 Reartes Guillermo 2011-06-05 21:19:14 EDT
Continued testing with VM#2 (no flash installed, non-updated pristine F15), and it crashed.
So, no flash, no video in this case. 

VM#2: Workload at the crash:
Firefox, several tabs (sites: sprint, newegg, asus, asterids html5 canvas*)
Firefox, another window (io unknown html5 canvas)
Both windows were partially visible. They ran for some time. I went away and
when i came back and unlocked the machine i saw it crashed.


Jun  5 20:51:56 ulquiorra pulseaudio[1915]: ratelimit.c: 4 events suppressed
Jun  5 20:56:40 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 20:56:40 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 20:59:48 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 20:59:48 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 21:02:56 ulquiorra pulseaudio[1915]: ratelimit.c: 6 events suppressed
Jun  5 21:11:26 ulquiorra pulseaudio[1915]: ratelimit.c: 4 events suppressed
Jun  5 21:13:47 ulquiorra pulseaudio[1915]: ratelimit.c: 2 events suppressed
Jun  5 21:14:21 ulquiorra pulseaudio[1915]: ratelimit.c: 2 events suppressed
Jun  5 21:19:58 ulquiorra gnome-keyring-daemon[12048]: GLib-GIO: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications.
Jun  5 21:19:58 ulquiorra gnome-keyring-daemon[12048]: couldn't set environment variable in session: The name org.gnome.SessionManager was not provided by any .service files
Jun  5 21:20:29 ulquiorra pulseaudio[1915]: ratelimit.c: 2 events suppressed
Jun  5 21:22:35 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 21:22:35 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 21:24:32 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 21:24:32 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 21:25:56 ulquiorra pulseaudio[1915]: ratelimit.c: 4 events suppressed
Jun  5 21:32:12 ulquiorra dbus: [system] Activating service name='net.reactivated.Fprint' (using servicehelper)
Jun  5 21:32:12 ulquiorra dbus: [system] Successfully activated service 'net.reactivated.Fprint'
Jun  5 21:46:18 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 21:46:18 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 21:49:49 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 21:49:49 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.99 52:54:00:3e:c7:43 
Jun  5 21:57:45 ulquiorra kernel: [33209.946432] qemu-kvm[7674]: segfault at 24f3000 ip 00000038a5a8b444 sp 00007fff2da84e18 error 4 in libc-2.13.90.so[38a5a00000+192000]
Jun  5 21:58:12 ulquiorra abrt[20935]: saved core dump of pid 7674 (/usr/bin/qemu-kvm) to /var/spool/abrt/ccpp-2011-06-05-21:57:45-7674.new/coredump (5432279040 bytes)
Jun  5 21:58:12 ulquiorra abrtd: Directory 'ccpp-2011-06-05-21:57:45-7674' creation detected
Jun  5 21:58:12 ulquiorra abrt[20935]: /var/spool/abrt is 10320797730 bytes (more than 1562 MB), deleting 'ccpp-2011-06-05-20:47:51-11916'
Jun  5 21:58:12 ulquiorra abrtd: Size of '/var/spool/abrt' >= 1000 MB, deleting 'ccpp-2011-06-05-20:47:51-11916'
Jun  5 21:58:12 ulquiorra abrtd: Lock file '/var/spool/abrt/ccpp-2011-06-05-20:47:51-11916/.lock' is locked by process 20935
Jun  5 21:58:13 ulquiorra libvirtd: 21:58:13.036: 1415: error : qemuMonitorJSONCommandWithFd:238 : cannot send monitor command '{"execute":"query-balloon"}': Connection reset by peer
Jun  5 21:58:13 ulquiorra kernel: [33237.366529] virbr0: port 3(vnet2) entering forwarding state
Jun  5 21:58:13 ulquiorra avahi-daemon[889]: Withdrawing address record for fe80::fc54:ff:fe3e:c743 on vnet2.
Jun  5 21:58:13 ulquiorra kernel: [33237.381144] virbr0: port 3(vnet2) entering disabled state
Jun  5 21:58:13 ulquiorra kernel: [33237.386154] device vnet2 left promiscuous mode
Jun  5 21:58:13 ulquiorra kernel: [33237.386187] virbr0: port 3(vnet2) entering disabled state
Jun  5 21:58:13 ulquiorra avahi-daemon[889]: Withdrawing workstation service for vnet2.
Jun  5 21:58:13 ulquiorra abrtd: '/var/spool/abrt/ccpp-2011-06-05-20:47:51-11916' does not exist
Jun  5 21:58:13 ulquiorra abrtd: New dump directory /var/spool/abrt/ccpp-2011-06-05-21:57:45-7674, processing
Jun  5 22:09:24 ulquiorra dbus: [system] Activating service name='net.reactivated.Fprint' (using servicehelper)
Jun  5 22:09:24 ulquiorra dbus: [system] Successfully activated service 'net.reactivated.Fprint'

ABRT:
component = qemu
reason = Process /usr/bin/qemu-kvm was killed by signal 11 (SIGSEGV)
cmdline = /usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name GNM3c -uuid 7947c00e-74f9-084a-0597-af001f6c88e0 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/GNM3c.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot order=c,menu=off -drive file=/var/lib/libvirt/images/gnm3c.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=21,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:3e:c7:43,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:2 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

Only VM#1 remains.
Comment 11 Reartes Guillermo 2011-06-05 22:15:42 EDT
Finally, VM#1 crashed.
Workload: Firefox (with hardware acceleration disabled), three windows:
* 3d earth html5 canvas demo
* some fancy mouse follower ligths html5 canvas demo
* some forest with snow falling html5 canvas demo


Jun  5 22:09:24 ulquiorra dbus: [system] Activating service name='net.reactivated.Fprint' (using servicehelper)
Jun  5 22:09:24 ulquiorra dbus: [system] Successfully activated service 'net.reactivated.Fprint'
Jun  5 22:12:02 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 22:12:02 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 22:37:37 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 22:37:37 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 22:42:39 ulquiorra pulseaudio[1915]: ratelimit.c: 6 events suppressed
Jun  5 22:48:13 ulquiorra pulseaudio[1915]: ratelimit.c: 119 events suppressed
Jun  5 22:52:50 ulquiorra dbus: [system] Activating service name='net.reactivated.Fprint' (using servicehelper)
Jun  5 22:52:50 ulquiorra dbus: [system] Successfully activated service 'net.reactivated.Fprint'
Jun  5 23:01:29 ulquiorra dnsmasq-dhcp[1505]: DHCPREQUEST(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 23:01:29 ulquiorra dnsmasq-dhcp[1505]: DHCPACK(virbr0) 192.168.122.10 52:54:00:54:ab:90 
Jun  5 23:05:33 ulquiorra yum[10507]: Installed: lm_sensors-3.3.0-2.fc15.x86_64
Jun  5 23:05:33 ulquiorra dbus: [system] Activating service name='org.freedesktop.PackageKit' (using servicehelper)
Jun  5 23:05:33 ulquiorra dbus: [system] Successfully activated service 'org.freedesktop.PackageKit'
Jun  5 23:07:07 ulquiorra kernel: [37372.017869] qemu-kvm[5624]: segfault at 2b93000 ip 00000038a5a8b270 sp 00007fffbfde2958 error 4 in libc-2.13.90.so[38a5a00000+192000]
Jun  5 23:07:25 ulquiorra pulseaudio[1915]: ratelimit.c: 4 events suppressed
Jun  5 23:07:37 ulquiorra abrt[11068]: saved core dump of pid 5624 (/usr/bin/qemu-kvm) to /var/spool/abrt/ccpp-2011-06-05-23:07:07-5624.new/coredump (5471817728 bytes)
Jun  5 23:07:37 ulquiorra abrtd: Directory 'ccpp-2011-06-05-23:07:07-5624' creation detected
Jun  5 23:07:37 ulquiorra abrt[11068]: /var/spool/abrt is 10904154570 bytes (more than 1562 MB), deleting 'ccpp-2011-06-05-21:57:45-7674'
Jun  5 23:07:37 ulquiorra abrtd: Size of '/var/spool/abrt' >= 1000 MB, deleting 'ccpp-2011-06-05-21:57:45-7674'
Jun  5 23:07:37 ulquiorra abrtd: Lock file '/var/spool/abrt/ccpp-2011-06-05-21:57:45-7674/.lock' is locked by process 11068
Jun  5 23:07:38 ulquiorra libvirtd: 23:07:38.047: 1414: error : qemuMonitorJSONCommandWithFd:238 : cannot send monitor command '{"execute":"query-balloon"}': Connection reset by peer
Jun  5 23:07:38 ulquiorra kernel: [37402.383061] virbr0: port 2(vnet1) entering forwarding state
Jun  5 23:07:38 ulquiorra avahi-daemon[889]: Withdrawing address record for fe80::fc54:ff:fe54:ab90 on vnet1.
Jun  5 23:07:38 ulquiorra kernel: [37402.400784] virbr0: port 2(vnet1) entering disabled state
Jun  5 23:07:38 ulquiorra kernel: [37402.408070] device vnet1 left promiscuous mode
Jun  5 23:07:38 ulquiorra kernel: [37402.408082] virbr0: port 2(vnet1) entering disabled state
Jun  5 23:07:38 ulquiorra avahi-daemon[889]: Withdrawing workstation service for vnet1.
Jun  5 23:07:38 ulquiorra abrtd: '/var/spool/abrt/ccpp-2011-06-05-21:57:45-7674' does not exist
Jun  5 23:07:38 ulquiorra abrtd: New dump directory /var/spool/abrt/ccpp-2011-06-05-23:07:07-5624, processing

ABRT:
component = qemu
reason = Process /usr/bin/qemu-kvm was killed by signal 11 (SIGSEGV)
cmdline = /usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name GNM3b -uuid 3c71198d-d718-a19b-74b0-ce70b7c56e82 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/GNM3b.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/var/lib/libvirt/images/GNM3b.img,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=21,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:54:ab:90,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

now, al 4 guests crashed so no more test for today...
Comment 12 Reartes Guillermo 2011-06-05 22:28:27 EDT
Created attachment 503143 [details]
Messages from recent kvm issues

I set SELinux to permissive before starting guest (at least for now).
Comment 13 Reartes Guillermo 2011-06-06 08:53:09 EDT
The host was powered off several hours and then on again. Tried again, with just only one guest (VM#0 [Gnome-LiveCD-i386] from yesterday)

But this time i set the host to use another window manager (enlightenment)
and set the guest to use also another window manager (windowmaker).
It ran 2 firefox windows with one youtube each one, it reproduced the videos several times, and then crashed.


Jun  6 08:51:21 ulquiorra yum[3517]: Installed: lua-debuginfo-5.1.4-8.fc15.x86_64
Jun  6 08:51:21 ulquiorra yum[3517]: Installed: embryo-debuginfo-1.0.0-2.fc15.x86_64
Jun  6 08:51:22 ulquiorra yum[3517]: Installed: ecore-debuginfo-1.0.0-2.fc15.x86_64
Jun  6 08:51:22 ulquiorra yum[3517]: Installed: edje-debuginfo-1.0.0-2.fc15.x86_64
Jun  6 09:02:01 ulquiorra yum[3611]: Installed: spice-debuginfo-0.8.1-1.fc15.x86_64
Jun  6 09:02:10 ulquiorra yum[3611]: Installed: 2:qemu-debuginfo-0.14.0-7.fc15.x86_64
Jun  6 09:02:11 ulquiorra yum[3611]: Installed: libaio-debuginfo-0.3.109-3.fc15.x86_64
Jun  6 09:02:11 ulquiorra yum[3611]: Installed: SDL-debuginfo-1.2.14-11.fc15.x86_64
Jun  6 09:02:12 ulquiorra yum[3611]: Installed: celt051-debuginfo-0.5.1.3-3.fc15.x86_64
Jun  6 09:02:12 ulquiorra dbus: [system] Activating service name='org.freedesktop.PackageKit' (using servicehelper)
Jun  6 09:02:12 ulquiorra dbus: [system] Successfully activated service 'org.freedesktop.PackageKit'
Jun  6 09:06:55 ulquiorra dbus: [system] Activating via systemd: service name='org.freedesktop.NetworkManager' unit='dbus-org.freedesktop.NetworkManager.service'
Jun  6 09:06:55 ulquiorra dbus: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.NetworkManager.service': Unit dbus-org.freedesktop.NetworkManager.service failed to load: No such file or directory. See system logs and 'systemctl status' for details.
Jun  6 09:13:59 ulquiorra dnsmasq-dhcp[1498]: DHCPREQUEST(virbr0) 192.168.122.65 52:54:00:de:5e:cc 
Jun  6 09:13:59 ulquiorra dnsmasq-dhcp[1498]: DHCPACK(virbr0) 192.168.122.65 52:54:00:de:5e:cc 
Jun  6 09:38:32 ulquiorra kernel: [ 5671.155314] qemu-kvm[3381]: segfault at 2680000 ip 00000038a5a8b404 sp 00007fff33798558 error 4 in libc-2.13.90.so[38a5a00000+192000]
Jun  6 09:38:59 ulquiorra abrt[6233]: saved core dump of pid 3381 (/usr/bin/qemu-kvm) to /var/spool/abrt/ccpp-2011-06-06-09:38:32-3381.new/coredump (5463896064 bytes)
Jun  6 09:38:59 ulquiorra abrtd: Directory 'ccpp-2011-06-06-09:38:32-3381' creation detected
Jun  6 09:38:59 ulquiorra abrt[6233]: /var/spool/abrt is 10935771979 bytes (more than 1562 MB), deleting 'ccpp-2011-06-05-23:07:07-5624'
Jun  6 09:38:59 ulquiorra abrtd: Size of '/var/spool/abrt' >= 1000 MB, deleting 'ccpp-2011-06-05-23:07:07-5624'
Jun  6 09:38:59 ulquiorra abrtd: Lock file '/var/spool/abrt/ccpp-2011-06-05-23:07:07-5624/.lock' is locked by process 6233
Jun  6 09:38:59 ulquiorra kernel: [ 5697.654028] virbr0: port 1(vnet0) entering forwarding state
Jun  6 09:38:59 ulquiorra avahi-daemon[872]: Withdrawing address record for fe80::fc54:ff:fede:5ecc on vnet0.
Jun  6 09:38:59 ulquiorra kernel: [ 5697.669790] virbr0: port 1(vnet0) entering disabled state
Jun  6 09:38:59 ulquiorra kernel: [ 5697.673126] device vnet0 left promiscuous mode
Jun  6 09:38:59 ulquiorra kernel: [ 5697.673155] virbr0: port 1(vnet0) entering disabled state
Jun  6 09:38:59 ulquiorra avahi-daemon[872]: Withdrawing workstation service for vnet0.
Jun  6 09:38:59 ulquiorra abrtd: '/var/spool/abrt/ccpp-2011-06-05-23:07:07-5624' does not exist
Jun  6 09:39:00 ulquiorra abrtd: New dump directory /var/spool/abrt/ccpp-2011-06-06-09:38:32-3381, processing

component = qemu
reason = Process /usr/bin/qemu-kvm was killed by signal 11 (SIGSEGV
cmdline =  /usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name GNM3 -uuid a7ab8865-924a-3a93-ac46-e1071ae662d8 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/GNM3.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot order=c,menu=off -drive file=/var/lib/libvirt/images/GNM3.img,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/data0/software/iso/Fedora/15/Fedora-15-i686-Live-Desktop.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=21,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:de:5e:cc,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
Comment 14 Reartes Guillermo 2011-06-06 17:59:15 EDT
Installed another guest, Slackware, configured with XFCE and firefox executing a single asteroids html5 canvas demo, it ran for a long time then
crashed. This time i choose scsi disk and e1000 nic.


Jun  6 17:52:55 ulquiorra dbus: [system] Successfully activated service 'net.reactivated.Fprint'
Jun  6 18:00:28 ulquiorra dnsmasq-dhcp[1498]: DHCPREQUEST(virbr0) 192.168.122.42 52:54:00:be:2b:95 
Jun  6 18:00:28 ulquiorra dnsmasq-dhcp[1498]: DHCPACK(virbr0) 192.168.122.42 52:54:00:be:2b:95 slk1
Jun  6 18:07:18 ulquiorra pulseaudio[1923]: ratelimit.c: 4 events suppressed
Jun  6 18:21:47 ulquiorra pulseaudio[1923]: ratelimit.c: 4 events suppressed
Jun  6 18:28:16 ulquiorra dnsmasq-dhcp[1498]: DHCPREQUEST(virbr0) 192.168.122.42 52:54:00:be:2b:95 
Jun  6 18:28:16 ulquiorra dnsmasq-dhcp[1498]: DHCPACK(virbr0) 192.168.122.42 52:54:00:be:2b:95 slk1
Jun  6 18:44:30 ulquiorra kernel: [38428.836821] qemu-kvm[3313]: segfault at 1d9b000 ip 00000038a5a8b41d sp 00007fff2d90ab18 error 4 in libc-2.13.90.so[38a5a00000+192000]
Jun  6 18:44:51 ulquiorra abrt[8558]: saved core dump of pid 3313 (/usr/bin/qemu-kvm) to /var/spool/abrt/ccpp-2011-06-06-18:44:30-3313.new/coredump (4907651072 bytes)
Jun  6 18:44:51 ulquiorra abrtd: Directory 'ccpp-2011-06-06-18:44:30-3313' creation detected
Jun  6 18:44:51 ulquiorra abrtd: Size of '/var/spool/abrt' >= 1000 MB, deleting 'ccpp-2011-06-06-14:19:50-25010'
Jun  6 18:44:51 ulquiorra abrt[8558]: /var/spool/abrt is 4914522977 bytes (more than 1562 MB), deleting 'ccpp-2011-06-06-14:19:50-25010'
Jun  6 18:44:51 ulquiorra abrt[8558]: Lock file '/var/spool/abrt/ccpp-2011-06-06-14:19:50-25010/.lock' is locked by process 879
Jun  6 18:44:51 ulquiorra abrt[8558]: '/var/spool/abrt/ccpp-2011-06-06-14:19:50-25010' does not exist
Jun  6 18:44:51 ulquiorra kernel: [38449.915932] virbr0: port 1(vnet0) entering forwarding state
Jun  6 18:44:51 ulquiorra avahi-daemon[872]: Withdrawing address record for fe80::fc54:ff:febe:2b95 on vnet0.
Jun  6 18:44:51 ulquiorra kernel: [38449.927948] virbr0: port 1(vnet0) entering disabled state
Jun  6 18:44:51 ulquiorra libvirtd: 18:44:51.610: 27820: error : qemuMonitorJSONCommandWithFd:238 : cannot send monitor command '{"execute":"query-balloon"}': Connection reset by peer
Jun  6 18:44:51 ulquiorra avahi-daemon[872]: Withdrawing workstation service for vnet0.
Jun  6 18:44:51 ulquiorra kernel: [38449.940154] device vnet0 left promiscuous mode
Jun  6 18:44:51 ulquiorra kernel: [38449.940188] virbr0: port 1(vnet0) entering disabled state
Jun  6 18:44:51 ulquiorra abrtd: New dump directory /var/spool/abrt/ccpp-2011-06-06-18:44:30-3313, processing

ABRT:
component = qemu
reason = Process /usr/bin/qemu-kvm was killed by signal 11 (SIGSEGV)
cmdline = /usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name TESTx1 -uuid 8508979e-afc1-53be-1cc9-6efa9b6e81ba -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/TESTx1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -device lsi,id=scsi0,bus=pci.0,addr=0x5 -drive file=/data1/vm/guille/virt-manager/TESTx1.img,if=none,id=drive-scsi0-0-0,boot=on,format=raw -device scsi-disk,bus=scsi0.0,scsi-id=0,drive=drive-scsi0-0-0,id=scsi0-0-0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=21,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:be:2b:95,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

---

# cat /proc/cpuinfo 
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 4
model name      : AMD Phenom(tm) II X3 710 Processor
stepping        : 2
cpu MHz         : 800.000
cache size      : 512 KB
physical id     : 0
siblings        : 3
core id         : 0
cpu cores       : 3
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt arat npt lbrv svm_lock nrip_save
bogomips        : 5200.69
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 4
model name      : AMD Phenom(tm) II X3 710 Processor
stepping        : 2
cpu MHz         : 800.000
cache size      : 512 KB
physical id     : 0
siblings        : 3
core id         : 1
cpu cores       : 3
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt arat npt lbrv svm_lock nrip_save
bogomips        : 5199.98
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor       : 2
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 4
model name      : AMD Phenom(tm) II X3 710 Processor
stepping        : 2
cpu MHz         : 800.000
cache size      : 512 KB
physical id     : 0
siblings        : 3
core id         : 2
cpu cores       : 3
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt arat npt lbrv svm_lock nrip_save
bogomips        : 5199.98
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate


# cat /proc/interrupts 
           CPU0       CPU1       CPU2       
  0:        127          0        226   IO-APIC-edge      timer
  1:          0          0          4   IO-APIC-edge      i8042
  7:          1          0          0   IO-APIC-edge    
  8:          0          0          1   IO-APIC-edge      rtc0
  9:          0          0          0   IO-APIC-fasteoi   acpi
 12:          0          0          7   IO-APIC-edge      i8042
 14:          0          0          0   IO-APIC-edge      pata_amd
 15:          0          0          0   IO-APIC-edge      pata_amd
 16:        586          0         21   IO-APIC-fasteoi   firewire_ohci
 18:          0     104575        305   IO-APIC-fasteoi   snd_ca0106
 19:     467140          1         57   IO-APIC-fasteoi   eth1
 20:     701938          1         28   IO-APIC-fasteoi   ohci_hcd:usb3
 21:          0          0         36   IO-APIC-fasteoi   ehci_hcd:usb2
 22:          0          0         21   IO-APIC-fasteoi   ehci_hcd:usb1
 23:      28448         28         32   IO-APIC-fasteoi   ohci_hcd:usb4
 40:    1328112         16       6782   PCI-MSI-edge      ahci
 41:    2211645          2       4222   PCI-MSI-edge      radeon
 42:          0        662         31   PCI-MSI-edge      hda_intel
NMI:   26671018   28107296   38276452   Non-maskable interrupts
LOC:   50036768   48118752   46066104   Local timer interrupts
SPU:          0          0          0   Spurious interrupts
PMI:   25158410   26783311   37237356   Performance monitoring interrupts
IWI:      26131      27567      37461   IRQ work interrupts
RES:   15497541   16267306    9042648   Rescheduling interrupts
CAL:      85225      80548      71769   Function call interrupts
TLB:     145322     112394     100123   TLB shootdowns
TRM:          0          0          0   Thermal event interrupts
THR:          0          0          0   Threshold APIC interrupts
MCE:          0          0          0   Machine check exceptions
MCP:        130        130        130   Machine check polls
ERR:          1
MIS:          0

# virsh capabilities | grep kvm
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>

# virsh dumpxml TESTx1 | grep kvm
<domain type='kvm' id='4'>
    <emulator>/usr/bin/qemu-kvm</emulator>

# lsmod | grep kvm
kvm_amd                44852  3 
kvm                   306680  1 kvm_amd
Comment 15 Reartes Guillermo 2011-06-06 19:24:40 EDT
Booted the xp instance, tried with several youtube instances and it did not crash, cpu usage was high but a litle lower than previous examples. Then i used mplayer to play an avi, i let it play and after half the video it crashed.

It seems that with debug virt-manager uses some qt look, this does not happen if i launch it normally (ALT+F2 or menu).


/var/log/messages:

Jun  6 20:00:17 ulquiorra kernel: [42975.806717] qemu-kvm[11075]: segfault at 2368000 ip 00000038a5a8b45b sp 00007fffe29deb18 error 4 in libc-2.13.90.so[38a5a00000+192000]



virt-manager debug (more or less near the time of the segfault since the file is too big):

20:00:39.881: 14115: debug : remoteIODecodeMessageLength:9821 : Got length, now need 256 total (252 more)
20:00:39.947: 14115: debug : remoteIOEventLoop:10339 : Giving up the buck 16 0x7f0fd01c7810 (nil)
20:00:39.947: 14115: debug : remoteIO:10513 : All done with our call 16 (nil) 0x7f0fd01c7810
Exception in thread Tick thread:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/share/virt-manager/virtManager/engine.py", line 431, in _tick
    conn.tick()
  File "/usr/share/virt-manager/virtManager/connection.py", line 1592, in tick
    vm.tick(now)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1484, in tick
    info = self.get_info()
  File "/usr/share/virt-manager/virtManager/domain.py", line 1024, in get_info
    return self._backend.info()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 823, in info
    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
libvirtError: cannot send monitor command '{"execute":"query-balloon"}': Connection reset by peer

20:00:40.615: 14134: debug : virNodeGetInfo:4018 : conn=0x7f0ff000d220, info=0x7f0fdbffe040
20:00:40.615: 14134: debug : remoteIO:10413 : Do proc=6 serial=95473 length=28 wait=(nil)
20:00:40.615: 14134: debug : remoteIO:10485 : We have the buck 6 0x7f0fd025c220 0x7f0fd025c220
Comment 16 Reartes Guillermo 2011-06-17 14:28:00 EDT
The situation seem to have improved, either by an update or because i 
changed the kernel command line and added the kernel parameter "nohz=off".
I will first confirm that the situation has improved and then i will
remove the "nohz=off" parameter and check if the issue returns.

-----------------------------------------------------------------------------------------------------------------
   PerfTop:    2732 irqs/sec  kernel:50.6%  exact:  0.0% [1000Hz cycles],  (all, 3 CPUs)
-----------------------------------------------------------------------------------------------------------------

             samples  pcnt function                                       DSO
             _______ _____ ______________________________________________ __________________________

            12197.00 32.1% svm_vcpu_run                                   [kvm_amd]                 
             3363.00  8.9% _wordcopy_fwd_dest_aligned                     /lib64/libc-2.14.so       
             3190.00  8.4% longest_match                                  /lib64/libz.so.1.2.5      
             1230.00  3.2% g_atomic_pointer_get                           libglib-2.0.so.0.2800.8   
              635.00  1.7% radeon_connector_analog_encoder_conflict_solve [radeon]                  
              548.00  1.4% deflate_slow                                   /lib64/libz.so.1.2.5      
              475.00  1.3% __GI_memmove                                   /lib64/libc-2.14.so       
              419.00  1.1% tight_detect_smooth_image16                    /usr/bin/qemu-kvm         
              285.00  0.8% pqdownheap                                     /lib64/libz.so.1.2.5      
              252.00  0.7% resampler_basic_interpolate_single             libspeexdsp.so.1.5.0      
              251.00  0.7% find_vma                                       [kernel.kallsyms]         
              246.00  0.6% inflate_fast                                   /lib64/libz.so.1.2.5  

Each core 60-70% cpu utilization

# virsh list
 Id Name                 State
----------------------------------
  1 GNM3                 running
  2 XPWa                 running
  3 GNM3b                running
  4 GNM3c                running
  5 TESTx1               running
  6 CENTOS               running
  7 CENT                 running

The XPWa was rendering youtube videos.

The host cpu never reached 100%, previously 100% cpu usage could be reached by just one guest and crashing that guest was easy.

Unchanged versions:

gpxe-roms-qemu.noarch             1.0.1-4.fc15               @fedora          
qemu-common.x86_64                2:0.14.0-7.fc15            @fedora          
qemu-debuginfo.x86_64             2:0.14.0-7.fc15            @fedora-debuginfo
qemu-img.x86_64                   2:0.14.0-7.fc15            @fedora          
qemu-kvm.x86_64                   2:0.14.0-7.fc15            @fedora          
qemu-system-x86.x86_64            2:0.14.0-7.fc15            @fedora 

libvirt.x86_64                 0.8.8-4.fc15       @fedora                     
libvirt-client.x86_64          0.8.8-4.fc15       @fedora                     
libvirt-python.x86_64          0.8.8-4.fc15       @fedora                     

python-virtinst.noarch         0.500.6-2.fc15     @fedora                     

virt-manager.noarch            0.8.7-4.fc15       @fedora                     
virt-viewer.x86_64             0.3.1-1.fc15       @fedora                     

Changed versions:

xorg-x11-drv-ati.x86_64  6.14.1-1.20110504gita6d2dba6.fc15 @koji-override-0/$releasever

kernel 2.6.38.8-32.fc15.x86_64

Also glibc was updated.

# cat /proc/interrupts 
           CPU0       CPU1       CPU2       
  0:        127          0         92   IO-APIC-edge      timer
  1:          0          0          4   IO-APIC-edge      i8042
  7:          1          0          0   IO-APIC-edge    
  8:          0          0          1   IO-APIC-edge      rtc0
  9:          0          0          0   IO-APIC-fasteoi   acpi
 12:          0          0          7   IO-APIC-edge      i8042
 14:          0          0          0   IO-APIC-edge      pata_amd
 15:          0          0          0   IO-APIC-edge      pata_amd
 16:          0        226         20   IO-APIC-fasteoi   firewire_ohci
 18:      83320          3        294   IO-APIC-fasteoi   sata_sil24, snd_ca0106
 19:      98617          0         55   IO-APIC-fasteoi   eth1
 20:     134299          1         27   IO-APIC-fasteoi   ohci_hcd:usb3
 21:          0          0         36   IO-APIC-fasteoi   ehci_hcd:usb2
 22:          0          0         21   IO-APIC-fasteoi   ehci_hcd:usb1
 23:       4831         18         32   IO-APIC-fasteoi   ohci_hcd:usb4
 40:     293416         28       8324   PCI-MSI-edge      ahci
 41:      73505        739      13564   PCI-MSI-edge      radeon
 42:         58        587         31   PCI-MSI-edge      hda_intel
NMI:     526967     526129     527215   Non-maskable interrupts
LOC:   20159868   20632458   20061784   Local timer interrupts
SPU:          0          0          0   Spurious interrupts
PMI:     526967     526129     527215   Performance monitoring interrupts
IWI:        514        512        510   IRQ work interrupts
RES:    6128225    6562501    3415773   Rescheduling interrupts
CAL:       5627       5994       6027   Function call interrupts
TLB:      21439      25065      25803   TLB shootdowns
TRM:          0          0          0   Thermal event interrupts
THR:          0          0          0   Threshold APIC interrupts
MCE:          0          0          0   Machine check exceptions
MCP:         53         53         53   Machine check polls
ERR:          1
MIS:          0
Comment 17 Reartes Guillermo 2011-07-15 18:54:40 EDT
I reinstalled and recreated all filesystems except /home.
The default storage pool is now a raid0 and i will add another pool later.

Creating an xp guest with all default settings, while the screen is at the partition disk screen, top reports 100% cpu usage by qemu-kvm, mpstat reports 100% guest for one core. htop also reports 100% usage by qemu-kvm

---

qemu-kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name WINX1 -uuid 9760b38f-65e2-890b-95b8-920f917a94f9 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/WINX1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-reboot -boot dc -drive file=/var/lib/libvirt/images/WINX1.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/data1/iso/win/TinyXP-Rev09.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=21,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:e2:76:db,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -vga std -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

---

qemu-kvm.x86_64  2:0.14.0-7.fc15
kernel           2.6.38.8-35.fc15.x86_64

----

virt-manager was launches as root, disabling KDE DE does not change cpu usage pattern.

SELinux is disabled for now.
Comment 18 Reartes Guillermo 2011-07-15 18:55:51 EDT
Sorry, SELinux is permissive for now
Comment 19 Justin M. Forbes 2011-08-16 20:00:33 EDT
Using spice instead of the cirrus VGA will make a very significant difference in video performance, including the ability to watch fullscreen video.  Additionally, using virtio devices increase performance massively over rtl8189 and ide.
Comment 20 Nikolai Maziashvili 2011-09-04 06:43:32 EDT
I do have another example of extremely poor kvm/qemu performance. I do not even go to runlevel 5.
I have tested this over and over again and results were consistent - qemu on F15 host performs twice (relative of course) as worth as virtualbox on the same F15 host, while 10 times worth then vbox on windows 7 host.
This what led me to this conclusion:
My host set up (laptop) - 
    CPU    - i7 M 620  @ 2.67GHz
    Memory - 8G
    Hypervisor - qemu 
Guest:
    F16 alpha (fully updated)
    CPU - 2
    mem - 2024M
Guest is installed on logical volume (on SSD drive), booted into runlevel 3 and i logged into guest from remote console. I am trying to build rpm and was running ./configure to find out build requirements. It took 34 seconds to finish it up.

At work i have Win7 laptop where i use vbox to create my own working environment (Linux) and do most of my work from there. Out of curiosity i installed F16 alpha on that win7 laptop with same 2G (actually 1792M) of memory and 2 virtcpus. Basically same guest but different host.
Win7 laptop runs on i5 intel, don't remember clock speed, but it wasn't higher then on my home laptop. and 4G of memory.
It took 4 seconds to finish running configure there. 4 seconds!!!

Later at home i installed vbox on my home laptop and installed F16 as vbox guest with same specs as previously. It took 14 seconds to run configure there, still more then on win7 host.

I do not have any idea why, but it seems that virtualization on F15 host is under performing compare to Windows. And qemu is lagging behind vbox.
Comment 21 Reartes Guillermo 2011-09-10 18:44:08 EDT
The problem is that in addition to the slowness of qemu-kvm (amd in my case) there is the fact that ALL guests crashes allways if pushed a bit.

Perf top shows that '_wordcopy_fwd_dest_aligned' is very high prior to the guest crash (and the guest becomes hypersluggish: mouse desync)

I managed to crash linux and xp guests.

The last time that fedora worked fine was one month and a half before f13 EOLing. (i used f13 with a downgraded qemu-kvm/libvirt/vnc combo until moved to f15 to wich i could not test qemu-kvm during alpha/beta due to other problems at that time). At that time qemu-kvm did not have that issue.
Comment 22 Reartes Guillermo 2011-10-27 21:32:09 EDT
I think that i finally can see some light on the root of the issue...

Using Virt-Manager, on a F15 KDE Live-CD install, X86_64. 
(Phenom II X3 with 8gb RAM)

I have several issues combined:

* guest video hardware:
  The default is vnc/cirrus, wich is slower than spice/qlx.
  Changed it to spice/qlx and installed the guest os drivers (in this test,
  it was a wxp guest). 

* guest hard disks:
  A: the default disk formwat is RAW ---> should be QCOW2? (or whatever is the
  recommended disk format)

  B: virtio should be used whenever possible, with guest os drivers if 
  necesary.

* procesor: (this was a surprise)

  The default virt-manager options produces a "dwarf" cpu for the guest, in
  wxp guest, cpu-z reports it as a k6 with mmx and barely any other flags,
  using "copy_cpu_host" and apply produces a normal guest which cpu-z 
  reports as  a phenom quad-core (it is closer to the real phenom2 x3).

Each of these items slows down the thing, but when you have ALL of them, it is just unusable and crashes (vm segfault) linux/windows guests...
The worse part was the cpu issue combined with vnc/cirrus. 

I will test various distribution to check that it makes sense in all of them.
At least wxp with the correct procesor options and spice/glx video is much better (i will test how much in the following days).
Comment 23 Reartes Guillermo 2011-10-29 15:29:29 EDT
A centos 5.3 host (previously created, cirrus, ide:raw):

The default cpu offered to the guest was:

processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 6
model		: 2
model name	: QEMU Virtual CPU version 0.14.0
stepping	: 3
cpu MHz		: 2600.054
cache size	: 512 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 4
wp		: yes
flags		: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up pni cx16 popcnt lahf_lm altmovcr8 abm
bogomips	: 5267.46
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

And after using "copy_host_cpu" (it set it to opteron_g3)

processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 16
model		: 2
model name	: AMD Phenom(tm) 9550 Quad-Core Processor
stepping	: 3
cpu MHz		: 2600.060
cache size	: 512 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb lm up pni cx16 popcnt lahf_lm cmp_legacy svm cr8_legacy altmovcr8 abm sse4a misalignsse
bogomips	: 5419.09
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

The guest performance improved, of course, video is still a bit sluggish (is still using cirrus) but it did not crash in my test.

Why does virt-manager offer a different cpu that the host by default?
Using a different cpu on the guest should be an option, not the default.
Not only were the guests slow, but they crashed just by making circles with
the mouse in the guest display!!! after some minutes. (or they just 
crashed on they own, or if lucky, the machine was just slow).

Regarding storage, well, i am not so sure if there is only one to recommend, but maybe add some very brief description will suffice. (and keep raw as default).

Regarding video, yup, spice/qlx is way better, sadly it is not the default and
one must download drivers that are offered only in .zip and not in .iso

I am confident that 75% of my problems were created by the cpu issue, the rest were cirrus/qlx and virtio + qcow2 images also helped.

Currently, it is TOO EASY to create a under performing or crash prone guest
with virt-manager. It is an UI issue, probably adding small descriptions somewhere explaining each format pro an cons (without pointing to an internet url).
Comment 24 Reartes Guillermo 2011-10-29 15:45:23 EDT
One more thing:

Previously existing guest created with older virt-manager, automatically used
that "default" bad cpu. For me, it started in the last days of F13, when virt-manager updated.

Suddenly everything was slow. That was caused by both:

* cirrus/vnc, wich got slower but sharper and better looking (since i did notice it). I could workaround it in F13 by downgrading vnc to some version (forgot it).

* the "default" cpu, which suddenly and without notice stopped being the same
as the host and was changed to something else. I did not realize that that could be the case, i pointed cirrus/vnc as the sole culprit.

When i switched to F15, it was worse!
Then i reinstalled, recreated all filesystem (thankfully due to changing disk layout/new disks and not due to virt-manager itself :-) ), and the problem was still there.
So i recreated all guests!!! with the cpu problem. Sadly i did not test spice/qlx at that time due to lack of knowledge of spice workings. (it was not so difficult after all.)

So i was having a bad cpu + slow display guests and pointing to a display problem cirrus/vnc and i even suspected the host xorg-x11-drv-ati driver.
Comment 25 Reartes Guillermo 2011-11-28 16:25:34 EST
Please close this bug-report.

I splited this into three separate bug-reports
* cpu          (757872)
* display      (757874) 
* disk format  (757879)

The most important is the cpu issue.

Thanks.
Comment 26 Fedora Admin XMLRPC Client 2012-03-15 13:52:46 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 27 Cole Robinson 2012-05-28 19:21:38 EDT
Closing as suggested in comment #25