Description of problem: I have troubles to run qemu guests on multiple machines. May be there are different problems. I was not able to debug all of them, just this one. I am using fully updated Fedora 11 and Fedora 12 guests on fully updated Fedora 11 host (only stable updates). My hardware is HP ML380 server with intel processors (VT enabled), 16 GB RAM. Guests are using virtio drivers for network and disk. For each guest there are 2 iSCSI LUNs (from different arrays), in guest they are in mirror and LVM partitioned (except root). Version-Release number of selected component (if applicable): kernel-2.6.30.10-105.2.4.fc11.x86_64 qemu-kvm-0.10.6-9.fc11.x86_64 How reproducible: daily Steps to Reproduce: Unable to reproduce. Actual results: Frozen guest. Unable to ping guest from network, serial console is not responding (unable to type chars), no messages on VGA console, I can only press enter on VGA console, but can't type letters or other characters. By pressing enter, only cursor is moved, login prompt is not updated. Expected results: Response from guest. Additional info: This is a backtrack from qemu process: (gdb) bt #0 0x00007f7051773fc2 in select () from /lib64/libc.so.6 #1 0x0000000000409b97 in qemu_select (tv=<value optimized out>, xfds=<value optimized out>, wfds=<value optimized out>, rfds=<value optimized out>, max_fd=<value optimized out>) at /usr/src/debug/qemu-kvm-0.10.6/vl.c:3689 #2 main_loop_wait (tv=<value optimized out>, xfds=<value optimized out>, wfds=<value optimized out>, rfds=<value optimized out>, max_fd=<value optimized out>) at /usr/src/debug/qemu-kvm-0.10.6/vl.c:3788 #3 0x000000000051ec0a in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.10.6/qemu-kvm.c:596 #4 0x000000000040e981 in main_loop () at /usr/src/debug/qemu-kvm-0.10.6/vl.c:3851 #5 main () at /usr/src/debug/qemu-kvm-0.10.6/vl.c:6140 What else I can give?
Similar problem or another guest (this tame host was on 2.6.30.10-105.2.16.fc11.x86_64). Curious, that after aprox. one hour system goes into normal state. In guest log there is nothing in logs from this time, after this time there is this message: Feb 11 04:22:48 joblife kernel: BUG: soft lockup - CPU#3 stuck for 4098s! [bzip2 :26807] Feb 11 04:22:48 joblife kernel: Modules linked in: ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 i2c_piix4 i2c_core virtio_net virtio_balloon jo ydev raid1 virtio_blk virtio_pci virtio_ring virtio [last unloaded: scsi_wait_sc an] Feb 11 04:22:48 joblife kernel: CPU 3: Feb 11 04:22:48 joblife kernel: Modules linked in: ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 i2c_piix4 i2c_core virtio_net virtio_balloon jo ydev raid1 virtio_blk virtio_pci virtio_ring virtio [last unloaded: scsi_wait_sc an] Feb 11 04:22:48 joblife kernel: Pid: 26807, comm: bzip2 Not tainted 2.6.31.12-17 4.2.3.fc12.x86_64 #1 Feb 11 04:22:48 joblife kernel: RIP: 0033:[<00007fd6741f82a0>] [<00007fd6741f82a0>] 0x7fd6741f82a0 Feb 11 04:22:48 joblife kernel: RSP: 002b:00007fff5d836770 EFLAGS: 00000206 Feb 11 04:22:48 joblife kernel: RAX: 00000000016ad3e8 RBX: 00000000016ad550 RCX: 00007fff5d8374e5 Feb 11 04:22:48 joblife kernel: RDX: 0000000000012e60 RSI: 00000000000006f3 RDI: 0000000000000041 Feb 11 04:22:48 joblife kernel: RBP: ffffffff8101286e R08: 00007fd6743fa460 R09: 0000000000000020 Feb 11 04:22:48 joblife kernel: R10: 0000000000000020 R11: 0000000000000020 R12:: Feb 11 04:22:48 joblife kernel: R13: 00000000000009d3 R14: 00007fd6741e56a0 R15: 0000000000000000 Feb 11 04:22:48 joblife kernel: FS: 00007fd674611700(0000) GS:ffff880001a13000( 0000) knlGS:0000000000000000 Feb 11 04:22:48 joblife kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Feb 11 04:22:48 joblife kernel: CR2: 00007f7d29334000 CR3: 0000000011b67000 CR4: 00000000000006e0 Feb 11 04:22:48 joblife kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Feb 11 04:22:48 joblife kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Feb 11 04:22:48 joblife kernel: Call Trace: Feb 11 04:22:48 joblife kernel: Clocksource tsc unstable (delta = -4397562056334 ns) I am seeing similar (clocksource unstable) messages on multiple virtual machines, also they are freezing, but sometimes only for some minutes. Is it possible, that with very high load (>30), I am not able to ping my guest?
Is this still happening, does it reproduce on F-12?
(In reply to comment #2) > Is this still happening, does it reproduce on F-12? Did you mean f12 guest or host? I can't update my production servers to f12 host yet. It it still happening, mostly only for some minutes.
Created attachment 394873 [details] messages from last crash Next ganhup again. Today I was able to ping guest, but nothing else. Unable to log in serial or VGA console. In virt manager there was no CPU or disk usage for this guest, only small network activity.
(In reply to comment #2) > Is this still happening, does it reproduce on F-12? Guests are on F12, on different hardware there is no similar problem with F12, but I can't upgrade to F12 on same host due to bz#548198. Any chance to fix this in F11? Can help me to run F12 with F11 kernel on my host? At least ksm will not work, but can this fix my problem?
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
I wouldn't use the Fedora 12 kernel, but the same version (2.6.32.9) is now available in Fedora 11 updates-testing repository. Mind giving that a try?
I am now running my hosts on Fedora 12. No problems yet, but they are running only aprox. 2 days.
Looks like this happens with Fedora 12 (host and guest) too. Host: kernel-2.6.32.9-67.fc12.x86_64 qemu-kvm-0.11.0-13.fc12.x86_64 Guest: kernel-2.6.32.9-70.fc12.x86_64 Yesterday morning my guest load increased to 30, I was unable to find, what happened. SSH responses was sometimes slow. Evening there was no response from console or over network. I will try to strace/debug qemu process if this happens next time.
Thanks for the update, let's see if we can debug this next time it happens. Setting needinfo awaiting debug output.
Created attachment 405218 [details] Strace after crash. Same versions on host and guest. My guest was dead again. I was not able to ping it, serial console and VGA console was dead. There was no info (except normal login prompt) on VGA console, my serial console was not open during crash. Can I access serial console log history somewhere? With xen when serial console is opened, all history from last access is dumped. Attaching strace output and here is backtrace. I see nothing interesant. What else I can do? #0 0x00007fc2f88d73e3 in select () at ../sysdeps/unix/syscall-template.S:82 #1 0x000000000040a633 in main_loop_wait (timeout=<value optimized out>) at /usr/src/debug/qemu-kvm-0.11.0/vl.c:4181 #2 0x00000000004231aa in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.11.0/qemu-kvm.c:2079 #3 0x000000000040f157 in main_loop (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.11.0/vl.c:4393 #4 main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.11.0/vl.c:6263
Created attachment 405401 [details] dmesg output after problem Today I was able to log into guest. I can't access /proc/mdstat or run top or vmstat, but I can see dmesg output or some other files.
Created attachment 406751 [details] Strace after crash. Same problem with: kernel-2.6.32.11-99.fc12.x86_64 qemu-kvm-0.12.3-2.fc12.x86_64 (f12 updates testing) Today this happened multiple times for multiple servers. I have iSCSI attached disks for those servers. They are attached to host and then added to guest ad virtio disks. Is it possible, that this situation appears when an error occurs with iSCSI arrays? Strace log attached, gdb follows.
Created attachment 406752 [details] GDB backtrace after crash.
Same problem with <driver cache='off'>, after some time machine is stopped (no response on serial, VGA console or network, no response to SysRq). Because there is no real response from qemu developers, may be it is a kernel bug. Trying to change component to kernel. Kernel guys, please look at this bug. Same problem happens with all versions of qemu (starting with F11 ending with F12+testing).
This message is a reminder that Fedora 11 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 11. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '11'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 11's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 11 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Same problem on Fedora 12 or Fedora 12 testing. Changing version to Fedora 12.
Created attachment 410425 [details] kvm_stat --batch Attaching kvm_stat logs, may be it will help to fix this problem. There was no other guest running on this host, just this one. Is there anything else, I can do to debug this problem?
Created attachment 410427 [details] kvm_stat --log
Next time, please run this: ps Hax -o pid,command,state,wchan | grep '[k]vm' Also, in gdb, please run: info threads thread apply all bt A core dump of virtual machine could be useful -- it can be analyzed with 'crash' utility. To get a dump, use 'virsh dump' command.
Created attachment 412739 [details] ps Hax -o pid,command,state,wchan output My guest was restarted by my automatic restart script before I was able to get more info, but here is at least ps output.
Created attachment 413335 [details] gdb output for threads gdb output for: info threads thread apply all bt is in attachment (screen log), ps output: 847 [kvm-irqfd-clean] S worker_thread 4026 /usr/bin/qemu-kvm -S -M pc- S poll_schedule_timeout 4026 /usr/bin/qemu-kvm -S -M pc- R - 4026 /usr/bin/qemu-kvm -S -M pc- S kvm_vcpu_block 4026 /usr/bin/qemu-kvm -S -M pc- R - 4026 /usr/bin/qemu-kvm -S -M pc- R - 4026 /usr/bin/qemu-kvm -S -M pc- S kvm_vcpu_block 4026 /usr/bin/qemu-kvm -S -M pc- S kvm_vcpu_block 4026 /usr/bin/qemu-kvm -S -M pc- R - 4026 /usr/bin/qemu-kvm -S -M pc- S kvm_vcpu_block I was unable to dump guest on this machines. After command virsh dump nothing happens, just waits, also libvirtd does not respond to other commands. There was no dump file (as named on command line) in current directory. May be problem was, that this guest has 12 GB RAM. I will try to dump on a smaller guest. Can I do more to debug this problem?
Next crash of another machine again. These is very curious time travelling in logs, see: /var/log/messages: May 12 12:01:35 www auditd[937]: Audit daemon rotating log files May 28 21:24:50 www kernel: BUG: soft lockup - CPU#3 stuck for 1179849625s! [aws tats.pl:6340] May 28 21:24:50 www kernel: Modules linked in: ipv6 xt_connlimit i2c_piix4 i2c_core virtio_balloon virtio_net virtio_blk virtio_pci [last unloaded: scsi_wait_scan] May 28 21:24:50 www kernel: May 28 21:24:50 www kernel: Pid: 6340, comm: awstats.pl Not tainted (2.6.32.11-99.fc12.i686.PAE #1) May 28 21:24:50 www kernel: EIP: 0073:[<0065328c>] EFLAGS: 00000293 CPU: 3 May 28 21:24:50 www kernel: EIP is at 0x65328c May 28 21:24:50 www kernel: EAX: 00000047 EBX: 007344ac ECX: 00000000 EDX: 00000000 May 28 21:24:50 www kernel: ESI: 08d991a4 EDI: 00000040 EBP: bf8c6518 ESP: bf8c6480 May 28 21:24:50 www kernel: DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b May 28 21:24:50 www kernel: CR0: 80050033 CR2: 08d9d024 CR3: 08131000 CR4: 000006f0 May 28 21:24:50 www kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 May 28 21:24:50 www kernel: DR6: ffff0ff0 DR7: 00000400 May 28 21:24:50 www kernel: Call Trace: May 28 21:24:50 www kernel: BUG: soft lockup - CPU#2 stuck for 1179849612s! [mysqld: /var/log/httpd/access_log: 207.46.13.87 - - [12/May/2010:13:01:21 +0200] "GET /pravnicka-fakulta/veda-a-vys 207.46.13.87 - - [12/May/2010:13:01:21 +0200] "GET /lekarska-fakulta/klinicke-us 193.87.75.250 - - [28/May/1914:21:25:20 +0100] "GET / HTTP/1.0" 302 303 "-" "Moz 193.87.75.82 - - [28/May/1914:21:25:20 +0100] "GET / HTTP/1.0" 302 304 "-" "Mozi 158.197.128.175 - - [28/May/1914:21:25:20 +0100] "GET /public/themes/forms.css H 158.197.128.175 - - [28/May/1914:21:25:20 +0100] "GET /public/themes/screen.css 158.197.128.175 - - [28/May/1914:21:25:20 +0100] "GET /public/themes/print.css H 158.197.128.175 - - [28/May/1914:21:25:20 +0100] "GET /favicon.ico HTTP/1.1" 200 This time server was down some time, then waked up (ping works), but I was not able to log in or access web pages. I have also backtraces and dump, but I think my dump/bt was recorded after it was wake up. If you think, I can attach them or upload to my server. Look at this very strange year 1914. Did we have Linux in 1914? :-) Any ideas, if I can experiment with clocksource, as described here? https://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.5/html/Virtualization_Guide/chap-Virtualization-KVM_guest_timing_management.html
What's the clocksource you're running? If it's kvmclock, can you switch to something else and check?
I am not sure, I use defaults. Can you tell me how to do this? Do I need to change something in libvirt or only in kernel parameters?
guest dmesg output: Switching to clocksource kvm-clock looks like it's kvm-clock. I also use ntpdate/ntpd to synchronize guest clock to a real clock. Is it enough to add clocksource=acpi_pm to guest kernel parameters or do I need divider or lpj=n also?
Right; you just need to add 'clocksource=acpi_pm' on your guest kernel cmd line. No other changes are necessary.
(In reply to comment #28) > Right; you just need to add 'clocksource=acpi_pm' on your guest kernel cmd > line. I think this clocksource does not work, this is in dmesg: Kernel command line: ro root=UUID=xxx console=tty0 console=ttyS0 clocksource=acpi_pm ... pnp: PnP ACPI init ACPI: bus type pnp registered pnp: PnP ACPI: found 6 devices ACPI: ACPI bus type pnp unregistered Switching to clocksource kvm-clock Override clocksource acpi_pm is not HRT compatible. Cannot switch while in HRT/NOHZ mode Now trying with hpet. Which one is better? hpet, tsc or other?
In that case you may be not able to switch to tsc as well.
(In reply to comment #30) > In that case you may be not able to switch to tsc as well. You don't understand me. I can use tsc, hpet on Fedora 12 and can use acpi_pm on Fedora 11. With hpet and tsc clocksource my machines are running well more than a week. Looks like something is wrong with kvm_clock. Is there any change to fix this?
I think this quote is relevant: http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/51041 "There are three issues with kvmclock due to sampling: 1) smp clock alignment may be slightly off due to timing conditions 2) kvmclock is resampled at each switch of vcpu to another pcpu 3) kvmclock granularity exceeds that of kernel timespec, which means sampling errors may show even on UP Recommend using a different clocksource (tsc is great if you have stable TSC and don't migrate across different-speed machines) until we have all the fixes in place."
(In reply to comment #31) > (In reply to comment #30) > > In that case you may be not able to switch to tsc as well. > > You don't understand me. > > I can use tsc, hpet on Fedora 12 and can use acpi_pm on Fedora 11. OK: you mean acpi_pm doesn't work on F12? That's a separate bug. > With hpet and tsc clocksource my machines are running well more than a week. > > Looks like something is wrong with kvm_clock. Is there any change to fix this? Yes, changes are continuously being pushed to make kvm-clock more reliable. However, can't say when they'll land in Fedora 12.
(In reply to comment #33) > > I can use tsc, hpet on Fedora 12 and can use acpi_pm on Fedora 11. > > OK: you mean acpi_pm doesn't work on F12? That's a separate bug. This message is displayed during boot, when trying to set clocksource=acpi_pm on Fedora 12: Override clocksource acpi_pm is not HRT compatible. Cannot switch while in HRT/NOHZ mode Where I should report this bug? To which component (kernel or qemu)? > > With hpet and tsc clocksource my machines are running well more than a week. > > > > Looks like something is wrong with kvm_clock. Is there any change to fix this? > > Yes, changes are continuously being pushed to make kvm-clock more reliable. > However, can't say when they'll land in Fedora 12. OK, it is at least working well for me. Will them land earlier in Fedora 13?
This message is a reminder that Fedora 12 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 12. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '12'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 12's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 12 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
I am now running tsc or hpet clock sources on all guest machines, so I can't tell, if this still happens. I don't have high-load server on which I can test it. Leaving this bug for close to bugzapper. :-)
Fedora 12 changed to end-of-life (EOL) status on 2010-12-02. Fedora 12 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. Thank you for reporting this bug and we are sorry it could not be fixed.