Bug 741217
| Summary: | KVM guest install fails with libvirtError: cannot send monitor command '{"execute":"query-balloon"}': Connection reset by peer | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Kristijan <kbesu> | ||||
| Component: | libvirt | Assignee: | Jiri Denemark <jdenemar> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 6.1 | CC: | acathrow, dallan, jdenemar, jeder, jwest, linuxdev-ofuku, mjenner, mmahudha, moshiro, msvoboda, myamazak, mzhan, rwu, weizhan | ||||
| Target Milestone: | rc | Keywords: | ZStream | ||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | libvirt-0.9.4-16.el6 | Doc Type: | Bug Fix | ||||
| Doc Text: |
If the QEMU driver failed to update information about currently allocated memory, installing a new virtual machine could have failed with the following error message:
ERROR cannot send monitor command '{"execute":"query-balloon"}':
Connection reset by peer
With this update, the driver has been modified to not consider this behavior as fatal. Installation now proceeds and finishes as expected.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2011-12-06 11:34:32 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 708393, 747120, 747358 | ||||||
| Attachments: |
|
||||||
The error
ERROR cannot send monitor command '{"execute":"query-balloon"}':
Connection reset by peer
most likely means qemu-kvm process crashed. Can you send us /var/log/libvirt/qemu/csms02.zone31.log file, which contains the output of qemu-kvm process started for domain "csms02.zone31"?
Created attachment 524879 [details]
file
Please find csms02.zone31.log file in attachment That's interesting, there is nothing about any kind of failure in the qemu log. It just starts and after a few minutes (looks like not more than 4) it normally shuts down.
Also from virt-install output
Starting install...
Creating domain...
| 0 B 00:00
Domain installation still in progress. Waiting for installation to complete.
Domain has shutdown. Continuing.
Starting domain...
| 0 B 00:00
Domain installation still in progress. Waiting for installation to complete.
ERROR cannot send monitor command '{"execute":"query-balloon"}': Connection
reset by peer
it seems the domain installation was started, virt-install was waiting for reboot and wanted to start the installed domain, which failed. It looks like the "Connection reset by peer" may be caused by qemu shutting down early (and not crashing as I had thought) perhaps because no bootable disk was found. Do you have any idea what happened inside the guest OS between "Domain installation still in progress." and "Domain has shutdown"? Perhaps something during guest OS installation failed and it was automatically rebooted (from inside, by the install process), which made virt-install think installation is complete. Can you check that?
BTW, I'm a bit confused with the second occurrence of
| 0 B 00:00
Domain installation still in progress. Waiting for installation to complete.
after "Starting domain..." was it really there or is that just a copy&paste mistake?
Anyway, can you reproduce the issue with libvirt debug logs enabled and send us the libvirtd.log file? To enable libvirt debug logs, set
log_filters="1:qemu"
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
in /etc/libvirt/libvirtd.conf and restart libvirtd service.
Ok, I accidentally reproduced similar issue with virt-manager and understood what's going on here. The monitor error doesn't come from starting an already installed domain. It happens at the end of installation but not as a result of qemu crash. During the installation phase, virt-install polls every two seconds for domain shutdown using virDomainGetInfo, which tries to update balloon info. If virDomainGetInfo is called just at the moment when the domain is shutting down, the query-baloon monitor command may fail because qemu just exits instead of giving us any reply. Virt-install doesn't tolerate the error coming from virDomainGetInfo and aborts. Similar thing happens to virt-manager. This is now fixed upstream by
commit fcd2bd55d714b8e939c5bfd44fd2ddc32a09c141
Author: Jiri Denemark <jdenemar>
Date: Wed Oct 5 16:07:36 2011 +0200
qemu: Don't fail virDomainGetInfo if we can't update balloon info
Qemu driver tries to update balloon data in virDomainGetInfo and if it
can't do so because there is another monitor job running, it just
reports what's known in domain def. However, if there was no job running
but getting the data from qemu fails, we would fail the whole API. This
doesn't make sense. Let's make the failure nonfatal.
verify pass on kernel-2.6.32-206.el6.x86_64 qemu-kvm-0.12.1.2-2.195.el6.x86_64 libvirt-0.9.4-16.el6.x86_64 According to comment 10, reproduce on rhel6.1 with steps 1. start a win2003-x64 guest 2. on console 1, do # while true; do virsh dominfo win2008r2-x64 >> /tmp/dominfo_log ; done 3. on console 2, do # virsh shutdown win2003-x64 at the end of guest shutdown, on console 1, there will be an error line output: error: cannot send monitor command '{"execute":"query-balloon"}': Connection reset by peer on libvirt-0.9.4-16.el6.x86_64, there is no error, so verify pass
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
If the QEMU driver failed to update information about currently allocated memory, installing a new virtual machine could have failed with the following error message:
ERROR cannot send monitor command '{"execute":"query-balloon"}':
Connection reset by peer
With this update, the driver has been modified to not consider this behavior as fatal. Installation now proceeds and finishes as expected.
*** Bug 747555 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2011-1513.html |
Description of problem: installation of Windows guest failed. I've tried to install Windows 2003 virtual machine and got the following error from "virt-install": /usr/bin/virt-install --cdrom=/WINDOWS2003SP2_KVM_E08.00.00.01.iso -r 1000 -n csms02.zone31 --network bridge=br0,model=virtio --disk path=/dev/VX/csms02.zone31hddC,cache=writeback,bus=virtio --disk path=/dev/VX/csms02.zone31_setup,cache=writeback,bus=virtio --os-variant=win2k3 --vcpus=1,maxvcpus=1 --cpuset=7 --cpu=host --noautoconsole --graphics vnc Starting install... Creating domain... | 0 B 00:00 Domain installation still in progress. Waiting for installation to complete. Domain has shutdown. Continuing. Starting domain... | 0 B 00:00 Domain installation still in progress. Waiting for installation to complete. ERROR cannot send monitor command '{"execute":"query-balloon"}': Connection reset by peer Domain installation does not appear to have been successful. If it was, you can restart your domain by running: virsh --connect qemu:///system start csms02.zone31 otherwise, please restart your installation. ERROR: virt-install \ --cdrom=/WINDOWS2003SP2_KVM_E08.00.00.01.iso \ -r 1000 \ -n csms02.zone31 \ --network bridge=br0,model=virtio \ --disk path=/dev/VX/csms02.zone31hddC,cache=writeback,bus=virtio --disk path=/dev/VX/csms02.zone31_setup,cache=writeback,bus=virtio \ --os-variant=win2k3 \ --vcpus=1,maxvcpus=1 \ --cpuset=7\ --cpu=host \ --noautoconsole \ --graphics vnc failed for csms02.zone31. Version-Release number of selected component (if applicable): libvirt-client-0.8.7-18.el6.x86_64 Linux 2.6.32-131.6.1.el6.x86_64 #1 SMP Mon Jun 20 14:15:38 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux How reproducible: Executing following command /usr/bin/virt-install --cdrom=/WINDOWS2003SP2_KVM_E08.00.00.01.iso -r 1000 -n csms02.zone31 --network bridge=br0,model=virtio --disk path=/dev/VX/csms02.zone31hddC,cache=writeback,bus=virtio --disk path=/dev/VX/csms02.zone31_setup,cache=writeback,bus=virtio --os-variant=win2k3 --vcpus=1,maxvcpus=1 --cpuset=7 --cpu=host --noautoconsole --graphics vnc Actual results: Installation of Windows 2003 guest fails. Expected results: Installation of Windows 2003 guest should pass without problems. Additional info: 1. We first thought that the issue is related with memory, Customer has stop some guests running on the system, or clear the caches: echo 1 > /proc/sys/vm/drop_caches did not help 2. Customer has saw the same issue when installing Linux guests (however only 2 times) They haven't tried to install using virt-manager/virt-viewer as they are preparing un-attanded (silent) installation of Windows so they need to rely on "virt-install" (using GUI for installation of guests is not an option in their situation) 3. WINDOWS ISO which they're using for the installation of Windows guests, already includes in the image "virtio" drivers both for HDD and network (virtio drivers were included from /usr/share/virtio-win/virtio-win-1.2.0.iso delivered with virtio-win-1.2.0-1.el6.noarch.rpm). Even if they installed Windows guest without support for virtio (for HDD bus=ide, for network devices type=e1000 or "rtl8139") the installation crashed after several iterations (please see examples below) 4. after installation of Windows machine (which crashed with '{"execute":"query-balloon"}' message) directory /var/lib/libvirt/qemu/save is empty, so there was not .save file to remove. 5. Adding the following to Customer's /etc/libvirt/qemu.conf Locate this in your conf file: #cgroup_device_acl = [ # "/dev/null", "/dev/full", "/dev/zero", # "/dev/random", "/dev/urandom", # "/dev/ptmx", "/dev/kvm", "/dev/kqemu", # "/dev/rtc", "/dev/hpet", "/dev/net/tun", #] Then insert the following line: cgroup_controllers = [ ] Afterwards restart the libvirt service and try the install once more. Didn't help 6. specifying --os-variant but not --os-type, the virt-install man page says --os-type is not required, but perhaps it is. also didn't help.