Bug 634098 - catch and report qemu (and libvirt) errors to upper mgmt. layers
catch and report qemu (and libvirt) errors to upper mgmt. layers
Status: CLOSED WONTFIX
Product: oVirt
Classification: Community
Component: vdsm (Show other bugs)
unspecified
All Linux
low Severity medium
: ---
: ---
Assigned To: Dan Kenigsberg
infra
: Improvement
: 510421 (view as bug list)
Depends On:
Blocks: 739364
  Show dependency treegraph
 
Reported: 2010-09-15 04:48 EDT by Yaniv Kaul
Modified: 2013-07-03 22:06 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-03-12 12:00:51 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yaniv Kaul 2010-09-15 04:48:22 EDT
Description of problem:
Since VDSM provides no protection today against running VMs when not enough memory is available, it'll be nice to report a more accurate error to RHEVM than 'Unexpected Create Error'.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

From VDSM.log:

Thread-152965::DEBUG::2010-09-15 08:46:46,614::clientIF::642::vds::Total desktops after creation of 80bee098-6074-4e7c-90c7-5cf0e6d2061c is 9
Thread-152966::DEBUG::2010-09-15 08:46:46,614::clientIF::1011::vds::80bee098-6074-4e7c-90c7-5cf0e6d2061c: memAvailable -2487 memRequired 2113 Mb

...


Thread-152966::DEBUG::2010-09-15 08:46:48,986::vm::769::vds.vmlog.80bee098-6074-4e7c-90c7-5cf0e6d2061c::/usr/bin/sudo /usr/bin/tunctl -d virtio_17_1; /usr/bin/sudo /usr/bin/tunctl -b -u vdsm -t virtio_17_1;/
usr/bin/sudo /sbin/ip link set dev virtio_17_1 address fe:1a:4a:16:9a:30;/usr/bin/sudo /sbin/ip link set dev virtio_17_1 up;/usr/bin/sudo /usr/sbin/brctl addif rhevm virtio_17_1;TZ=UTC /bin/nice -n 5 /usr/li
bexec/qemu-kvm -no-hpet -usbdevice tablet  -rtc-td-hack  -startdate 2010-09-15T01:39:21  -name AL2008-v23  -smp 1,cores=1  -k en-us  -m 2048  -boot c  -net nic,vlan=1,macaddr=00:1a:4a:16:9a:30,model=virtio -
net tap,vlan=1,ifname=virtio_17_1,script=no  -drive file=/rhev/data-center/4663cd2e-8ec3-4f8b-a185-6377ec334ebf/16923b87-4229-43e3-b571-ad141a112c68/images/b62a11aa-615c-4ef1-b28c-dc089a27d67f/394bf1d5-7938-
48a7-be9e-4db450f7d48e,media=disk,if=virtio,cache=off,serial=f1-b28c-dc089a27d67f,boot=on,format=qcow2,werror=stop -pidfile /var/vdsm/80bee098-6074-4e7c-90c7-5cf0e6d2061c.pid -soundhw ac97  -spice sslpasswor
d=,sslciphersuite=DEFAULT,sslcert=/var/vdsm/ts/certs/vdsmcert.pem,sslkey=/var/vdsm/ts/keys/vdsmkey.pem,ssldhfile=/var/vdsm/ts/keys/dh.pem,sslcafile=/var/vdsm/ts/certs/cacert.pem,host=10.35.101.17,secure-chan
nels=main+inputs,ic=on,sport=5883,port=5917 -qxl 1  -incoming tcp::49162 -cpu qemu64,+sse2  -M rhel5.5.0  -notify all  -balloon none  -smbios type=1,manufacturer="Red Hat",product="RHEV Hypervisor",version=5
.5-2.2-5.1,serial="34373030-3634-4742-3838-3033384E5943_00:1c:c4:79:cf:98",uuid="80bee098-6074-4e7c-90c7-5cf0e6d2061c"  -vmchannel di:0200,unix:/var/vdsm/80bee098-6074-4e7c-90c7-5cf0e6d2061c.guest.socket,ser
ver -monitor unix:/var/vdsm/80bee098-6074-4e7c-90c7-5cf0e6d2061c.monitor.socket,server 1>/var/vdsm/80bee098-6074-4e7c-90c7-5cf0e6d2061c.stdio.dump 2>&1; /usr/bin/sudo /usr/bin/tunctl -d virtio_17_1;
Thread-152966::ERROR::2010-09-15 08:46:50,088::vm::334::vds.vmlog.80bee098-6074-4e7c-90c7-5cf0e6d2061c::Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 305, in _execqemu
  File "/usr/share/vdsm/vm.py", line 813, in _run
  File "/usr/share/vdsm/vm.py", line 726, in _initQemuMonitor
  File "/usr/share/vdsm/QemuMonitor.py", line 251, in __init__
  File "/usr/share/vdsm/QemuMonitor.py", line 79, in __init__
StoppedError

Thread-152966::DEBUG::2010-09-15 08:46:50,088::vm::1434::vds.vmlog.80bee098-6074-4e7c-90c7-5cf0e6d2061c::Changed state to Down: Unexpected Create Error
Thread-152972::DEBUG::2010-09-15 08:46:50,392::vm::1309::vds.vmlog.80bee098-6074-4e7c-90c7-5cf0e6d2061c::destroy Called
Thread-152972::DEBUG::2010-09-15 08:46:50,398::vm::1336::vds.vmlog.80bee098-6074-4e7c-90c7-5cf0e6d2061c::Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 1332, in destroy
OSError: [Errno 3] No such process

Thread-152972::DEBUG::2010-09-15 08:46:50,465::ksm::101::vds::/usr/bin/ksmctl start 512 2727: committed 22020096000, free 698773504, total 12596977664
Thread-152972::DEBUG::2010-09-15 08:46:50,506::vm::1361::vds.vmlog.80bee098-6074-4e7c-90c7-5cf0e6d2061c::Total desktops after destroy of 80bee098-6074-4e7c-90c7-5cf0e6d2061c is 8
Thread-152972::DEBUG::2010-09-15 08:46:50,506::vm::1365::vds.vmlog.80bee098-6074-4e7c-90c7-5cf0e6d2061c::qemu stdouterr: Could not allocate physical memory
Comment 1 Dan Kenigsberg 2010-09-21 18:33:45 EDT
Yaniv, I'd appreciate if you reproduce this on RHEL-6 and attach the logs. I suspect it would not be easy to extract failure reason from libvirt until we have qerror.
Comment 2 Dan Kenigsberg 2010-09-27 05:41:29 EDT
Yep, in RHEL-6 it is hidden by qemu and libvirt. If you really want it, we need 2 more bugs for these layers, and it certainly won't happen for RHEV 2.3.

Thread-21::ERROR::2010-09-27 11:24:14,867::vm::635::vds.vmlog.9ffe28b6-6134-4b1e-8804-1185f49c436f::(_startUnderlyingVm) Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 605, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/libvirtvm.py", line 651, in _run
    self._connection.createXML(domxml, flags),
  File "/usr/share/vdsm/libvirtvm.py", line 982, in wrapper
    raise e
libvirtError: internal error process exited while connecting to monitor: do_spice_init: starting 0.4.2-15..el6
interface_change_notifier: VD_INTERFACE_KEYBOARD
interface_change_notifier: VD_INTERFACE_MOUSE
Comment 3 Yaniv Kaul 2011-03-15 11:41:44 EDT
(In reply to comment #2)
> Yep, in RHEL-6 it is hidden by qemu and libvirt. If you really want it, we need
> 2 more bugs for these layers, and it certainly won't happen for RHEV 2.3.

I guess we might need it. Any other errors they hide which might be relevant for us?

> 
> Thread-21::ERROR::2010-09-27
> 11:24:14,867::vm::635::vds.vmlog.9ffe28b6-6134-4b1e-8804-1185f49c436f::(_startUnderlyingVm)
> Traceback (most recent call last):
>   File "/usr/share/vdsm/vm.py", line 605, in _startUnderlyingVm
>     self._run()
>   File "/usr/share/vdsm/libvirtvm.py", line 651, in _run
>     self._connection.createXML(domxml, flags),
>   File "/usr/share/vdsm/libvirtvm.py", line 982, in wrapper
>     raise e
> libvirtError: internal error process exited while connecting to monitor:
> do_spice_init: starting 0.4.2-15..el6
> interface_change_notifier: VD_INTERFACE_KEYBOARD
> interface_change_notifier: VD_INTERFACE_MOUSE
Comment 4 Dan Kenigsberg 2011-03-15 14:14:49 EDT
(In reply to comment #3)
> (In reply to comment #2)
> > Yep, in RHEL-6 it is hidden by qemu and libvirt. If you really want it, we need
> > 2 more bugs for these layers, and it certainly won't happen for RHEV 2.3.
> 
> I guess we might need it. Any other errors they hide which might be relevant
> for us?

It depends how much you care, and how much you understand qemu. If the answer is "allot", any qemu error could be relevant - we could be interested in the whole traceback.
Comment 5 Dan Kenigsberg 2011-09-17 06:22:06 EDT
*** Bug 510421 has been marked as a duplicate of this bug. ***
Comment 6 Dan Kenigsberg 2011-09-17 06:31:28 EDT
Eli, is there a rhev-m bug about receiving verbose traceback? Please make it depend on us.
Comment 7 Eli Mesika 2011-09-18 03:53:13 EDT
(In reply to comment #6)
> Eli, is there a rhev-m bug about receiving verbose traceback? Please make it
> depend on us.

Done
Comment 8 Itamar Heim 2013-03-12 12:00:51 EDT
Closing old bugs. If this issue is still relevant/important in current version, please re-open the bug.

Note You need to log in before you can comment on or make changes to this bug.