Bug 1027297 - hosted-engine --vm-shutdown throws an exception when ha agent tries to stop engine vm after maintenance mode on the host was set to "local".
Summary: hosted-engine --vm-shutdown throws an exception when ha agent tries to stop e...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 3.3.0
Assignee: Greg Padgett
QA Contact: Artyom
URL:
Whiteboard: sla
: 854694 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-11-06 13:52 UTC by Leonid Natapov
Modified: 2016-06-12 23:16 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Attempting to call the shutdown() method on a virtual machine which was in a 'Down' state resulted in an exception if called through xmlrpclib. This fix adopts the convention used by similar APIs such as migrate() and returns errCode['noVM'] instead. Virtual machines can be successfully shut down with no exceptions.
Clone Of:
Environment:
Last Closed: 2014-01-21 16:20:30 UTC
oVirt Team: SLA
Target Upstream Version:

Attachments (Terms of Use)
logs (478.09 KB, application/x-gzip)
2013-11-06 13:57 UTC, Leonid Natapov 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:0040 0 normal SHIPPED_LIVE vdsm bug fix and enhancement update 2014-01-21 20:26:21 UTC
oVirt gerrit 21937 0 None None None Never
oVirt gerrit 22218 0 None None None Never

Description Leonid Natapov 2013-11-06 13:52:52 UTC 
hosted-engine --vm-shutdown troughs an exception when trying to stop engine vm after maintenance mode was set to local. 

Scenario:
--------
HA with two hosts. Engine VM runs on host A.
Setting maintenance mode to "local" on host A.
Host A score drops to 0 and ha agent tries to stop engine vm on host A because of low score. Here is the exception that I get in agent.log.
--------
Eventually vm stopped after several retries. 
agent and vdsm logs attached.

MainThread::ERROR::2013-11-06 15:46:03,004::hosted_engine::1077::HostedEngine::(_stop_engine_vm) Failed to stop engine vm with /usr/sbin/hosted-engine --vm-shutdown: Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 2556, in <module>
    code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown
    response = self.s.shutdown(vmId, timeout, message)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">
Comment 1 Leonid Natapov 2013-11-06 13:57:42 UTC 
Created attachment 820386 [details]
logs
Comment 2 Doron Fediuck 2013-11-07 07:25:46 UTC 
This is actually the setup script, so assigning to Sandro.
Comment 3 Sandro Bonazzola 2013-11-07 07:35:22 UTC 
actually hosted-engine --vm-shutdown just call

vdsClient -s localhost shutdown "${vmid}" 120 "VM is shutting down!"

So this is a vdsClient error.
Comment 4 Greg Padgett 2013-11-14 16:05:18 UTC 
Leonid, do you know the vdsm version on which this occurred?
Comment 5 Leonid Natapov 2013-11-17 09:07:28 UTC 
vdsm-4.13.0-0.6.beta1.el6ev.x86_64
Comment 7 Leonid Natapov 2013-11-19 08:32:21 UTC 
Sure,Greg. Will try to reproduce it with the latest vdsm today or tomorrow. I want to use a new official build.
Comment 8 Leonid Natapov 2013-11-24 09:42:46 UTC 
still happening on vdsm-4.13.0-0.8.beta1.el6ev.x86_64.
MainThread::INFO::2013-11-24 11:38:02,533::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: 
MainThread::ERROR::2013-11-24 11:38:02,533::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:38:13,521::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:14,076::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down

MainThread::INFO::2013-11-24 11:38:14,076::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: 
MainThread::ERROR::2013-11-24 11:38:14,077::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:38:25,803::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:26,047::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down

MainThread::INFO::2013-11-24 11:38:26,048::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: 
MainThread::ERROR::2013-11-24 11:38:26,048::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::ERROR::2013-11-24 11:38:41,029::hosted_engine::737::HostedEngine::(_collect_all_host_stats) Host green-vdsc.qa.lab.tlv.redhat.com (id 2) is no longer updating its metadata
MainThread::INFO::2013-11-24 11:38:41,030::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:41,295::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down

MainThread::INFO::2013-11-24 11:38:41,295::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: 
MainThread::ERROR::2013-11-24 11:38:41,295::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:38:51,882::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:52,110::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down

MainThread::INFO::2013-11-24 11:38:52,111::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: 
MainThread::ERROR::2013-11-24 11:38:52,111::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:39:02,626::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:39:02,867::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: 
MainThread::INFO::2013-11-24 11:39:02,868::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 2556, in <module>
    code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown
    response = self.s.shutdown(vmId, timeout, message)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">

MainThread::ERROR::2013-11-24 11:39:02,868::hosted_engine::1120::HostedEngine::(_stop_engine_vm) Failed to stop engine vm with /usr/sbin/hosted-engine --vm-shutdown: Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 2556, in <module>
    code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown
    response = self.s.shutdown(vmId, timeout, message)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">

MainThread::ERROR::2013-11-24 11:39:02,870::hosted_engine::1091::HostedEngine::(_handle_stop) Failed to stop engine VM: Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 2556, in <module>
    code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown
    response = self.s.shutdown(vmId, timeout, message)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">
Comment 9 Michal Skrivanek 2013-12-09 07:28:33 UTC 
*** Bug 854694 has been marked as a duplicate of this bug. ***
Comment 11 Artyom 2013-12-12 12:59:46 UTC 
verified on ovirt-hosted-engine-ha-0.1.0-0.8.rc.el6ev.noarch, vm shut down was success without any exceptions
Comment 12 errata-xmlrpc 2014-01-21 16:20:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html
Note You need to log in before you can comment on or make changes to this bug.