Red Hat Bugzilla – Bug 1027297
hosted-engine --vm-shutdown throws an exception when ha agent tries to stop engine vm after maintenance mode on the host was set to "local".
Last modified: 2016-06-12 19:16:54 EDT
hosted-engine --vm-shutdown troughs an exception when trying to stop engine vm after maintenance mode was set to local. Scenario: -------- HA with two hosts. Engine VM runs on host A. Setting maintenance mode to "local" on host A. Host A score drops to 0 and ha agent tries to stop engine vm on host A because of low score. Here is the exception that I get in agent.log. -------- Eventually vm stopped after several retries. agent and vdsm logs attached. MainThread::ERROR::2013-11-06 15:46:03,004::hosted_engine::1077::HostedEngine::(_stop_engine_vm) Failed to stop engine vm with /usr/sbin/hosted-engine --vm-shutdown: Traceback (most recent call last): File "/usr/share/vdsm/vdsClient.py", line 2556, in <module> code, message = commands[command][0](commandArgs) File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown response = self.s.shutdown(vmId, timeout, message) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request return self._parse_response(h.getfile(), sock) File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response return u.close() File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">
Created attachment 820386 [details] logs
This is actually the setup script, so assigning to Sandro.
actually hosted-engine --vm-shutdown just call vdsClient -s localhost shutdown "${vmid}" 120 "VM is shutting down!" So this is a vdsClient error.
Leonid, do you know the vdsm version on which this occurred?
vdsm-4.13.0-0.6.beta1.el6ev.x86_64
Sure,Greg. Will try to reproduce it with the latest vdsm today or tomorrow. I want to use a new official build.
still happening on vdsm-4.13.0-0.8.beta1.el6ev.x86_64. MainThread::INFO::2013-11-24 11:38:02,533::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: MainThread::ERROR::2013-11-24 11:38:02,533::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2013-11-24 11:38:13,521::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown` MainThread::INFO::2013-11-24 11:38:14,076::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down MainThread::INFO::2013-11-24 11:38:14,076::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: MainThread::ERROR::2013-11-24 11:38:14,077::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2013-11-24 11:38:25,803::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown` MainThread::INFO::2013-11-24 11:38:26,047::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down MainThread::INFO::2013-11-24 11:38:26,048::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: MainThread::ERROR::2013-11-24 11:38:26,048::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::ERROR::2013-11-24 11:38:41,029::hosted_engine::737::HostedEngine::(_collect_all_host_stats) Host green-vdsc.qa.lab.tlv.redhat.com (id 2) is no longer updating its metadata MainThread::INFO::2013-11-24 11:38:41,030::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown` MainThread::INFO::2013-11-24 11:38:41,295::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down MainThread::INFO::2013-11-24 11:38:41,295::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: MainThread::ERROR::2013-11-24 11:38:41,295::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2013-11-24 11:38:51,882::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown` MainThread::INFO::2013-11-24 11:38:52,110::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down MainThread::INFO::2013-11-24 11:38:52,111::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: MainThread::ERROR::2013-11-24 11:38:52,111::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2013-11-24 11:39:02,626::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown` MainThread::INFO::2013-11-24 11:39:02,867::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: MainThread::INFO::2013-11-24 11:39:02,868::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: Traceback (most recent call last): File "/usr/share/vdsm/vdsClient.py", line 2556, in <module> code, message = commands[command][0](commandArgs) File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown response = self.s.shutdown(vmId, timeout, message) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request return self._parse_response(h.getfile(), sock) File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response return u.close() File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled"> MainThread::ERROR::2013-11-24 11:39:02,868::hosted_engine::1120::HostedEngine::(_stop_engine_vm) Failed to stop engine vm with /usr/sbin/hosted-engine --vm-shutdown: Traceback (most recent call last): File "/usr/share/vdsm/vdsClient.py", line 2556, in <module> code, message = commands[command][0](commandArgs) File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown response = self.s.shutdown(vmId, timeout, message) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request return self._parse_response(h.getfile(), sock) File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response return u.close() File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled"> MainThread::ERROR::2013-11-24 11:39:02,870::hosted_engine::1091::HostedEngine::(_handle_stop) Failed to stop engine VM: Traceback (most recent call last): File "/usr/share/vdsm/vdsClient.py", line 2556, in <module> code, message = commands[command][0](commandArgs) File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown response = self.s.shutdown(vmId, timeout, message) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request return self._parse_response(h.getfile(), sock) File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response return u.close() File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">
*** Bug 854694 has been marked as a duplicate of this bug. ***
verified on ovirt-hosted-engine-ha-0.1.0-0.8.rc.el6ev.noarch, vm shut down was success without any exceptions
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0040.html