Bug 1027297 - hosted-engine --vm-shutdown throws an exception when ha agent tries to stop engine vm after maintenance mode on the host was set to "local".
hosted-engine --vm-shutdown throws an exception when ha agent tries to stop e...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
3.3.0
Unspecified Unspecified
urgent Severity high
: ---
: 3.3.0
Assigned To: Greg Padgett
Artyom
sla
:
: 854694 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-06 08:52 EST by Leonid Natapov
Modified: 2016-06-12 19:16 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Attempting to call the shutdown() method on a virtual machine which was in a 'Down' state resulted in an exception if called through xmlrpclib. This fix adopts the convention used by similar APIs such as migrate() and returns errCode['noVM'] instead. Virtual machines can be successfully shut down with no exceptions.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-21 11:20:30 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: SLA
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs (478.09 KB, application/x-gzip)
2013-11-06 08:57 EST, Leonid Natapov
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 21937 None None None Never
oVirt gerrit 22218 None None None Never

  None (edit)
Description Leonid Natapov 2013-11-06 08:52:52 EST
hosted-engine --vm-shutdown troughs an exception when trying to stop engine vm after maintenance mode was set to local. 

Scenario:
--------
HA with two hosts. Engine VM runs on host A.
Setting maintenance mode to "local" on host A.
Host A score drops to 0 and ha agent tries to stop engine vm on host A because of low score. Here is the exception that I get in agent.log.
--------
Eventually vm stopped after several retries. 
agent and vdsm logs attached.

MainThread::ERROR::2013-11-06 15:46:03,004::hosted_engine::1077::HostedEngine::(_stop_engine_vm) Failed to stop engine vm with /usr/sbin/hosted-engine --vm-shutdown: Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 2556, in <module>
    code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown
    response = self.s.shutdown(vmId, timeout, message)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">
Comment 1 Leonid Natapov 2013-11-06 08:57:42 EST
Created attachment 820386 [details]
logs
Comment 2 Doron Fediuck 2013-11-07 02:25:46 EST
This is actually the setup script, so assigning to Sandro.
Comment 3 Sandro Bonazzola 2013-11-07 02:35:22 EST
actually hosted-engine --vm-shutdown just call

vdsClient -s localhost shutdown "${vmid}" 120 "VM is shutting down!"

So this is a vdsClient error.
Comment 4 Greg Padgett 2013-11-14 11:05:18 EST
Leonid, do you know the vdsm version on which this occurred?
Comment 5 Leonid Natapov 2013-11-17 04:07:28 EST
vdsm-4.13.0-0.6.beta1.el6ev.x86_64
Comment 7 Leonid Natapov 2013-11-19 03:32:21 EST
Sure,Greg. Will try to reproduce it with the latest vdsm today or tomorrow. I want to use a new official build.
Comment 8 Leonid Natapov 2013-11-24 04:42:46 EST
still happening on vdsm-4.13.0-0.8.beta1.el6ev.x86_64.
MainThread::INFO::2013-11-24 11:38:02,533::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: 
MainThread::ERROR::2013-11-24 11:38:02,533::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:38:13,521::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:14,076::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down

MainThread::INFO::2013-11-24 11:38:14,076::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: 
MainThread::ERROR::2013-11-24 11:38:14,077::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:38:25,803::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:26,047::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down

MainThread::INFO::2013-11-24 11:38:26,048::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: 
MainThread::ERROR::2013-11-24 11:38:26,048::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::ERROR::2013-11-24 11:38:41,029::hosted_engine::737::HostedEngine::(_collect_all_host_stats) Host green-vdsc.qa.lab.tlv.redhat.com (id 2) is no longer updating its metadata
MainThread::INFO::2013-11-24 11:38:41,030::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:41,295::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down

MainThread::INFO::2013-11-24 11:38:41,295::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: 
MainThread::ERROR::2013-11-24 11:38:41,295::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:38:51,882::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:52,110::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down

MainThread::INFO::2013-11-24 11:38:52,111::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: 
MainThread::ERROR::2013-11-24 11:38:52,111::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:39:02,626::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:39:02,867::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: 
MainThread::INFO::2013-11-24 11:39:02,868::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 2556, in <module>
    code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown
    response = self.s.shutdown(vmId, timeout, message)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">

MainThread::ERROR::2013-11-24 11:39:02,868::hosted_engine::1120::HostedEngine::(_stop_engine_vm) Failed to stop engine vm with /usr/sbin/hosted-engine --vm-shutdown: Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 2556, in <module>
    code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown
    response = self.s.shutdown(vmId, timeout, message)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">

MainThread::ERROR::2013-11-24 11:39:02,870::hosted_engine::1091::HostedEngine::(_handle_stop) Failed to stop engine VM: Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 2556, in <module>
    code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown
    response = self.s.shutdown(vmId, timeout, message)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">
Comment 9 Michal Skrivanek 2013-12-09 02:28:33 EST
*** Bug 854694 has been marked as a duplicate of this bug. ***
Comment 11 Artyom 2013-12-12 07:59:46 EST
verified on ovirt-hosted-engine-ha-0.1.0-0.8.rc.el6ev.noarch, vm shut down was success without any exceptions
Comment 12 errata-xmlrpc 2014-01-21 11:20:30 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html

Note You need to log in before you can comment on or make changes to this bug.