| Summary: | hosted-engine --vm-shutdown throws an exception when ha agent tries to stop engine vm after maintenance mode on the host was set to "local". | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Leonid Natapov <lnatapov> | ||||
| Component: | vdsm | Assignee: | Greg Padgett <gpadgett> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Artyom <alukiano> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 3.3.0 | CC: | acathrow, amureini, bazulay, dfediuck, fsimonce, gpadgett, iheim, knesenko, lnatapov, lpeer, mavital, msivak, oourfali, sbonazzo, yeylon | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 3.3.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | sla | ||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: |
Attempting to call the shutdown() method on a virtual machine which was in a 'Down' state resulted in an exception if called through xmlrpclib. This fix adopts the convention used by similar APIs such as migrate() and returns errCode['noVM'] instead. Virtual machines can be successfully shut down with no exceptions.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-01-21 16:20:30 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | SLA | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Attachments: |
|
||||||
Created attachment 820386 [details]
logs
This is actually the setup script, so assigning to Sandro. actually hosted-engine --vm-shutdown just call
vdsClient -s localhost shutdown "${vmid}" 120 "VM is shutting down!"
So this is a vdsClient error.
Leonid, do you know the vdsm version on which this occurred? vdsm-4.13.0-0.6.beta1.el6ev.x86_64 Sure,Greg. Will try to reproduce it with the latest vdsm today or tomorrow. I want to use a new official build. still happening on vdsm-4.13.0-0.8.beta1.el6ev.x86_64.
MainThread::INFO::2013-11-24 11:38:02,533::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr:
MainThread::ERROR::2013-11-24 11:38:02,533::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:38:13,521::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:14,076::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down
MainThread::INFO::2013-11-24 11:38:14,076::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr:
MainThread::ERROR::2013-11-24 11:38:14,077::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:38:25,803::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:26,047::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down
MainThread::INFO::2013-11-24 11:38:26,048::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr:
MainThread::ERROR::2013-11-24 11:38:26,048::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::ERROR::2013-11-24 11:38:41,029::hosted_engine::737::HostedEngine::(_collect_all_host_stats) Host green-vdsc.qa.lab.tlv.redhat.com (id 2) is no longer updating its metadata
MainThread::INFO::2013-11-24 11:38:41,030::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:41,295::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down
MainThread::INFO::2013-11-24 11:38:41,295::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr:
MainThread::ERROR::2013-11-24 11:38:41,295::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:38:51,882::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:38:52,110::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout: Machine shut down
MainThread::INFO::2013-11-24 11:38:52,111::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr:
MainThread::ERROR::2013-11-24 11:38:52,111::hosted_engine::1123::HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost
MainThread::INFO::2013-11-24 11:39:02,626::hosted_engine::1110::HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown`
MainThread::INFO::2013-11-24 11:39:02,867::hosted_engine::1114::HostedEngine::(_stop_engine_vm) stdout:
MainThread::INFO::2013-11-24 11:39:02,868::hosted_engine::1115::HostedEngine::(_stop_engine_vm) stderr: Traceback (most recent call last):
File "/usr/share/vdsm/vdsClient.py", line 2556, in <module>
code, message = commands[command][0](commandArgs)
File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown
response = self.s.shutdown(vmId, timeout, message)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
verbose=self.__verbose
File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
return self._parse_response(h.getfile(), sock)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
return u.close()
File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">
MainThread::ERROR::2013-11-24 11:39:02,868::hosted_engine::1120::HostedEngine::(_stop_engine_vm) Failed to stop engine vm with /usr/sbin/hosted-engine --vm-shutdown: Traceback (most recent call last):
File "/usr/share/vdsm/vdsClient.py", line 2556, in <module>
code, message = commands[command][0](commandArgs)
File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown
response = self.s.shutdown(vmId, timeout, message)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
verbose=self.__verbose
File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
return self._parse_response(h.getfile(), sock)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
return u.close()
File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">
MainThread::ERROR::2013-11-24 11:39:02,870::hosted_engine::1091::HostedEngine::(_handle_stop) Failed to stop engine VM: Traceback (most recent call last):
File "/usr/share/vdsm/vdsClient.py", line 2556, in <module>
code, message = commands[command][0](commandArgs)
File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown
response = self.s.shutdown(vmId, timeout, message)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
verbose=self.__verbose
File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
return self._parse_response(h.getfile(), sock)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
return u.close()
File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">
*** Bug 854694 has been marked as a duplicate of this bug. *** verified on ovirt-hosted-engine-ha-0.1.0-0.8.rc.el6ev.noarch, vm shut down was success without any exceptions Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0040.html |
hosted-engine --vm-shutdown troughs an exception when trying to stop engine vm after maintenance mode was set to local. Scenario: -------- HA with two hosts. Engine VM runs on host A. Setting maintenance mode to "local" on host A. Host A score drops to 0 and ha agent tries to stop engine vm on host A because of low score. Here is the exception that I get in agent.log. -------- Eventually vm stopped after several retries. agent and vdsm logs attached. MainThread::ERROR::2013-11-06 15:46:03,004::hosted_engine::1077::HostedEngine::(_stop_engine_vm) Failed to stop engine vm with /usr/sbin/hosted-engine --vm-shutdown: Traceback (most recent call last): File "/usr/share/vdsm/vdsClient.py", line 2556, in <module> code, message = commands[command][0](commandArgs) File "/usr/share/vdsm/vdsClient.py", line 340, in do_shutdown response = self.s.shutdown(vmId, timeout, message) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request return self._parse_response(h.getfile(), sock) File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response return u.close() File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">