Bug 1311177

Summary: implementation of soft shutdown on vmware hypervisors
Product: Red Hat OpenStack Reporter: GE Scott Knauss <sknauss>
Component: openstack-novaAssignee: Eoghan Glynn <eglynn>
Status: CLOSED DEFERRED QA Contact: nlevinki <nlevinki>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.0 (Juno)CC: berrange, dasmith, eglynn, kchamart, mbooth, ndipanov, sbauza, sferdjao, sgordon, sknauss, vromanso, yeylon
Target Milestone: ---Keywords: ZStream
Target Release: 6.0 (Juno)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Openstack 6.0 with vmware hypervisors
Last Closed: 2016-04-01 16:30:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description GE Scott Knauss 2016-02-23 14:46:56 UTC
What problem/issue/behavior are you having trouble with?  What do you expect to see?

nova stop does not use the vmware client for shutdown the vm

We implemented the soft shutdown, using the vmware client, for nova stop.

could your please look at the following diffs and implement this in OSP 6? thanks

diff --git a/nova/virt/vmwareapi/driver.py b/nova/virt/vmwareapi/driver.py
index 82fabee..46abaed 100644
--- a/nova/virt/vmwareapi/driver.py
+++ b/nova/virt/vmwareapi/driver.py
@@ -513,7 +513,7 @@ class VMwareVCDriver(driver.ComputeDriver):
         """Power off the specified instance."""
         # TODO(PhilDay): Add support for timeout (clean shutdown)
         LOG.info("Shutdown timeout: %s, Retry interval: %s", timeout, retry_interval)
-        self._vmops.power_off(instance)
+        self._vmops.power_off(instance, timeout, retry_interval)

     def power_on(self, context, instance, network_info,
                  block_device_info=None):

diff --git a/nova/virt/vmwareapi/vmops.py b/nova/virt/vmwareapi/vmops.py
index db820b5..b3b8afd 100644
--- a/nova/virt/vmwareapi/vmops.py
+++ b/nova/virt/vmwareapi/vmops.py
@@ -64,6 +64,8 @@ CONF.import_opt('my_ip', 'nova.netconf')

 LOG = logging.getLogger(__name__)

+SHUTDOWN_TIME_INCREMENT = 5
+
 VMWARE_POWER_STATES = {
                    'poweredOff': power_state.SHUTDOWN,
                     'poweredOn': power_state.RUNNING,
@@ -974,13 +976,78 @@ class VMwareVMOps(object):
         if power_on:
             vm_util.power_on_instance(self._session, instance, vm_ref=vm_ref)

-    def power_off(self, instance):
+    def power_off(self, instance, timeout=0, retry_interval=0):
         """Power off the specified instance.

         :param instance: nova.objects.instance.Instance
         """
+
+        if retry_interval <= 0:
+            retry_interval = SHUTDOWN_TIME_INCREMENT
+
+        if timeout and self._soft_shutdown(instance,
+                                           timeout,
+                                           retry_interval):
+            return
+
+        LOG.info("Performing Hard shutdown on instance", instance=instance)
         vm_util.power_off_instance(self._session, instance)

+    def _soft_shutdown(self, instance,
+                       timeout=60,
+                       retry_interval=SHUTDOWN_TIME_INCREMENT):
+        """Perform a soft shutdown on the VM.
+           :return: True if the instance was shutdown within time limit,
+           False otherwise.
+        """
+        LOG.info("Performing Soft shutdown on instance", instance=instance)
+        vm_ref = vm_util.get_vm_ref(self._session, instance)
+        lst_properties = ["summary.guest.toolsStatus", "runtime.powerState",
+                          "summary.guest.toolsRunningStatus"]
+        props = self._session._call_method(vim_util, "get_object_properties",
+                           None, vm_ref, "VirtualMachine",
+                           lst_properties)
+        query = vm_util.get_values_from_object_properties(self._session, props)
+        pwr_state = query['runtime.powerState']
+        tools_status = query['summary.guest.toolsStatus']
+        tools_running_status = query['summary.guest.toolsRunningStatus']
+
+        if pwr_state not in ["poweredOn"]:
+            return False
+
+        if (tools_status == "toolsOk" and
+                tools_running_status == "guestToolsRunning"):
+
+            LOG.info("Soft shutdown instance, timeout: %d",
+                    timeout, instance=instance)
+            self._session._call_method(self._session._get_vim(), "ShutdownGuest",
+                    vm_ref)
+
+            while timeout > 0:
+                wait_time = min(retry_interval, timeout)
+                try:
+                    props = self._session._call_method(vim_util, "get_object_properties",
+                            None, vm_ref, "VirtualMachine",
+                            lst_properties)
+                    query = vm_util.get_values_from_object_properties(self._session, props)
+                    if query['runtime.powerState'] in ["poweredOff"]:
+                        LOG.info("Soft shutdown succeeded.",
+                                  instance=instance)
+                        return True
+                except Exception as e:
+                    LOG.info("Soft shutdown failed: %s", e, instance=instance)
+
+                time.sleep(wait_time)
+                timeout -= retry_interval
+
+            LOG.warning("Timed out while waiting for soft shutdown.",
+                        instance=instance)
+        else:
+            LOG.info("VMWare Tools not running", instance=instance)
+
+        return False
+
+
     def power_on(self, instance):
         vm_util.power_on_instance(self._session, instance

Where are you experiencing the behavior?  What environment?

applications on the os could not stop gentle

When does the behavior occur? Frequently?  Repeatedly?   At certain times?

every time

Comment 5 GE Scott Knauss 2016-04-01 12:01:01 UTC
Matt,
     They are going to submit to the upstream. Should we close this BZ and open a new once they have their code accepted? If so, feel free... 

-Scott

Comment 6 Matthew Booth 2016-04-01 16:30:46 UTC
(In reply to GE Scott Knauss from comment #5)
> Matt,
>      They are going to submit to the upstream. Should we close this BZ and
> open a new once they have their code accepted? If so, feel free... 

Thanks, Scott.