Bug 1475939 - director update done to openstack causes a configuration on the compute nodes disallowing migrate/resize [NEEDINFO]
director update done to openstack causes a configuration on the compute nodes...
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
10.0 (Newton)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Ollie Walsh
Joe H. Rahme
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-27 11:14 EDT by Ruchika K
Modified: 2017-09-08 05:03 EDT (History)
19 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-09-08 05:03:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
owalsh: needinfo? (rkharwar)


Attachments (Terms of Use)

  None (edit)
Description Ruchika K 2017-07-27 11:14:46 EDT
Description of problem:

Update to an osp10 install loses some configuration for compute nodes after a update.

Version-Release number of selected component (if applicable):
Latest OSP10 version with these 

openstack-nova-console-14.0.6-2.el7ost.noarch 
openstack-nova-common-14.0.6-2.el7ost.noarch 
openstack-nova-cert-14.0.6-2.el7ost.noarch 
python-nova-14.0.6-2.el7ost.noarch 
openstack-nova-novncproxy-14.0.6-2.el7ost.noarch 
openstack-nova-scheduler-14.0.6-2.el7ost.noarch 
openstack-nova-api-14.0.6-2.el7ost.noarch 
puppet-nova-9.5.0-6.el7ost.noarch 

How reproducible:

- Was able to reproduce it twice.


Steps to Reproduce:
1. Install osp10 with 1 controller and 2 compute nodes (no shared storage)
2. Check resize/migration and it works well.
3. Do a director update (in my example I wanted to integrate an external ceph storage cluster) so ran an update for this
4. After it finishes, spin up an instance, try to do a migrate and you will see ssh errors with the compute nodes unable to migrate

like this
ecute ssh command: Unexpected error while running command.
Command: ssh -o BatchMode=yes 10.0.0.50 mkdir -p /var/lib/nova/instances/1d3f063f-eaac-4ce1-a7fc-89b18607e55b
Exit code: 255
Stdout: u''
Stderr: u'Host key verification failed.\r\n'
2017-07-27 02:30:26.098 157162 INFO nova.compute.manager [req-55d3c245-08df-460a-b461-47286d3a9ba7 24d888afa37844e393519e1e4eae8944 e7e1f85b81b14d5bb58a1217dccf0df8 - - -] [instance: 1d3f063f-eaac-4ce1-a7fc-89b18607e55b] Successfully reverted task state from None on failure for instance.
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server [req-55d3c245-08df-460a-b461-47286d3a9ba7 24d888afa37844e393519e1e4eae8944 e7e1f85b81b14d5bb58a1217dccf0df8 - - -] Exception during message handling
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     migration.instance_uuid, exc_info=True)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3846, in resize_instance
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.instance_events.clear_events_for_instance(instance)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.gen.throw(type, value, traceback)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6618, in _error_out_instance_on_exception
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     raise error.inner_exception
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server Command: ssh -o BatchMode=yes 10.0.0.50 mkdir -p /var/lib/nova/instances/1d3f063f-eaac-4ce1-a7fc-89b18607e55b
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server Exit code: 255
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server Stdout: u''
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server Stderr: u'Host key verification failed.\r\n'
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server
2017-07-27 02:31:05.073 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Auditing locally available compute resources for node overcloud-compute-1.localdomain
2017-07-27 02:31:05.495 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:31:05.548 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] No authentication information found for placement API. Placement is optional in Newton, but required in Ocata. Please enable the placement service before upgrading.
2017-07-27 02:31:05.581 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Total usable vcpus: 4, total allocated vcpus: 1
2017-07-27 02:31:05.581 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Final resource view: name=overcloud-compute-1.localdomain phys_ram=8191MB used_ram=2304MB phys_disk=49GB used_disk=4GB total_vcpus=4 used_vcpus=1 pci_stats=[]
2017-07-27 02:31:05.619 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:31:05.619 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Compute_service record updated for overcloud-compute-1.localdomain:overcloud-compute-1.localdomain
2017-07-27 02:32:07.071 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Auditing locally available compute resources for node overcloud-compute-1.localdomain
2017-07-27 02:32:07.466 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:32:07.543 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Total usable vcpus: 4, total allocated vcpus: 1
2017-07-27 02:32:07.543 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Final resource view: name=overcloud-compute-1.localdomain phys_ram=8191MB used_ram=2304MB phys_disk=49GB used_disk=4GB total_vcpus=4 used_vcpus=1 pci_stats=[]
2017-07-27 02:32:07.574 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:32:07.574 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Compute_service record updated for overcloud-compute-1.localdomain:overcloud-compute-1.localdomain
2017-07-27 02:33:09.096 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Auditing locally available compute resources for node overcloud-compute-1.localdomain
2017-07-27 02:33:09.516 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:33:09.564 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] No authentication information found for placement API. Placement is optional in Newton, but required in Ocata. Please enable the placement service before upgrading.
2017-07-27 02:33:09.596 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Total usable vcpus: 4, total allocated vcpus: 1
2017-07-27 02:33:09.597 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Final resource view: name=overcloud-compute-1.localdomain phys_ram=8191MB used_ram=2304MB phys_disk=49GB used_disk=4GB total_vcpus=4 used_vcpus=1 pci_stats=[]
2017-07-27 02:33:09.639 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:33:09.640 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Compute_service record updated for overcloud-compute-1.localdomain:overcloud-compute-1.localdomain


5. I deleted the stack and installed it all with the external ceph cluster in a single step and the resize/migration issue does not exist anymore so the 

Actual results:
conclusion is the update is causing an error in the compute configuration to allow resize/migrate.


Expected results:
resize/migrate should not get impacted 

Additional info:
This is the ceph puppet file I used to add in the external ceph cluster as part of the update. Note rbd nova backed is disabled as per customer requirement.



# A Heat environment file which can be used to enable the
# use of an externally managed Ceph cluster.
resource_registry:
  OS::TripleO::Services::CephExternal: /usr/share/openstack-tripleo-heat-templates/puppet/services/ceph-external.yaml
  OS::TripleO::Services::CephMon: OS::Heat::None
  OS::TripleO::Services::CephClient: OS::Heat::None
  OS::TripleO::Services::CephOSD: OS::Heat::None
 
parameter_defaults:
  # NOTE: These example parameters are required when using CephExternal
  CephClusterFSID: '82a3778b-4907-4252-9514-562194159891'
  CephClientKey: 'AQCKQXZZznR7ABAAIdRSCBiqU9QkOnqNxZ0AiQ=='
  CephExternalMonHost: '10.0.0.60, 10.0.0.61, 10.0.0.62'
 
  #CephClusterFSID: '4b5c8c0a-ff60-454b-a1b4-9747aa737d19'
  #CephClientKey: 'AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ=='
  #CephExternalMonHost: '172.16.1.7, 172.16.1.8'
 
  # the following parameters enable Ceph backends for Cinder, Glance, Gnocchi and Nova
  NovaEnableRbdBackend: true
  CinderEnableRbdBackend: true
  CinderBackupBackend: ceph
  GlanceBackend: rbd
  GnocchiBackend: rbd
  # If the Ceph pools which host VMs, Volumes and Images do not match these
  # names OR the client keyring to use is not named 'openstack',  edit the
  # following as needed.
  NovaRbdPoolName: vmsosp10
  CinderRbdPoolName: volumesosp10
  GlanceRbdPoolName: imagesosp10
  GnocchiRbdPoolName: metricsosp10
  CinderBackupRbdPoolName: backupsosp10
  CephClientUserName: openstackosp10
  # finally we disable the Cinder LVM backend
  CinderEnableIscsiBackend: false
 
  # Backward compatibility setting, will be removed in the future
  CephAdminKey: ''
Comment 2 Matthew Booth 2017-09-08 05:03:18 EDT
Closing this as we don't have enough information to work on it. Please feel free to reopen it if it is still affecting you and you are able to supply to requested information.

Note You need to log in before you can comment on or make changes to this bug.