Description of problem:

Update to an osp10 install loses some configuration for compute nodes after a update.

Version-Release number of selected component (if applicable):
Latest OSP10 version with these 

openstack-nova-console-14.0.6-2.el7ost.noarch 
openstack-nova-common-14.0.6-2.el7ost.noarch 
openstack-nova-cert-14.0.6-2.el7ost.noarch 
python-nova-14.0.6-2.el7ost.noarch 
openstack-nova-novncproxy-14.0.6-2.el7ost.noarch 
openstack-nova-scheduler-14.0.6-2.el7ost.noarch 
openstack-nova-api-14.0.6-2.el7ost.noarch 
puppet-nova-9.5.0-6.el7ost.noarch 

How reproducible:

- Was able to reproduce it twice.


Steps to Reproduce:
1. Install osp10 with 1 controller and 2 compute nodes (no shared storage)
2. Check resize/migration and it works well.
3. Do a director update (in my example I wanted to integrate an external ceph storage cluster) so ran an update for this
4. After it finishes, spin up an instance, try to do a migrate and you will see ssh errors with the compute nodes unable to migrate

like this
ecute ssh command: Unexpected error while running command.
Command: ssh -o BatchMode=yes 10.0.0.50 mkdir -p /var/lib/nova/instances/1d3f063f-eaac-4ce1-a7fc-89b18607e55b
Exit code: 255
Stdout: u''
Stderr: u'Host key verification failed.\r\n'
2017-07-27 02:30:26.098 157162 INFO nova.compute.manager [req-55d3c245-08df-460a-b461-47286d3a9ba7 24d888afa37844e393519e1e4eae8944 e7e1f85b81b14d5bb58a1217dccf0df8 - - -] [instance: 1d3f063f-eaac-4ce1-a7fc-89b18607e55b] Successfully reverted task state from None on failure for instance.
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server [req-55d3c245-08df-460a-b461-47286d3a9ba7 24d888afa37844e393519e1e4eae8944 e7e1f85b81b14d5bb58a1217dccf0df8 - - -] Exception during message handling
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     migration.instance_uuid, exc_info=True)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3846, in resize_instance
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.instance_events.clear_events_for_instance(instance)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     self.gen.throw(type, value, traceback)
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6618, in _error_out_instance_on_exception
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server     raise error.inner_exception
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server Command: ssh -o BatchMode=yes 10.0.0.50 mkdir -p /var/lib/nova/instances/1d3f063f-eaac-4ce1-a7fc-89b18607e55b
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server Exit code: 255
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server Stdout: u''
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server Stderr: u'Host key verification failed.\r\n'
2017-07-27 02:30:26.106 157162 ERROR oslo_messaging.rpc.server
2017-07-27 02:31:05.073 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Auditing locally available compute resources for node overcloud-compute-1.localdomain
2017-07-27 02:31:05.495 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:31:05.548 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] No authentication information found for placement API. Placement is optional in Newton, but required in Ocata. Please enable the placement service before upgrading.
2017-07-27 02:31:05.581 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Total usable vcpus: 4, total allocated vcpus: 1
2017-07-27 02:31:05.581 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Final resource view: name=overcloud-compute-1.localdomain phys_ram=8191MB used_ram=2304MB phys_disk=49GB used_disk=4GB total_vcpus=4 used_vcpus=1 pci_stats=[]
2017-07-27 02:31:05.619 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:31:05.619 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Compute_service record updated for overcloud-compute-1.localdomain:overcloud-compute-1.localdomain
2017-07-27 02:32:07.071 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Auditing locally available compute resources for node overcloud-compute-1.localdomain
2017-07-27 02:32:07.466 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:32:07.543 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Total usable vcpus: 4, total allocated vcpus: 1
2017-07-27 02:32:07.543 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Final resource view: name=overcloud-compute-1.localdomain phys_ram=8191MB used_ram=2304MB phys_disk=49GB used_disk=4GB total_vcpus=4 used_vcpus=1 pci_stats=[]
2017-07-27 02:32:07.574 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:32:07.574 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Compute_service record updated for overcloud-compute-1.localdomain:overcloud-compute-1.localdomain
2017-07-27 02:33:09.096 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Auditing locally available compute resources for node overcloud-compute-1.localdomain
2017-07-27 02:33:09.516 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:33:09.564 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] No authentication information found for placement API. Placement is optional in Newton, but required in Ocata. Please enable the placement service before upgrading.
2017-07-27 02:33:09.596 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Total usable vcpus: 4, total allocated vcpus: 1
2017-07-27 02:33:09.597 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Final resource view: name=overcloud-compute-1.localdomain phys_ram=8191MB used_ram=2304MB phys_disk=49GB used_disk=4GB total_vcpus=4 used_vcpus=1 pci_stats=[]
2017-07-27 02:33:09.639 157162 WARNING nova.scheduler.client.report [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Unable to refresh my resource provider record
2017-07-27 02:33:09.640 157162 INFO nova.compute.resource_tracker [req-1c9b7a4b-507a-4392-8c6d-35b4c89f0383 - - - - -] Compute_service record updated for overcloud-compute-1.localdomain:overcloud-compute-1.localdomain


5. I deleted the stack and installed it all with the external ceph cluster in a single step and the resize/migration issue does not exist anymore so the 

Actual results:
conclusion is the update is causing an error in the compute configuration to allow resize/migrate.


Expected results:
resize/migrate should not get impacted 

Additional info:
This is the ceph puppet file I used to add in the external ceph cluster as part of the update. Note rbd nova backed is disabled as per customer requirement.



# A Heat environment file which can be used to enable the
# use of an externally managed Ceph cluster.
resource_registry:
  OS::TripleO::Services::CephExternal: /usr/share/openstack-tripleo-heat-templates/puppet/services/ceph-external.yaml
  OS::TripleO::Services::CephMon: OS::Heat::None
  OS::TripleO::Services::CephClient: OS::Heat::None
  OS::TripleO::Services::CephOSD: OS::Heat::None
 
parameter_defaults:
  # NOTE: These example parameters are required when using CephExternal
  CephClusterFSID: '82a3778b-4907-4252-9514-562194159891'
  CephClientKey: 'AQCKQXZZznR7ABAAIdRSCBiqU9QkOnqNxZ0AiQ=='
  CephExternalMonHost: '10.0.0.60, 10.0.0.61, 10.0.0.62'
 
  #CephClusterFSID: '4b5c8c0a-ff60-454b-a1b4-9747aa737d19'
  #CephClientKey: 'AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ=='
  #CephExternalMonHost: '172.16.1.7, 172.16.1.8'
 
  # the following parameters enable Ceph backends for Cinder, Glance, Gnocchi and Nova
  NovaEnableRbdBackend: true
  CinderEnableRbdBackend: true
  CinderBackupBackend: ceph
  GlanceBackend: rbd
  GnocchiBackend: rbd
  # If the Ceph pools which host VMs, Volumes and Images do not match these
  # names OR the client keyring to use is not named 'openstack',  edit the
  # following as needed.
  NovaRbdPoolName: vmsosp10
  CinderRbdPoolName: volumesosp10
  GlanceRbdPoolName: imagesosp10
  GnocchiRbdPoolName: metricsosp10
  CinderBackupRbdPoolName: backupsosp10
  CephClientUserName: openstackosp10
  # finally we disable the Cinder LVM backend
  CinderEnableIscsiBackend: false
 
  # Backward compatibility setting, will be removed in the future
  CephAdminKey: ''