Bug 1882110

Created attachment 1716125 [details]
Contents of /var/lib/mistra/config-download-latest

Description of problem: 

After a successful OSP13 to OSP16.1 upgrade. Any subsequent attempts to update the environment fail at the Ansible task which checks 'Check Keystone public endpoint status'. I have been able to reproduce this consistently in the 3 attempts that I have tried.


Version-Release number of selected component (if applicable): 

RHOSP 16.1


How reproducible: Always


Steps to Reproduce:

1. Perform an OSP13 to OSP 16.1 upgrade successfully.
2. Perform a stack update.
3. The heat stack update completes successfully.
3. During the Ansible/config-downloads steps.  Beginning on the [External deployment step 4] steps, the Ansible run fails on the 'Check Keystone public endpoint status' task.

Actual results:
Overcloud stack update does not complete.


Expected results:
Overcloud update to complete successfully.


Additional info:

OpenStack Deployment Details

3 x Controllers (Virtual) 
2 x Computes (Baremetal)
Director Hosted Container Image Registry
NFS storage for Glance images and Cinder Volumes
TLS Encryption on ExternalAPI
Neutron OVN+DVR
Predictable Hostnames
Predictable IP Addresses
Satellite Server 6.7 for content managemen


OS: RHEL 8.1
OSP Version: 16.1


Overcloud Upgrade Converge Command
----------------------------------
THT_DEFAULT="/usr/share/openstack-tripleo-heat-templates"
THT_CUSTOM="/home/stack/overcloud_deployment/templates"

openstack overcloud upgrade converge --templates \
  -r ${THT_CUSTOM}/roles_data.yaml \
  -n ${THT_CUSTOM}/network_data.yaml \
  -e ${THT_DEFAULT}/environments/network-isolation.yaml \
  -e ${THT_DEFAULT}/environments/services/neutron-ovn-dvr-ha.yaml \
  -e ${THT_DEFAULT}/environments/ssl/tls-endpoints-public-dns.yaml \
  -e ${THT_CUSTOM}/node-info.yaml \
  -e ${THT_CUSTOM}/containers-prepare-parameter.yaml \
  -e ${THT_CUSTOM}/customizations.yaml \
  -e ${THT_CUSTOM}/first_boot.yaml \
  -e ${THT_CUSTOM}/environments/network-environment.yaml \
  -e ${THT_CUSTOM}/ips-from-pool-all.yaml \
  -e ${THT_CUSTOM}/environments/inject-trust-anchor-hiera.yaml \
  -e ${THT_CUSTOM}/environments/enable-tls.yaml \
  -e ${THT_CUSTOM}/environments/glance-nfs.yaml \
  -e ${THT_CUSTOM}/environments/cinder-nfs.yaml \
  -e ${THT_CUSTOM}/environments/sshd-banner.yaml \
  -e ${THT_CUSTOM}/environments/disable-telemetry.yaml \
  -e ${THT_CUSTOM}/environments/disable-swift.yaml \
  -e ${THT_CUSTOM}/environments/disable-swift.yaml \
  -e ${THT_CUSTOM}/leapp-upgrade.yaml
----------------------------------

Overcloud Deploy/Update Command
----------------------------------
THT_DEFAULT="/usr/share/openstack-tripleo-heat-templates"
THT_CUSTOM="/home/stack/overcloud_deployment/templates"

openstack overcloud deploy --templates \
  -r ${THT_CUSTOM}/roles_data.yaml \
  -n ${THT_CUSTOM}/network_data.yaml \
  -e ${THT_DEFAULT}/environments/network-isolation.yaml \
  -e ${THT_DEFAULT}/environments/services/neutron-ovn-dvr-ha.yaml \
  -e ${THT_DEFAULT}/environments/ssl/tls-endpoints-public-dns.yaml \
  -e ${THT_CUSTOM}/node-info.yaml \
  -e ${THT_CUSTOM}/containers-prepare-parameter.yaml \
  -e ${THT_CUSTOM}/customizations.yaml \
  -e ${THT_CUSTOM}/first_boot.yaml \
  -e ${THT_CUSTOM}/environments/network-environment.yaml \
  -e ${THT_CUSTOM}/ips-from-pool-all.yaml \
  -e ${THT_CUSTOM}/environments/inject-trust-anchor-hiera.yaml \
  -e ${THT_CUSTOM}/environments/enable-tls.yaml \
  -e ${THT_CUSTOM}/environments/glance-nfs.yaml \
  -e ${THT_CUSTOM}/environments/cinder-nfs.yaml \
  -e ${THT_CUSTOM}/environments/sshd-banner.yaml \
  -e ${THT_CUSTOM}/environments/disable-telemetry.yaml \
  -e ${THT_CUSTOM}/environments/disable-swift.yaml
----------------------------------



Troubleshooting Details

After enabling logging on the failed task (/usr/share/ansible/roles/tripleo-keystone-resources/tasks/endpoints.yml) and attempting another update. I was able to capture the details in the failure which appears there are multiple endpoints found for the cinderv3 service which causes the particular task to fail.

-------------------------------------
TASK [tripleo-keystone-resources : Check Keystone public endpoint status] ******
Wednesday 23 September 2020  13:37:24 -0400 (0:00:09.193)       0:31:07.466 ***
.
.
.
failed: [undercloud] (item={'started': 1, 'finished': 0, 'ansible_job_id': '724656674095.376605', 'results_file': '/root/.ansible_async/724656674095.376605', 'changed': True, 'failed': False, 'tripleo_keystone_resources_data': {'key': 'cinderv3', 'value': {'endpoints': {'admin': 'http://172.25.51.251:8776/v3/%(tenant_id)s', 'internal': 'http://172.25.51.251:8776/v3/%(tenant_id)s', 'public': 'https://osp13.holaehlo.net:13776/v3/%(tenant_id)s'}, 'region': 'regionOne', 'service': 'volumev3', 'users': {'cinderv3': {'password': '6D9sdP6D3npHGy6X6tUk4YdQP', 'roles': ['admin', 'service']}}}}, 'ansible_loop_var': 'tripleo_keystone_resources_data'}) => {"ansible_job_id": "724656674095.376605", "ansible_loop_var": "tripleo_keystone_resources_endpoint_async_result_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Multiple matches found for cinderv3", "tripleo_keystone_resources_endpoint_async_result_item": {"ansible_job_id": "724656674095.376605", "ansible_loop_var": "tripleo_keystone_resources_data", "changed": true, "failed": false, "finished": 0, "results_file": "/root/.ansible_async/724656674095.376605", "started": 1, "tripleo_keystone_resources_data": {"key": "cinderv3", "value": {"endpoints": {"admin": "http://172.25.51.251:8776/v3/%(tenant_id)s", "internal": "http://172.25.51.251:8776/v3/%(tenant_id)s", "public": "https://osp13.holaehlo.net:13776/v3/%(tenant_id)s"}, "region": "regionOne", "service": "volumev3", "users": {"cinderv3": {"password": "6D9sdP6D3npHGy6X6tUk4YdQP", "roles": ["admin", "service"]}}}}}}
.
.
.


# Unable to reproduce that output with cinderv2 as the service name when listing endpoints
-------------------------------------

(overcloud) [stack@osp-dir osp16_upgrade]$ openstack endpoint list --service cinderv2
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                               |
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------+
| 7e3c2a13712e41f7a4242ee52183a929 | regionOne | cinderv2     | volumev2     | True    | internal  | http://172.25.51.251:8776/v2/%(tenant_id)s        |
| beedfe79545c4a57bc1cf85328d44dad | regionOne | cinderv2     | volumev2     | True    | admin     | http://172.25.51.251:8776/v2/%(tenant_id)s        |
| cdc3538fd22e491481885fa676213068 | regionOne | cinderv2     | volumev2     | True    | public    | https://osp13.holaehlo.net:13776/v2/%(tenant_id)s |
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------+


# Definitely able to reproduce the output when specifying cinderv3 as the service:
-------------------------------------
(overcloud) [stack@osp-dir osp16_upgrade]$ openstack endpoint list --service cinderv3
Multiple service matches found for 'cinderv3', use an ID to be more specific.
-------------------------------------

# Output of openstack service list
(overcloud) [stack@osp-dir ~]$ openstack service list
+----------------------------------+-----------+----------------+
| ID                               | Name      | Type           |
+----------------------------------+-----------+----------------+
| 048d8940ab4c4ed3bb93d90399dbcc3c | cinderv3  | volume         |
| 10d795f1b7bc405995684d3dafce8d4d | keystone  | identity       |
| 1ebc7fb4d4d34a3e98a7c5a5bc3326c5 | glance    | image          |
| 2f19995ed2e84170a431dff213160d44 | placement | placement      |
| 4781027f384a421a82079a59decd1307 | cinderv2  | volumev2       |
| 4e622dda93b746ae863b6a47fad3cf06 | neutron   | network        |
| 5ebfd19b1ee74ba99c796ea1a833d327 | heat-cfn  | cloudformation |
| 6152ac6d401e4eb1ae3ad3f9ea1278c0 | nova      | compute        |
| e2743c2ee3d747a79da526f23c95838c | heat      | orchestration  |
| e2cd852487f84c57a85557a28c6ba255 | cinderv3  | volumev3       |
+----------------------------------+-----------+----------------+


# Output of openstack endpoint list
(overcloud) [stack@osp-dir ~]$ openstack endpoint list
+----------------------------------+-----------+--------------+----------------+---------+-----------+---------------------------------------------------+
| ID                               | Region    | Service Name | Service Type   | Enabled | Interface | URL                                               |
+----------------------------------+-----------+--------------+----------------+---------+-----------+---------------------------------------------------+
| 0724d40e9c884339ab968138bac073e0 | regionOne | heat         | orchestration  | True    | admin     | http://172.25.51.251:8004/v1/%(tenant_id)s        |
| 1e281687becb49a295b00e8edd383121 | regionOne | placement    | placement      | True    | internal  | http://172.25.51.251:8778/placement               |
| 21b301ec20f44f7b869285d918b23919 | regionOne | heat-cfn     | cloudformation | True    | public    | https://osp13.holaehlo.net:13005/v1               |
| 2f573a5188764ac6afc9136db0927964 | regionOne | nova         | compute        | True    | admin     | http://172.25.51.251:8774/v2.1                    |
| 390ac33b37d042a184ae2203c1339da9 | regionOne | heat         | orchestration  | True    | internal  | http://172.25.51.251:8004/v1/%(tenant_id)s        |
| 39729789a6fb4a938882ced4ea03cf28 | regionOne | placement    | placement      | True    | public    | https://osp13.holaehlo.net:13778/placement        |
| 49f1f95285624eedab799508e2b54968 | regionOne | keystone     | identity       | True    | admin     | http://172.16.24.9:35357                          |
| 4f23fcb9b2634f249d14ee92b04bebb5 | regionOne | neutron      | network        | True    | public    | https://osp13.holaehlo.net:13696                  |
| 5466620359734d95af5703eaf3dec647 | regionOne | heat-cfn     | cloudformation | True    | internal  | http://172.25.51.251:8000/v1                      |
| 58d688de75c847618e65634baed5bfc3 | regionOne | neutron      | network        | True    | internal  | http://172.25.51.251:9696                         |
| 5a0b4222ff9143b5a63f5984b5ab34dd | regionOne | cinderv3     | volume         | True    | public    | https://osp13.holaehlo.net:13776/v3/%(tenant_id)s |
| 5fcf1dd6cf28478b964ddf50e93d3a88 | regionOne | cinderv3     | volumev3       | True    | admin     | http://172.25.51.251:8776/v3/%(tenant_id)s        |
| 6a676d4dc03b4f08a9cd43dca7ecbf45 | regionOne | cinderv3     | volume         | True    | internal  | http://172.25.51.251:8776/v3/%(tenant_id)s        |
| 6b0d20c7f05b430b93af2729852269e1 | regionOne | cinderv3     | volumev3       | True    | public    | https://osp13.holaehlo.net:13776/v3/%(tenant_id)s |
| 7e3c2a13712e41f7a4242ee52183a929 | regionOne | cinderv2     | volumev2       | True    | internal  | http://172.25.51.251:8776/v2/%(tenant_id)s        |
| 816581bc111043d99e0f198053fc46e9 | regionOne | heat         | orchestration  | True    | public    | https://osp13.holaehlo.net:13004/v1/%(tenant_id)s |
| 884be6362f9a471d8cd174609f783b08 | regionOne | glance       | image          | True    | internal  | http://172.25.51.251:9292                         |
| 8d327310cc1d454882a005210add1392 | regionOne | keystone     | identity       | True    | public    | https://osp13.holaehlo.net:13000                  |
| 94a85b739aeb4b54b9f169fc690e949f | regionOne | placement    | placement      | True    | admin     | http://172.25.51.251:8778/placement               |
| 9a829a67301f479fa04b568d5f92c5c6 | regionOne | neutron      | network        | True    | admin     | http://172.25.51.251:9696                         |
| a68c3f3bf9d145c3b953889782bb0797 | regionOne | nova         | compute        | True    | internal  | http://172.25.51.251:8774/v2.1                    |
| ae840f1ee82c46e3a8cae4d84ee28417 | regionOne | keystone     | identity       | True    | internal  | http://172.25.51.251:5000                         |
| af37e8176f7d4a968e72ba19c034458f | regionOne | heat-cfn     | cloudformation | True    | admin     | http://172.25.51.251:8000/v1                      |
| b97fd191ac84458387f270b2248b2123 | regionOne | nova         | compute        | True    | public    | https://osp13.holaehlo.net:13774/v2.1             |
| beedfe79545c4a57bc1cf85328d44dad | regionOne | cinderv2     | volumev2       | True    | admin     | http://172.25.51.251:8776/v2/%(tenant_id)s        |
| c1f175c4ada04939b7bbc3b499474d68 | regionOne | glance       | image          | True    | public    | https://osp13.holaehlo.net:13292                  |
| cdc3538fd22e491481885fa676213068 | regionOne | cinderv2     | volumev2       | True    | public    | https://osp13.holaehlo.net:13776/v2/%(tenant_id)s |
| d9af2f69a9e14c7780e4bb9ee28236e9 | regionOne | cinderv3     | volumev3       | True    | internal  | http://172.25.51.251:8776/v3/%(tenant_id)s        |
| dbfd0463e69d46ee91d5f737b2808e84 | regionOne | glance       | image          | True    | admin     | http://172.25.51.251:9292                         |
| e6533390fe96424d845f0298f909bbe9 | regionOne | cinderv3     | volume         | True    | admin     | http://172.25.51.251:8776/v3/%(tenant_id)s        |
+----------------------------------+-----------+--------------+----------------+---------+-----------+---------------------------------------------------+



Testing a workaround by deleting the duplicate volumev3 Keystone endpoints with the 'volume' as the Service Type and deleting the openstack service associated with the Type 'volume' seems to help with the issue and subsequent stack updates are successful, however I am looking to see if this is a known issue but not yet documented or if it is a legitimate issue with the OSP13 -> OSP16 upgrade not cleaning up deprecated endpoints and/or services (such as the case with nova-consoleauth).


Let me know if I can provide any additional information.  This is a lab environment so I am able to reconstruct the OSP13 deployment and begin the OSP16 upgrade again to provide information as needed throughout the process.