Bug 1856906

Description of problem:

We have successfully upgraded our HCI environment from OSP13 to OSP16.1 by following the upgrade procedure[1] and now trying to migrate from OVS to OVN as per guide[2]. While running "ovn_migration.sh start-migration" ansible-playbook executes stack update task[3] which in turn executes[4] and fails every time.

>> We are not able to trace any error in "/var/lib/mistral/msufiyan/ansible.log" except a few of the errors related to container "container-puppet-keystone_init_tasks" which was already removed by ansible.

~~~
2020-07-14 15:32:39,086 p=341859 u=mistral n=ansible | TASK [Debug output for task: Run container-puppet tasks (bootstrap tasks) for step 3] ***
2020-07-14 15:32:39,087 p=341859 u=mistral n=ansible | Tuesday 14 July 2020  15:32:39 +0000 (0:00:10.152)       0:28:33.611 **********
2020-07-14 15:32:39,211 p=341859 u=mistral n=ansible | ok: [controller-0] => {
    "bootstrap_tasks_outputs.stdout_lines | default([]) | union(bootstrap_tasks_outputs.stderr_lines | default([]))": [
        "2020-07-14 15:32:28,421 INFO: 315311 -- Running container-puppet",
        "2020-07-14 15:32:28,422 INFO: 315311 -- Service compilation completed.",
        "2020-07-14 15:32:28,436 INFO: 315327 -- Starting configuration of keystone_init_tasks using image 192.168.24.1:8787/rhosp-beta/openstack-keystone:16.1-38",
        "2020-07-14 15:32:28,505 ERROR: 315327 -- Error executing ['podman', 'container', 'exists', 'container-puppet-keystone_init_tasks']: returned 1",
        "2020-07-14 15:32:28,505 INFO: 315327 -- Removing container: container-puppet-keystone_init_tasks",
        "2020-07-14 15:32:28,785 INFO: 315327 -- Image already exists: 192.168.24.1:8787/rhosp-beta/openstack-keystone:16.1-38",
        "2020-07-14 15:32:38,006 WARNING: 315327 -- + mkdir -p /etc/puppet",
        "+ cp -dR /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/hiera.yaml.rpmnew /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet",
        "+ rm -Rf /etc/puppet/ssl",
        "+ echo '{\"step\": 3}'",
        "+ TAGS=",
        "+ '[' -n file,file_line,concat,augeas,cron,keystone_config ']'",
        "+ TAGS='--tags \"file,file_line,concat,augeas,cron,keystone_config\"'",
        "+ '[' '!' -z ']'",
        "+ CHECK_MODE=",
        "+ '[' -d /tmp/puppet-check-mode ']'",
        "+ conf_data_path=/var/lib/config-data/keystone_init_tasks",
        "+ origin_of_time=/var/lib/config-data/keystone_init_tasks.origin_of_time",
        "+ touch /var/lib/config-data/keystone_init_tasks.origin_of_time",
        "+ sync",
        "+ export NET_HOST=true",
        "+ NET_HOST=true",
        "+ set +e",
        "+ '[' true == false ']'",
        "+ export FACTER_deployment_type=containers",
        "+ FACTER_deployment_type=containers",
        "++ cat /sys/class/dmi/id/product_uuid",
        "++ tr '[:upper:]' '[:lower:]'",
        "+ export FACTER_uuid=76d9a518-8c54-4f2d-a5e5-677cf9b66131",
        "+ FACTER_uuid=76d9a518-8c54-4f2d-a5e5-677cf9b66131",
        "+ echo 'Running puppet'",
        "+ set -x",
        "+ logger -s -t puppet-user",
        "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '\"file,file_line,concat,augeas,cron,keystone_config\"' /etc/config.pp",
        "<13>Jul 14 15:32:29 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5",
        "<13>Jul 14 15:32:37 puppet-user:    (file: /etc/puppet/hiera.yaml)",
        "<13>Jul 14 15:32:37 puppet-user: Warning: Undefined variable '::deploy_config_name'; \\n   (file & line not available)",
        "<13>Jul 14 15:32:37 puppet-user: Warning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)",
        "<13>Jul 14 15:32:37 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\\n   (file & line not available)",
        "<13>Jul 14 15:32:37 puppet-user: Notice: Compiled catalog for controller-0.redhat.local in environment production in 0.09 seconds",
        "<13>Jul 14 15:32:37 puppet-user: Notice: Applied catalog in 0.01 seconds",
        "<13>Jul 14 15:32:37 puppet-user: Changes:",
        "<13>Jul 14 15:32:37 puppet-user: Events:",
        "<13>Jul 14 15:32:37 puppet-user: Resources:",
        "<13>Jul 14 15:32:37 puppet-user:           Skipped: 7",
        "<13>Jul 14 15:32:37 puppet-user:             Total: 7",
        "<13>Jul 14 15:32:37 puppet-user: Time:",
        "<13>Jul 14 15:32:37 puppet-user:    Transaction evaluation: 0.01",
        "<13>Jul 14 15:32:37 puppet-user:    Catalog application: 0.01",
        "<13>Jul 14 15:32:37 puppet-user:    Config retrieval: 0.14",
        "<13>Jul 14 15:32:37 puppet-user:          Last run: 1594740757",
        "<13>Jul 14 15:32:37 puppet-user:             Total: 0.01",
        "<13>Jul 14 15:32:37 puppet-user: Version:",
        "<13>Jul 14 15:32:37 puppet-user:            Config: 1594740757",
        "<13>Jul 14 15:32:37 puppet-user:            Puppet: 5.5.10",
        "+ rc=0",
        "+ '[' False = false ']'",
        "+ set -e",
        "+ '[' 0 -ne 2 -a 0 -ne 0 ']'",
        "+ verbosity=",
        "+ verbosity=-v",
        "+ '[' -z true ']'",
        "",
        "2020-07-14 15:32:38,006 INFO: 315327 -- Removing container: container-puppet-keystone_init_tasks",
        "2020-07-14 15:32:38,167 INFO: 315327 -- Finished processing puppet configs for keystone_init_tasks"
    ],
    "failed_when_result": false
}
~~~

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1-beta/html-single/framework_for_upgrades_13_to_16.1/index?lb_target=preview#upgrading-a-standard-overcloud
[2] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1-beta/html-single/networking_with_open_virtual_network/index#migrating-ml2ovs-to-ovn

[3] 
~~~
TASK [tripleo-update : Updating the overcloud stack with OVN services] 
task path: /home/stack/ovn_migration/playbooks/roles/tripleo-update/tasks/main.yml:20
~~~

[4]
 
~~~
TASK [tripleo-keystone-resources : Check Keystone public endpoint status] 
~~~



Version-Release number of selected component (if applicable):
OSP16.1

How reproducible:
Every time when stack update procedure executes "Check Keystone public endpoint status"