Bug 1810445

Description of problem:

When upgrading the Undercloud from OSP15 to OSP16, the operation fails during the deployment steps in the puppet configuration for the create_dnsmasq_wrapper container.

The error is the following:

\"['podman', 'ps', '-a', '--filter', 'label=container_name=create_dnsmasq_wrapper', '--format', '{{.Names}}']\"\nError running ['podman', 'run', '--name', 'create_dnsmasq_wrapper', '--label', 'config_id=tripleo_step2', '--label', 'container_name=create_dnsmasq_wrapper', '--label', 'managed_by=tripleo-Undercloud', '--label', 'config_data={\"command\": [\"/container_puppet_apply.sh\", \"4\", \"file\", \"include ::tripleo::profile::base::neutron::dhcp_agent_wrappers\"], \"detach\": false, \"environment\": {\"TRIPLEO_DEPLOY_IDENTIFIER\": \"1583369143\"}, \"image\": \"undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-dhcp-agent:20200303.1\", \"net\": \"host\", \"pid\": \"host\", \"start_order\": 1, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro\", \"/run/openvswitch:/run/openvswitch:shared,z\", \"/var/lib/neutron:/var/lib/neutron:shared,z\"]}', '--conmon-pidfile=/var/run/create_dnsmasq_wrapper.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/create_dnsmasq_wrapper.log', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1583369143', '--net=host', '--pid=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro', '--volume=/etc/puppet:/tmp/puppet-etc:ro', '--volume=/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro', '--volume=/run/openvswitch:/run/openvswitch:shared,z', '--volume=/var/lib/neutron:/var/lib/neutron:shared,z', '--cpuset-cpus=0,1,2,3,4,5,6,7', 'undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-dhcp-agent:20200303.1', '/container_puppet_apply.sh', '4', 'file', 'include ::tripleo::profile::base::neutron::dhcp_agent_wrappers']. [1]\n\nstdout: Info: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\n\nstderr: + STEP=4\n+ TAGS=file\n+ CONFIG='include ::tripleo::profile::base::neutron::dhcp_agent_wrappers'\n+ EXTRA_ARGS=\n+ '[' -d /tmp/puppet-etc ']'\n+ cp -a /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet\n+ echo '{\"step\": 4}'\n+ export FACTER_deployment_type=containers\n+ FACTER_deployment_type=containers\n+ set +e\n+ puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file -e 'noop_resource('\\''package'\\''); include ::tripleo::profile::base::neutron::dhcp_agent_wrappers'\nWarning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)\nError: Evaluation Error: Error while evaluating a Function Call, Could not find class ::tripleo::profile::base::neutron::dhcp_agent_wrappers for undercloud-0.redhat.local (line: 1, column: 27) on node undercloud-0.redhat.local\n+ rc=1\n+ set -e\n+ set +ux\n\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=create_keepalived_wrapper', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=create_keepalived_wrapper', '--format', '{{.Names}}']\"\nError running ['podman', 'run', '--name', 'create_keepalived_wrapper', '--label', 'config_id=tripleo_step2', '--label', 'container_name=create_keepalived_wrapper', '--label', 'managed_by=tripleo-Undercloud', '--label', 'config_data={\"command\": [\"/container_puppet_apply.sh\", \"4\", \"file\", \"include ::tripleo::profile::base::neutron::l3_agent_wrappers\"], \"detach\": false, \"environment\": {\"TRIPLEO_DEPLOY_IDENTIFIER\": \"1583369143\"}, \"image\": \"undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-l3-agent:20200303.1\", \"net\": \"host\", \"pid\": \"host\", \"start_order\": 1, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro\", \"/run/openvswitch:/run/openvswitch:shared,z\", \"/var/lib/neutron:/var/lib/neutron:shared,z\"]}', '--conmon-pidfile=/var/run/create_keepalived_wrapper.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/create_keepalived_wrapper.log', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1583369143', '--net=host', '--pid=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro', '--volume=/etc/puppet:/tmp/puppet-etc:ro', '--volume=/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro', '--volume=/run/openvswitch:/run/openvswitch:shared,z', '--volume=/var/lib/neutron:/var/lib/neutron:shared,z', '--cpuset-cpus=0,1,2,3,4,5,6,7', 'undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-l3-agent:20200303.1', '/container_puppet_apply.sh', '4', 'file', 'include ::tripleo::profile::base::neutron::l3_agent_wrappers']. [1]\n\

Log File: http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-upgrade-upgrade-15-16_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-HA/12/undercloud-0.tar.gz?undercloud-0/home/stack/undercloud_upgrade.log

Job logs: http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-upgrade-upgrade-15-16_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-HA/12/
 
Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Run the Ci job: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/upgrades/view/upgrade/job/DFG-upgrades-upgrade-upgrade-15-16_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-HA/
2.
3.

Actual results:


Expected results:


Additional info: