Bug 1767071

Description of problem:
This is a similar issue as https://bugzilla.redhat.com/show_bug.cgi?id=1757979 (haproxy) except this time it is for neutron_dhcp_agent. TLDR: podman 1.4.2 enforces the fact that you may not have twice the same bind mount (previous podman versions did not).

Alistair had a job fail with:
RED HAT CONFIDENTIAL

I am seeing this problem on OSP15 and RHEL8.0 deployment.

https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/hardware_provisioning/view/rqci/job/DFG-hardware_provisioning-rqci-15_director-8.0-virthost-3cont_1comp_3ceph_1ipa_2ironic-ipv4-geneve-IR-OC_Ironic_TLS_EW-20190627-0807/15/

From the downloadable file at:

https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/hardware_provisioning/view/rqci/job/DFG-hardware_provisioning-rqci-15_director-8.0-virthost-3cont_1comp_3ceph_1ipa_2ironic-ipv4-geneve-IR-OC_Ironic_TLS_EW-20190627-0807/15/artifact/undercloud-0.tar.gz

in var/lib/mistral/overcloud/ansible.log:
All three controllers see the issue in step4 controller startup on multiple containers.

2019-10-25 09:43:59,904 p=890 u=mistral |  TASK [Debug output for task: Start containers for step 4] **********************
2019-10-25 09:43:59,904 p=890 u=mistral |  Friday 25 October 2019  09:43:59 +0000 (0:03:06.393)       0:45:02.610 ******** 
2019-10-25 09:44:00,040 p=890 u=mistral |  fatal: [controller-0]: FAILED! => {

 "b'Error: /etc/ipa/ca.crt: duplicate mount destination\\n'

2019-10-25 09:44:00,178 p=890 u=mistral |  fatal: [controller-1]: FAILED! => {


I have anther TLS everywhere job in OSP15 that I have yet to dig through but a cursory glance seems to show an identical failure point and timeline.


And we can see the double ca.crt mount point here:

2019-10-25 09:43:06.168 117719 DEBUG paunch [  ] $ podman create --name neutron_dhcp --label config_id=tripleo_step4 --label container_name=neutron_dhcp --label managed_by=paunch --label config_data={"depends_on": ["openvswitch"], "environment": ["KOLLA_CONFIG_STRATEGY=COPY_ALWAYS", "TRIPLEO_CONFIG_HASH=553e07d2d41900907a6cb501d10e15af"], "healthcheck": {"test": "/openstack/healthcheck 5672"}, "image": "192.168.24.1:8787/rh-osbs/rhosp15-openstack-neutron-dhcp-agent:20191022.1", "net": "host", "pid": "host", "privileged": true, "restart": "always", "security_opt": "label=disable", "start_order": 10, "ulimit": ["nofile=16384"], "volumes": ["/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro", "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro", "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro", "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro", "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro", "/etc/puppet:/etc/puppet:ro", "/var/log/containers/neutron:/var/log/neutron:z", "/var/lib/kolla/config_files/neutron_dhcp.json:/var/lib/kolla/config_files/config.json:ro", "/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro", "/lib/modules:/lib/modules:ro", "/run/openvswitch:/run/openvswitch:shared,z", "/var/lib/neutron:/var/lib/neutron:shared,z", "/run/netns:/run/netns:shared", "/var/lib/neutron/dnsmasq_wrapper:/usr/local/bin/dnsmasq:ro", "/var/lib/neutron/dhcp_haproxy_wrapper:/usr/local/bin/haproxy:ro", "/etc/pki/tls/certs/neutron.crt:/var/lib/kolla/config_files/src-tls/etc/pki/tls/certs/neutron.crt:ro", "/etc/pki/tls/private/neutron.key:/var/lib/kolla/config_files/src-tls/etc/pki/tls/private/neutron.key:ro", "/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro"]} --conmon-pidfile=/var/run/neutron_dhcp.pid --detach=true --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/neutron_dhcp.log --env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS --env=TRIPLEO_CONFIG_HASH=553e07d2d41900907a6cb501d10e15af --net=host --pid=host --ulimit=nofile=16384 --privileged=true --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/containers/neutron:/var/log/neutron:z --volume=/var/lib/kolla/config_files/neutron_dhcp.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro --volume=/lib/modules:/lib/modules:ro --volume=/run/openvswitch:/run/openvswitch:shared,z --volume=/var/lib/neutron:/var/lib/neutron:shared,z --volume=/run/netns:/run/netns:shared --volume=/var/lib/neutron/dnsmasq_wrapper:/usr/local/bin/dnsmasq:ro --volume=/var/lib/neutron/dhcp_haproxy_wrapper:/usr/local/bin/haproxy:ro --volume=/etc/pki/tls/certs/neutron.crt:/var/lib/kolla/config_files/src-tls/etc/pki/tls/certs/neutron.crt:ro --volume=/etc/pki/tls/private/neutron.key:/var/lib/kolla/config_files/src-tls/etc/pki/tls/private/neutron.key:ro --volume=/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro --security-opt=label=disable --cpuset-cpus=0,1,2,3 192.168.24.1:8787/rh-osbs/rhosp15-openstack-neutron-dhcp-agent:20191022.1