Created attachment 1652302 [details]
ansible log

Description of problem:
OSP 16 DCN with Multi-stack and Spine-Leaf network topology
Deployment of stack dcn1 (3 HCI nodes) failed with error:
"Error: error creating container storage: the container name "container-puppet-nova_libvirt" is already in use by "ddddb3cc59933cf1bd71644f8e575c09eea275f758d70bc5a00934044725f06b". You have to remove that container to be able to reuse that name.: that name is already in use"


Version-Release number of selected component (if applicable):
RHOS_TRUNK-16.0-RHEL-8-20200109.n.1
podman version 1.4.2-stable3

How reproducible:
Deploy OSP 16 DCN with Multi-stack and Spine-Leaf network topology
Deploy central stack with 3 Controller nodes and 2 Compute nodes
Start deploy dcn1 stack with 3 HCI nodes
Templates can be found
https://code.engineering.redhat.com/gerrit/gitweb?p=rhos-infrared.git;a=tree;f=settings/installer/ospd/deployment/edge/osp-16-spine-leaf-multistack-hci;h=629719da84b1b5388e3be77393d4ed3cc66283cf;hb=HEAD

I'll hold environment for a while for an investigation
Please, ping me on IRCC #edge #rhos-qe

Steps to Reproduce:
1.
2.
3.

Actual results:
More details in attached ansible.log.gz

Expected results:
Should be passed

Additional info:
Restart of HCI node did not help failed with the same error:
[root@dcn1-computehci1-2 ~]# sudo podman run --user 0 --name container-puppet-nova_libvirt --env PUPPET_TAGS=file,file_line,concat,augeas,cron,nova_config,nova_paste_api_ini,libvirtd_config,nova_config,file,libvirt_tls_password --env NAME=nova_libvirt --env HOSTNAME=dcn1-computehci1-2 --env NO_ARCHIVE= --env STEP=6 --env NET_HOST=true --env DEBUG=False --volume /etc/localtime:/etc/localtime:ro --volume /tmp/tmptxcjlr00:/etc/config.pp:ro --volume /etc/puppet/:/tmp/puppet-etc/:ro --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /var/lib/config-data:/var/lib/config-data/:rw --volume /var/lib/container-puppet/puppetlabs/facter.conf:/etc/puppetlabs/facter/facter.conf:ro --volume /var/lib/container-puppet/puppetlabs/:/opt/puppetlabs/:ro --volume /dev/log:/dev/log:rw --rm --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/container-puppet-nova_libvirt.log --security-opt label=disable --volume /usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro --entrypoint /var/lib/container-puppet/container-puppet.sh --net host --volume /etc/hosts:/etc/hosts:ro --volume /var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro site-undercloud-0.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-compute:20200107.2
Error: error creating container storage: the container name "container-puppet-nova_libvirt" is already in use by "ddddb3cc59933cf1bd71644f8e575c09eea275f758d70bc5a00934044725f06b". You have to remove that container to be able to reuse that name.: that name is already in use

On another HIC node it failed with another error:
[root@dcn1-computehci1-0 ~]# sudo podman run --user 0 --name container-puppet-nova_libvirt --env PUPPET_TAGS=file,file_line,concat,augeas,cron,nova_config,nova_paste_api_ini,libvirtd_config,nova_config,file,libvirt_tls_password --env NAME=nova_libvirt --env HOSTNAME=dcn1-computehci1-2 --env NO_ARCHIVE= --env STEP=6 --env NET_HOST=true --env DEBUG=False --volume /etc/localtime:/etc/localtime:ro --volume /tmp/tmptxcjlr00:/etc/config.pp:ro --volume /etc/puppet/:/tmp/puppet-etc/:ro --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /var/lib/config-data:/var/lib/config-data/:rw --volume /var/lib/container-puppet/puppetlabs/facter.conf:/etc/puppetlabs/facter/facter.conf:ro --volume /var/lib/container-puppet/puppetlabs/:/opt/puppetlabs/:ro --volume /dev/log:/dev/log:rw --rm --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/container-puppet-nova_libvirt.log --security-opt label=disable --volume /usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro --entrypoint /var/lib/container-puppet/container-puppet.sh --net host --volume /etc/hosts:/etc/hosts:ro --volume /var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro site-undercloud-0.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-compute:20200107.2
+ mkdir -p /etc/puppet
+ cp -dR /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet
+ rm -Rf /etc/puppet/ssl
+ echo '{"step": 6}'
+ TAGS=
+ '[' -n file,file_line,concat,augeas,cron,nova_config,nova_paste_api_ini,libvirtd_config,nova_config,file,libvirt_tls_password ']'
+ TAGS='--tags "file,file_line,concat,augeas,cron,nova_config,nova_paste_api_ini,libvirtd_config,nova_config,file,libvirt_tls_password"'
+ CHECK_MODE=
+ '[' -d /tmp/puppet-check-mode ']'
+ conf_data_path=/var/lib/config-data/nova_libvirt
+ origin_of_time=/var/lib/config-data/nova_libvirt.origin_of_time
+ touch /var/lib/config-data/nova_libvirt.origin_of_time
+ sync
+ export NET_HOST=true
+ NET_HOST=true
+ set +e
+ '[' true == false ']'
+ export FACTER_deployment_type=containers
+ FACTER_deployment_type=containers
++ cat /sys/class/dmi/id/product_uuid
++ tr '[:upper:]' '[:lower:]'
+ export FACTER_uuid=a38bcc1f-b634-48f8-ad80-ee8419461b12
+ FACTER_uuid=a38bcc1f-b634-48f8-ad80-ee8419461b12
Running puppet
+ echo 'Running puppet'
+ set -x
+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,nova_config,nova_paste_api_ini,libvirtd_config,nova_config,file,libvirt_tls_password"' /etc/config.pp
+ logger -s -t puppet-user
<13>Jan 14 20:58:31 puppet-user: Notice: Compiled catalog for dcn1-computehci1-0.redhat.local in environment production in 0.01 seconds
<13>Jan 14 20:58:40 puppet-user: Notice: Applied catalog in 0.01 seconds
<13>Jan 14 20:58:40 puppet-user: Changes:
<13>Jan 14 20:58:40 puppet-user: Events:
<13>Jan 14 20:58:40 puppet-user: Resources:
<13>Jan 14 20:58:40 puppet-user:           Skipped: 7
<13>Jan 14 20:58:40 puppet-user:             Total: 7
<13>Jan 14 20:58:40 puppet-user: Time:
<13>Jan 14 20:58:40 puppet-user:    Transaction evaluation: 0.00
<13>Jan 14 20:58:40 puppet-user:    Catalog application: 0.01
<13>Jan 14 20:58:40 puppet-user:    Config retrieval: 0.07
<13>Jan 14 20:58:40 puppet-user:          Last run: 1579035520
<13>Jan 14 20:58:40 puppet-user:             Total: 0.01
<13>Jan 14 20:58:40 puppet-user: Version:
<13>Jan 14 20:58:40 puppet-user:            Config: 1579035520
<13>Jan 14 20:58:40 puppet-user:            Puppet: 5.5.10
+ rc=0
+ '[' False = false ']'
+ set -e
+ '[' 0 -ne 2 -a 0 -ne 0 ']'
+ verbosity=
+ '[' False = false ']'
+ verbosity=-v
+ '[' -z '' ']'
+ archivedirs=("/etc" "/root" "/opt" "/var/lib/ironic/tftpboot" "/var/lib/ironic/httpboot" "/var/www" "/var/spool/cron" "/var/lib/nova/.ssh")
+ rsync_srcs=
+ for d in "${archivedirs[@]}"
+ '[' -d /etc ']'
+ rsync_srcs+=' /etc'
+ for d in "${archivedirs[@]}"
+ '[' -d /root ']'
+ rsync_srcs+=' /root'
+ for d in "${archivedirs[@]}"
+ '[' -d /opt ']'
+ rsync_srcs+=' /opt'
+ for d in "${archivedirs[@]}"
+ '[' -d /var/lib/ironic/tftpboot ']'
+ for d in "${archivedirs[@]}"
+ '[' -d /var/lib/ironic/httpboot ']'
+ for d in "${archivedirs[@]}"
+ '[' -d /var/www ']'
+ for d in "${archivedirs[@]}"
+ '[' -d /var/spool/cron ']'
+ rsync_srcs+=' /var/spool/cron'
+ for d in "${archivedirs[@]}"
+ '[' -d /var/lib/nova/.ssh ']'
+ rsync_srcs+=' /var/lib/nova/.ssh'
+ password_files=/root/.my.cnf
+ exclude_files=
+ for p in $password_files
+ '[' -f /root/.my.cnf -a -f /var/lib/config-data/nova_libvirt/root/.my.cnf ']'
+ ro_files='/etc/puppetlabs/ /opt/puppetlabs/'
+ for ro in $ro_files
+ '[' -e /etc/puppetlabs/ ']'
+ exclude_files+=' --exclude=/etc/puppetlabs/'
+ for ro in $ro_files
+ '[' -e /opt/puppetlabs/ ']'
+ exclude_files+=' --exclude=/opt/puppetlabs/'
+ echo 'Evaluating config files to be removed for the nova_libvirt configuration'
Evaluating config files to be removed for the nova_libvirt configuration
++ mktemp /tmp/tmp.XXXXXXXXXX
+ TMPFILE=/tmp/tmp.Ncq0OwzANS
++ mktemp /tmp/tmp.XXXXXXXXXX
+ TMPFILE2=/tmp/tmp.V4VHXKk2Dn
+ trap 'rm -rf $TMPFILE $TMPFILE2' EXIT INT HUP
+ rsync -av -R --dry-run --delete-after --exclude=/etc/puppetlabs/ --exclude=/opt/puppetlabs/ /etc /root /opt /var/spool/cron /var/lib/nova/.ssh /var/lib/config-data/nova_libvirt
+ awk '/^deleting/ {print $2}'
Rsyncing config files from  /etc /root /opt /var/spool/cron /var/lib/nova/.ssh into /var/lib/config-data/nova_libvirt
+ echo 'Rsyncing config files from  /etc /root /opt /var/spool/cron /var/lib/nova/.ssh into /var/lib/config-data/nova_libvirt'
+ rsync -a -v -R --delay-updates --delete-after --exclude=/etc/puppetlabs/ --exclude=/opt/puppetlabs/ /etc /root /opt /var/spool/cron /var/lib/nova/.ssh /var/lib/config-data/nova_libvirt
building file list ... done
/etc/
/etc/hostname
/etc/login.defs
/etc/resolv.conf
/etc/config.pp/
/etc/libvirt/
/etc/libvirt/libvirtd.conf
/etc/libvirt/qemu.conf
/etc/nova/
/etc/nova/nova.conf
/etc/nova/migration/
/etc/nova/migration/authorized_keys
/etc/nova/migration/identity
/etc/puppet/
/etc/puppet/auth.conf
/etc/puppet/hiera.yaml
/etc/puppet/puppet.conf
/etc/puppet/hieradata/
/etc/puppet/hieradata/all_nodes.json
/etc/puppet/hieradata/bootstrap_node.json
/etc/puppet/hieradata/cloud_domain.json
/etc/puppet/hieradata/config_step.json
/etc/puppet/hieradata/docker_puppet.json
/etc/puppet/hieradata/extraconfig.json
/etc/puppet/hieradata/fqdn.json
/etc/puppet/hieradata/net_ip_map.json
/etc/puppet/hieradata/role_extraconfig.json
/etc/puppet/hieradata/service_configs.json
/etc/puppet/hieradata/service_names.json
/etc/puppet/hieradata/vip_data.json
/etc/puppet/modules/
/etc/puppet/ssl/
/etc/puppet/ssl/certificate_requests/
/etc/puppet/ssl/certs/
/etc/puppet/ssl/private/
/etc/puppet/ssl/private_keys/
/etc/puppet/ssl/public_keys/
/etc/sasl2/
/etc/sasl2/libvirt.conf
/etc/ssh/
/etc/ssh/sshd_config
/opt/
/var/lib/
/var/lib/nova/.ssh/config
deleting etc/my.cnf.d/tripleo.cnf
deleting etc/my.cnf.d/
deleting etc/libvirt/passwd.db
deleting etc/nova/secret.xml

sent 393,222 bytes  received 832 bytes  788,108.00 bytes/sec
total size is 21,175,932  speedup is 53.74
++ stat -c %y /var/lib/config-data/nova_libvirt.origin_of_time
Gathering files modified after 2020-01-14 20:58:31.955102214 +0000
+ echo 'Gathering files modified after 2020-01-14 20:58:31.955102214 +0000'
+ puppet_generated_path=/var/lib/config-data/puppet-generated/nova_libvirt
+ mkdir -p /var/lib/config-data/puppet-generated/nova_libvirt
Ensuring the removed config files are also purged in /var/lib/config-data/puppet-generated/nova_libvirt:
+ echo 'Ensuring the removed config files are also purged in /var/lib/config-data/puppet-generated/nova_libvirt:'
+ cat /tmp/tmp.Ncq0OwzANS
+ sort
etc/libvirt/passwd.db
etc/my.cnf.d/
etc/my.cnf.d/tripleo.cnf
etc/nova/secret.xml
+ cat /tmp/tmp.Ncq0OwzANS
+ xargs -n1 -r '-I{}' bash -c 'rm -f /var/lib/config-data/puppet-generated/nova_libvirt/{}'
rm: cannot remove '/var/lib/config-data/puppet-generated/nova_libvirt/etc/my.cnf.d/': Is a directory
+ rm -rf /tmp/tmp.Ncq0OwzANS /tmp/tmp.V4VHXKk2Dn