Bug 1793634

Description of problem: Update from osp13 GA to 2020-01-15.3 fails during controller update:


2020-01-18 00:32:17 |         "Error running ['docker', 'run', '--name', 'neutron_dhcp', '--label', 'config_id=tripleo_step4', '--label', 'container_name=neutron_dhcp', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 10, \"ulimit\": [\"nofile=16384\"], \"healthcheck\": {\"test\": \"/openstack/healthcheck 5672\"}, \"image\": \"192.168.24.1:8787/rhosp13/openstack-neutron-dhcp-agent:2019-12-12.1rhel7.8\", \"pid\": \"host\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\", \"TRIPLEO_CONFIG_HASH=ee55e15e1aabdbdf501d3b59099c2f7b\"], \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/neutron:/var/log/neutron\", \"/var/lib/kolla/config_files/neutron_dhcp.json:/var/lib/kolla/config_files/config.json:ro\", \"/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro\", \"/lib/modules:/lib/modules:ro\", \"/run/openvswitch:/run/openvswitch\", \"/var/lib/neutron:/var/lib/neutron\", \"/run/netns:/run/netns:shared\", \"/var/lib/openstack:/var/lib/openstack\", \"/var/lib/neutron/dnsmasq_wrapper:/usr/local/bin/dnsmasq:ro\", \"/var/lib/neutron/dhcp_haproxy_wrapper:/usr/local/bin/haproxy:ro\"], \"net\": \"host\", \"privileged\": true, \"restart\": \"always\"}', '--detach=true', '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=TRIPLEO_CONFIG_HASH=ee55e15e1aabdbdf501d3b59099c2f7b', '--net=host', '--pid=host', '--ulimit=nofile=16384', '--health-cmd=/openstack/healthcheck 5672', '--privileged=true', '--restart=always', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/log/containers/neutron:/var/log/neutron', '--volume=/var/lib/kolla/config_files/neutron_dhcp.json:/var/lib/kolla/config_files/config.json:ro', '--volume=/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro', '--volume=/lib/modules:/lib/modules:ro', '--volume=/run/openvswitch:/run/openvswitch', '--volume=/var/lib/neutron:/var/lib/neutron', '--volume=/run/netns:/run/netns:shared', '--volume=/var/lib/openstack:/var/lib/openstack', '--volume=/var/lib/neutron/dnsmasq_wrapper:/usr/local/bin/dnsmasq:ro', '--volume=/var/lib/neutron/dhcp_haproxy_wrapper:/usr/local/bin/haproxy:ro', '192.168.24.1:8787/rhosp13/openstack-neutron-dhcp-agent:2019-12-12.1rhel7.8']. [125]", 
2020-01-18 00:32:17 |         "stdout: 57cea0fdb732ae7a4576bb3b0ddaa80d965b2b6c12b2442b78eec2b78068186a", 
2020-01-18 00:32:17 |         "stderr: /usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:89: jailing process inside rootfs caused \\\\\\\"pivot_root invalid argument\\\\\\\"\\\"\".", 
2020-01-18 00:32:17 |         "stdout: 5703439c28f08b667bcd664bee7655526d907ebcba018b1136e955a0218a580c", 
2020-01-18 00:32:17 |         "Error running ['docker', 'run', '--name', 'neutron_l3_agent', '--label', 'config_id=tripleo_step4', '--label', 'container_name=neutron_l3_agent', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 10, \"ulimit\": [\"nofile=16384\"], \"healthcheck\": {\"test\": \"/openstack/healthcheck 5672\"}, \"image\": \"192.168.24.1:8787/rhosp13/openstack-neutron-l3-agent:2019-12-12.1rhel7.8\", \"pid\": \"host\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\", \"TRIPLEO_CONFIG_HASH=ee55e15e1aabdbdf501d3b59099c2f7b\"], \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/neutron:/var/log/neutron\", \"/var/lib/kolla/config_files/neutron_l3_agent.json:/var/lib/kolla/config_files/config.json:ro\", \"/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro\", \"/lib/modules:/lib/modules:ro\", \"/run/openvswitch:/run/openvswitch\", \"/var/lib/neutron:/var/lib/neutron\", \"/run/netns:/run/netns:shared\", \"/var/lib/openstack:/var/lib/openstack\", \"/var/lib/neutron/keepalived_wrapper:/usr/local/bin/keepalived:ro\", \"/var/lib/neutron/l3_haproxy_wrapper:/usr/local/bin/haproxy:ro\", \"/var/lib/neutron/dibbler_wrapper:/usr/local/bin/dibbler_client:ro\"], \"net\": \"host\", \"privileged\": true, \"restart\": \"always\"}', '--detach=true', '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=TRIPLEO_CONFIG_HASH=ee55e15e1aabdbdf501d3b59099c2f7b', '--net=host', '--pid=host', '--ulimit=nofile=16384', '--health-cmd=/openstack/healthcheck 5672', '--privileged=true', '--restart=always', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/log/containers/neutron:/var/log/neutron', '--volume=/var/lib/kolla/config_files/neutron_l3_agent.json:/var/lib/kolla/config_files/config.json:ro', '--volume=/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro', '--volume=/lib/modules:/lib/modules:ro', '--volume=/run/openvswitch:/run/openvswitch', '--volume=/var/lib/neutron:/var/lib/neutron', '--volume=/run/netns:/run/netns:shared', '--volume=/var/lib/openstack:/var/lib/openstack', '--volume=/var/lib/neutron/keepalived_wrapper:/usr/local/bin/keepalived:ro', '--volume=/var/lib/neutron/l3_haproxy_wrapper:/usr/local/bin/haproxy:ro', '--volume=/var/lib/neutron/dibbler_wrapper:/usr/local/bin/dibbler_client:ro', '192.168.24.1:8787/rhosp13/openstack-neutron-l3-agent:2019-12-12.1rhel7.8']. [125]", 
2020-01-18 00:32:17 |         "stdout: 42057239f3375060e7804989fad0040df8d061b133544616f7bb971772ec1042", 
2020-01-18 00:32:18 |         "Error running ['docker', 'run', '--name', 'neutron_ovs_agent', '--label', 'config_id=tripleo_step4', '--label', 'container_name=neutron_ovs_agent', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 10, \"ulimit\": [\"nofile=16384\"],

2020-01-18 00:32:17 |         "stderr: /usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:89: jailing process inside rootfs caused \\\\\\\"pivot_root invalid argument\\\\\\\"\\\"\".", 
2020-01-18 00:32:17 |         "stdout: 5703439c28f08b667bcd664bee7655526d907ebcba018b1136e955a0218a580c", 
2020-01-18 00:32:17 |         "Error running ['docker', 'run', '--name', 'neutron_l3_agent', '--label', 'config_id=tripleo_step4', '--label', 'container_name=neutron_l3_agent', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 10, \"ulimit\": [\"nofile=16384\"], \"healthcheck\": {\"test\": \"/openstack/healthcheck 5672\"}, \"image\": \"192.168.24.1:8787/rhosp13/openstack-neutron-l3-agent:2019-12-12.1rhel7.8\", \"pid\": \"host\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\", \"TRIPLEO_CONFIG_HASH=ee55e15e1aabdbdf501d3b59099c2f7b\"], \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/neutron:/var/log/neutron\", \"/var/lib/kolla/config_files/neutron_l3_agent.json:/var/lib/kolla/config_files/config.json:ro\", \"/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro\", \"/lib/modules:/lib/modules:ro\", \"/run/openvswitch:/run/openvswitch\", \"/var/lib/neutron:/var/lib/neutron\", \"/run/netns:/run/netns:shared\", \"/var/lib/openstack:/var/lib/openstack\", \"/var/lib/neutron/keepalived_wrapper:/usr/local/bin/keepalived:ro\", \"/var/lib/neutron/l3_haproxy_wrapper:/usr/local/bin/haproxy:ro\", \"/var/lib/neutron/dibbler_wrapper:/usr/local/bin/dibbler_client:ro\"], \"net\": \"host\", \"privileged\": true, \"restart\": \"always\"}', '--detach=true', '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=TRIPLEO_CONFIG_HASH=ee55e15e1aabdbdf501d3b59099c2f7b', '--net=host', '--pid=host', '--ulimit=nofile=16384', '--health-cmd=/openstack/healthcheck 5672', '--privileged=true', '--restart=always', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/log/containers/neutron:/var/log/neutron', '--volume=/var/lib/kolla/config_files/neutron_l3_agent.json:/var/lib/kolla/config_files/config.json:ro', '--volume=/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro', '--volume=/lib/modules:/lib/modules:ro', '--volume=/run/openvswitch:/run/openvswitch', '--volume=/var/lib/neutron:/var/lib/neutron', '--volume=/run/netns:/run/netns:shared', '--volume=/var/lib/openstack:/var/lib/openstack', '--volume=/var/lib/neutron/keepalived_wrapper:/usr/local/bin/keepalived:ro', '--volume=/var/lib/neutron/l3_haproxy_wrapper:/usr/local/bin/haproxy:ro', '--volume=/var/lib/neutron/dibbler_wrapper:/usr/local/bin/dibbler_client:ro', '192.168.24.1:8787/rhosp13/openstack-neutron-l3-agent:2019-12-12.1rhel7.8']. [125]", 
2020-01-18 00:32:17 |         "stdout: 42057239f3375060e7804989fad0040df8d061b133544616f7bb971772ec1042", 
2020-01-18 00:32:18 |         "Error running ['docker', 'run', '--name', 'neutron_ovs_agent', '--label', 'config_id=tripleo_step4', '--label', 'container_name=neutron_ovs_agent', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 10, \"ulimit\": [\"nofile=16384\"], \"healthcheck\": {\"test\": \"/openstack/healthcheck 5672\"}, \"image\": \"192.168.24.1:8787/rhosp13/openstack-neutron-openvswitch-agent:2019-12-12.1rhel7.8\", \"pid\": \"host\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\", \"TRIPLEO_CONFIG_HASH=ee55e15e1aabdbdf501d3b59099c2f7b\"], \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/neutron:/var/log/neutron\", \"/var/lib/kolla/config_files/neutron_ovs_agent.json:/var/lib/kolla/config_files/config.json:ro\", \"/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro\", \"/var/lib/docker-config-scripts/neutron_ovs_agent_launcher.sh:/neutron_ovs_agent_launcher.sh:ro\", \"/lib/modules:/lib/modules:ro\", \"/run/openvswitch:/run/openvswitch\"], \"net\": \"host\", \"privileged\": true, \"restart\": \"always\"}', '--detach=true', '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=TRIPLEO_CONFIG_HASH=ee55e15e1aabdbdf501d3b59099c2f7b', '--net=host', '--pid=host', '--ulimit=nofile=16384', '--health-cmd=/openstack/healthcheck 5672', '--privileged=true', '--restart=always', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/log/containers/neutron:/var/log/neutron', '--volume=/var/lib/kolla/config_files/neutron_ovs_agent.json:/var/lib/kolla/config_files/config.json:ro', '--volume=/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro', '--volume=/var/lib/docker-config-scripts/neutron_ovs_agent_launcher.sh:/neutron_ovs_agent_launcher.sh:ro', '--volume=/lib/modules:/lib/modules:ro',Update failed with: Ansible failed, check log at /var/lib/mistral/c6d9ed35-36cb-4f7f-89bc-da77a60c87ca/ansible.log.
2020-01-18 00:32:18 |  '--volume=/run/openvswitch:/run/openvswitch', '192.168.24.1:8787/rhosp13/openstack-neutron-openvswitch-agent:2019-12-12.1rhel7.8']. [125]", 
2020-01-18 00:32:18 |         "stdout: dde42028c920df0548e9d77825bae548868cac129ec70f03f051dbdbb237d73a", 
2020-01-18 00:32:18 |         "stderr: /usr/bin/docker-current: Error response from daemon: service endpoint with name neutron_ovs_agent already exists."
2020-01-18 00:32:18 |     ]
2020-01-18 00:32:18 | }


Version-Release number of selected component (if applicable):

Containers are version 2019-12-12.1rhel7.8

docker-1.13.1-108.git4ef4b30.el7.x86_64

Rhel is updated to 7.8.

Not sure about the component, so tht by default.