Bug 1794846

Summary: OSP DCN 13 minor update from z9 to z10 failed on OC nodes update "stderr: /usr/bin/docker-current: Error response from daemon..."
Product: Red Hat OpenStack Reporter: Yuri Obshansky <yobshans>
Component: openstack-tripleo-commonAssignee: Adriano Petrich <apetrich>
Status: CLOSED DUPLICATE QA Contact: Alexander Chuzhoy <sasha>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 13.0 (Queens)CC: bdobreli, mburns, slinaber
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-27 09:35:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
overcloud_update_prepare.log
none
overcloud_update_run_Controller0.log
none
overcloud_update_run_Compute0.log
none
overcloud_update_run_Compute1.log
none
overcloud_update_run_Compute2.log none

Description Yuri Obshansky 2020-01-24 21:10:24 UTC
Created attachment 1655147 [details]
overcloud_update_prepare.log

Description of problem:
Minor update of Openstack DCN 13 from z9 up to z10 failed on Overcloud nodes update. 
Most errors like:
"Error running ['docker', 'run', '--name', 'logrotate_crond', '--label', 'config_id=tripleo_step4', '--label', 'cont
ainer_name=logrotate_crond', '--label', 'managed_by=paunch', '--label', 'config_data={\"image\": \"192.168.24.1:8787/rhosp13/openstack-cron:2019-1
2-12.1\", \"pid\": \"host\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\", \"TRIPLEO_CONFIG_HASH=ab5b437b824ad12f21cbb36e626feea1\"], \
"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-tru
st/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs
/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/ce
rt.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/kolla/con
fig_files/logrotate-crond.json:/var/lib/kolla/config_files/config.json:ro\", \"/var/lib/config-data/puppet-generated/crond/:/var/lib/kolla/config_
files/src:ro\", \"/var/log/containers:/var/log/containers\"], \"net\": \"none\", \"privileged\": true, \"restart\": \"always\"}', '--detach=true',
 '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=TRIPLEO_CONFIG_HASH=ab5b437b824ad12f21cbb36e626feea1', '--net=none', '--pid=host', '--privilege
d=true', '--restart=always', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/c
a-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc
/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.t
rust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh
/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/kolla/config_files/logrotate-crond.json:/var/lib/kolla/config_file
s/config.json:ro', '--volume=/var/lib/config-data/puppet-generated/crond/:/var/lib/kolla/config_files/src:ro', '--volume=/var/log/containers:/var/
log/containers', '192.168.24.1:8787/rhosp13/openstack-cron:2019-12-12.1']. [125]", 
2020-01-24 13:50:06 |         "stdout: 31ac33273aa61c24ef9c6ed10d9bba40c0c31cf31cdf910f48dc16d620d0732f", 
2020-01-24 13:50:06 |         "stderr: /usr/bin/docker-current: Error response from daemon: service endpoint with name logrotate_crond already exi
sts.", 

for different docker images
Detailed logs attached bug

Version-Release number of selected component (if applicable):
OSP 13 Minor update from 
z9 - 2019-11-04.1
z10 - 2019-12-13.1

How reproducible:
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/edge/job/DFG-edge-day2-13-rhel-virthost-3cont-2comp-2leafs-x-2comp-update/5/
or
https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/edge/job/DFG-edge-day2-13-rhel-virthost-3cont-2comp-2leafs-x-2comp-update/1/

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Yuri Obshansky 2020-01-24 21:11:09 UTC
Created attachment 1655148 [details]
overcloud_update_run_Controller0.log

Comment 2 Yuri Obshansky 2020-01-24 21:11:38 UTC
Created attachment 1655149 [details]
overcloud_update_run_Compute0.log

Comment 3 Yuri Obshansky 2020-01-24 21:12:13 UTC
Created attachment 1655150 [details]
overcloud_update_run_Compute1.log

Comment 4 Yuri Obshansky 2020-01-24 21:12:43 UTC
Created attachment 1655151 [details]
overcloud_update_run_Compute2.log

Comment 5 Bogdan Dobrelya 2020-01-27 09:35:32 UTC

*** This bug has been marked as a duplicate of bug 1790792 ***