Bug 1779281

Summary: Overcloud Update timeout error
Product: Red Hat OpenStack Reporter: Bruna Bonguardo <bbonguar>
Component: rhosp-directorAssignee: RHOS Maint <rhos-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: abregman, cgoncalves, dbecker, jfrancoa, mbollo, mbultel, mburns, morazi, sathlang, sgolovat
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-17 15:42:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1743518    

Description Bruna Bonguardo 2019-12-03 16:34:51 UTC
OSP13 update jobs [1][2] are stuck for ~4 hours until they reach timeout.


Error message while stuck:

[stack@undercloud-0 ~]$ ps -ef | grep -i update
swift     2195     1  0 08:29 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-object-updater /etc/swift/object-server.conf
swift     2241     1  0 08:29 ?        00:00:05 /usr/bin/python2 /usr/bin/swift-container-updater /etc/swift/container-server.conf
stack    19681 19676  0 11:05 pts/0    00:00:00 /bin/sh -c set -o pipefail  bash /home/stack/overcloud_update_run-Compute.sh 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S |"), $0; fflush(); }' > /home/stack/overcloud_update_run_Compute.log
stack    19682 19681  0 11:05 pts/0    00:00:00 bash /home/stack/overcloud_update_run-Compute.sh
stack    19687 19682  0 11:05 pts/0    00:00:01 /usr/bin/python2 /usr/bin/openstack overcloud update run --stack overcloud --nodes Compute --playbook all
mistral  19729  2203 29 11:05 ?        00:01:24 /usr/bin/python2 /bin/ansible-playbook -v /var/lib/mistral/eb97ea43-118c-4fe7-8ce0-c65d66eeb7b8/update_steps_playbook.yaml --limit Compute --module-path /usr/share/ansible-modules --user heat-admin --become --become-user root --inventory-file /var/lib/mistral/eb97ea43-118c-4fe7-8ce0-c65d66eeb7b8/inventory.yaml --private-key /var/lib/mistral/eb97ea43-118c-4fe7-8ce0-c65d66eeb7b8/ssh_private_key
mistral  22466 19729  0 11:09 ?        00:00:00 /usr/bin/python2 /bin/ansible-playbook -v /var/lib/mistral/eb97ea43-118c-4fe7-8ce0-c65d66eeb7b8/update_steps_playbook.yaml --limit Compute --module-path /usr/share/ansible-modules --user heat-admin --become --become-user root --inventory-file /var/lib/mistral/eb97ea43-118c-4fe7-8ce0-c65d66eeb7b8/inventory.yaml --private-key /var/lib/mistral/eb97ea43-118c-4fe7-8ce0-c65d66eeb7b8/ssh_private_key
mistral  22468 19729  0 11:09 ?        00:00:00 /usr/bin/python2 /bin/ansible-playbook -v /var/lib/mistral/eb97ea43-118c-4fe7-8ce0-c65d66eeb7b8/update_steps_playbook.yaml --limit Compute --module-path /usr/share/ansible-modules --user heat-admin --become --become-user root --inventory-file /var/lib/mistral/eb97ea43-118c-4fe7-8ce0-c65d66eeb7b8/inventory.yaml --private-key /var/lib/mistral/eb97ea43-118c-4fe7-8ce0-c65d66eeb7b8/ssh_private_key
stack    22651 22500  0 11:10 pts/1    00:00:00 grep --color=auto -i update
[stack@undercloud-0 ~]$ 


[1] https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/octavia/job/DFG-network-octavia-update-13_director-rhel-virthost-3cont_2comp_3ceph_1ipa-ipv4-vxlan-tls/19/
[2] https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/octavia/job/DFG-network-octavia-update-13_director-rhel-virthost-3cont_2comp_3ceph_1ipa-ipv4-vxlan-tls/20/

Comment 4 Sofer Athlan-Guyot 2019-12-09 13:47:44 UTC
Hi,

logs are not available and current osp13 jobs in update are (mostly) working, so without any further information we cannot handle.

Thanks,

Comment 7 Sofer Athlan-Guyot 2020-03-17 15:42:23 UTC
Hi,

closing this as it's old.  Please open an new issue if your update job is currently failing.