Bug 1797889
| Summary: | compute node reboot takes too long because nova_libvirt times out during OSP16 update | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Eduardo Olivares <eolivare> | ||||||
| Component: | openstack-nova | Assignee: | Piotr Kopec <pkopec> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | OSP DFG:Compute <osp-dfg-compute> | ||||||
| Severity: | unspecified | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 16.0 (Train) | CC: | dasmith, eglynn, jhakimra, kchamart, sathlang, sbauza, sgordon, stephenfin, vromanso | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2020-02-25 14:00:14 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Eduardo Olivares
2020-02-04 07:19:24 UTC
Created attachment 1657483 [details]
compute-1 artifacts
This problems was reproduced twice with our update job: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-network-networking-ovn-update-16_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/52/ - update from RHOS_TRUNK-16.0-RHEL-8-20200122.n.1 to RHOS_TRUNK-16.0-RHEL-8-20200130.n.0 https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-network-networking-ovn-update-16_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/54/ - update from RHOS_TRUNK-16.0-RHEL-8-20200130.n.0 to RHOS_TRUNK-16.0-RHEL-8-20200131.n.0 But in latest execution, overcloud reboot worked correctly: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-network-networking-ovn-update-16_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/56/ - update from RHOS_TRUNK-16.0-RHEL-8-20200122.n.1 to RHOS_TRUNK-16.0-RHEL-8-20200131.n.0 So, we this data, what we have is "How reproducible: 66.6%" Hi Piotr,
so the command "podman images" only reflects what has been pulled on
the local node, while tripleo-container-image prepare update the
repository.
We currently have no user interface to see what's in the local repo,
beside looking directly into /var/lib/image-serve directly.
The real issue happens there in
undercloud-0/home/stack/overcloud_update_run_Controller.log
2020-02-20 22:05:46 | "$ podman stop nova_scheduler",
2020-02-20 22:05:46 | "b'fd2cf249eb4484ea424b3686df97e4b2603629aa039a3e5125fbe45036e83906\\n'",
2020-02-20 22:05:46 | "$ podman rm nova_scheduler",
2020-02-20 22:05:46 | "b'Error: cannot remove container fd2cf249eb4484ea424b3686df97e4b2603629aa039a3e5125fbe45036e83906 as it has active exec sessions: container state improper\\n'",
2020-02-20 22:05:46 | "Error removing container: nova_scheduler",
2020-02-20 22:05:46 | "Error: cannot remove container fd2cf249eb4484ea424b3686df97e4b2603629aa039a3e5125fbe45036e83906 as it has active exec sessions: container state improper",
and later:
2020-02-20 21:40:05 | "Running container: nova_scheduler",
2020-02-20 21:40:05 | "Skipping existing container: nova_scheduler",
so the nova_schedule is silently not updated.
We have two issues here:
1. paunch is unable to remove the container because it has some process still running
2. paunch doesn't raise an error when this happens.
1. is a solved in bz#1802573, as for 2., as paunch is going to be
replaced soon with an ansible module, not sure if it's going to be
solved.
Closing this one as a duplicate, to keep everything nice and tidy.
Thanks.
*** This bug has been marked as a duplicate of bug 1802573 ***
|