Bug 1656409
| Summary: | [UPDATE] Old openstack-neutron-metadata-agent-ovn container still running after update | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Eran Kuris <ekuris> |
| Component: | openstack-tripleo | Assignee: | James Slagle <jslagle> |
| Status: | CLOSED NOTABUG | QA Contact: | Arik Chernetsky <achernet> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 14.0 (Rocky) | CC: | amuller, bcafarel, dalvarez, jfrancoa, mburns, rbartal |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-12-10 11:03:54 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1637626 | ||
After minor update the old openstack-neutron-metadata-agent-ovn continer with tag:2018-11-29.1 is still running. The new container is also running on the same node with tag 2018-11-30.2 from log file: 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-11-30.2 present in /home/stack/composable_roles/docker-images.yaml 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-11-29.1 not present in /home/stack/composable_roles/docker-images.yaml @Jose, does this ring a bell to you? In the minor update we did last week we didn’t hit such a failure. Perhaps it’s infrared related? After a double check, it looks like the sidecar container hasn’t been taken into account by the update process. [root@compute-0 ~]# docker ps -a | grep ovn fb3cf0030a47 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-11-22.1 "ip netns exec ovn..." 15 minutes ago Up 15 minutes neutron-haproxy-ovnmeta-e718694b-87f7-46bf-bb64-0446cb6325a3 95a2fdf60bb2 192.168.24.1:8787/rhosp14/openstack-ovn-controller:2018-12-07.1 "kolla_start" About an hour ago Up About an hour ovn_controller cace050f7461 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-12-07.1 "kolla_start" About an hour ago Up About an hour (healthy) ovn_metadata_agent 0ca0a4ac2e49 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-12-07.1 "/docker_puppet_ap..." About an hour ago Exited (0) About an hour ago setup_ovs_manager b74280a053aa 192.168.24.1:8787/rhosp14/openstack-ovn-controller:2018-12-07.1 "/bin/bash -c 'CMS..." About an hour ago Exited (0) About an hour ago configure_cms_options 96eb5976d715 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-12-07.1 "/docker_puppet_ap..." About an hour ago Exited (0) About an hour ago create_haproxy_wrapper This BZ should be closed as not a bug, as it's a problem of the testing framework. The fault is in tripleo-upgrade, which is not a deliverable. It's a set of ansible tasks to automate the TripleO upgrade/update and which is being used in upstream and downstream testing. Therefore, a launchpad bug could be opened to track the issue, if wanted, submit the patch to fix the issue in tripleo-upgrade (the problem seems to be in the docker images validation script: https://github.com/openstack/tripleo-upgrade/blob/ba36d95c384a02734ca277a4facc2ae1506b4c08/templates/validate_docker_images_versions.sh.j2), but it shouldn't be considered as a blocker bug. Indeed, I filled https://bugs.launchpad.net/tripleo/+bug/1807687 and uploaded a tentative fix for review. Closing the BZ per last comments, we will track the tripleo-upgrade fix and get it in ASAP |
Description of problem: the ovn minor update failed : it looks like 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-11-29.1 not present in /home/stack/composable_roles/docker-images.yaml was not updated ASK [tripleo-upgrade : validate overcloud docker images/containers] *********** task path: /home/rhos-ci/jenkins/workspace/DFG-network-networking-ovn-update-14_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/infrared/plugins/tripleo-upgrade/infrared_plugin/roles/tripleo-upgrade/tasks/update/main.yml:122 Wednesday 05 December 2018 10:48:37 +0000 (0:00:00.047) 2:25:43.265 **** fatal: [undercloud-0]: FAILED! => { "changed": true, "cmd": "set -o pipefail\n source /home/stack/stackrc\n bash /home/stack/validate_docker_images_versions.sh 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S |\"), $0; fflush(); }' > /home/stack/validate_oc_images_containers.log", "delta": "0:00:17.854951", "end": "2018-12-05 05:48:55.694877", "rc": 2, "start": "2018-12-05 05:48:37.839926" } MSG: non-zero return code to retry, use: --limit @/home/rhos-ci/jenkins/workspace/DFG-network-networking-ovn-update-14_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/infrared/plugins/tripleo-upgrade/infrared_plugin/main.retry PLAY RECAP ********************************************************************* undercloud-0 : ok=72 change Version-Release number of selected component (if applicable): 2018-12-05 05:48:55 | ================================================================================ 2018-12-05 05:48:55 | Validate docker images at host 192.168.24.15 2018-12-05 05:48:55 | ================================================================================ 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-nova-compute:2018-11-30.2 present in /home/stack/composable_roles/docker-images.yaml 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-ovn-controller:2018-11-30.2 present in /home/stack/composable_roles/docker-images.yaml 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-11-30.2 present in /home/stack/composable_roles/docker-images.yaml 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-cron:2018-11-30.2 present in /home/stack/composable_roles/docker-images.yaml 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-ceilometer-compute:2018-11-30.2 present in /home/stack/composable_roles/docker-images.yaml 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-nova-compute:2018-11-30.2 present in /home/stack/composable_roles/docker-images.yaml 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-iscsid:2018-11-30.2 present in /home/stack/composable_roles/docker-images.yaml 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-nova-libvirt:2018-11-30.2 present in /home/stack/composable_roles/docker-images.yaml 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-nova-libvirt:2018-11-30.2 present in /home/stack/composable_roles/docker-images.yaml 2018-12-05 05:48:55 | Image 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-11-29.1 not present in /home/stack/composable_roles/docker-images.yaml How reproducible: 100% Steps to Reproduce: 1.run the ovn minor update job 2. 3. Actual results: Expected results: Additional info: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-update-14_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/28/console