Description of problem: When doing a yum update from the GA release of Director to the latest poodle (or latest puddle). yum is hanged on the cleanup of nova-compute package. No timeout appear with ssh or yum ... still waiting infinitely.. (I let it wait for few hours...) Reproducible : 100% The workaround is to stop services : - openstack-nova-api.service - openstack-nova-compute.service - openstack-ironic-api.service - openstack-ironic-discoverd.service - openstack-ironic-conductor.service
With any luck this will be fixed by the integration between yum and puppet that we already have planned. Either way, we'll need to retest once that stuff has landed.
Verified as follows. No hang of yum process was observed. Logs ===== [stack@instack ~]$ yum list installed | grep openstack-tripleo openstack-tripleo.noarch 0.0.7-0.1.1664e566.el7ost openstack-tripleo-common.noarch 0.0.1.dev6-1.git49b57eb.el7ost openstack-tripleo-heat-templates.noarch openstack-tripleo-image-elements.noarch openstack-tripleo-puppet-elements.noarch [stack@instack ~]$ rpm -qv openstack-tripleo-heat-templates.noarch openstack-tripleo-heat-templates-0.8.6-46.el7ost.noarch [root@rhos-compute-node-02 ~]# rpm -ivh http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm Retrieving http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm Preparing... ################################# [100%] file /usr/bin/rhos-release from install of rhos-release-0.69-1.noarch conflicts with file from package rhos-release-0.68-1.noarch [root@instack ~]# [root@instack ~]# [root@instack ~]# yum update -y Resolving Dependencies --> Running transaction check ---> Package diskimage-builder.noarch 0:0.1.46-3.el7ost will be updated ---> Package diskimage-builder.noarch 0:0.1.47-dev3.el7.centos will be an update ---> Package instack.noarch 0:0.0.7-1.el7ost will be updated ---> Package instack.noarch 0:0.0.8-dev1.el7.centos will be an update ---> Package openstack-heat-templates.noarch 0:0-0.6.20150605git.el7ost will be updated ---> Package openstack-heat-templates.noarch 0:0.0.0-post350.el7.centos will be an update ---> Package openstack-tripleo-heat-templates.noarch 0:0.8.6-46.el7ost will be updated ---> Package openstack-tripleo-heat-templates.noarch 0:0.8.7-dev187.el7.centos will be an update ---> Package openstack-tripleo-image-elements.noarch 0:0.9.6-6.el7ost will be updated ---> Package openstack-tripleo-image-elements.noarch 0:0.9.7-dev23.el7.centos will be an update ---> Package openstack-tripleo-puppet-elements.noarch 0:0.0.1-4.el7ost will be updated ---> Package openstack-tripleo-puppet-elements.noarch 0:0.0.2-dev6.el7.centos will be an update ---> Package openstack-tuskar.noarch 0:0.4.18-3.el7ost will be updated ---> Package openstack-tuskar.noarch 0:2013.2.0-dev11.el7.centos will be an update ---> Package openstack-tuskar-ui.noarch 0:0.3.0-15.el7ost will be updated ---> Package openstack-tuskar-ui.noarch 0:2013.2.0-dev57.el7.centos will be an update ---> Package openstack-tuskar-ui-extras.noarch 0:0.0.4-1.el7ost will be updated ---> Package openstack-tuskar-ui-extras.noarch 0:2014.1-dev22.el7.centos will be an update ---> Package os-cloud-config.noarch 0:0.2.8-5.el7ost will be updated ---> Package os-cloud-config.noarch 0:0.2.10-dev12.el7.centos will be an update ---> Package python-hardware.noarch 0:0.14-6.el7ost will be updated ---> Package python-hardware.noarch 0:0.14.1-dev1.el7.centos will be an update ---> Package python-ironicclient.noarch 0:0.5.1-10.el7ost will be updated ---> Package python-ironicclient.noarch 0:0.5.2-dev12.el7.centos will be an update ---> Package python-openstackclient.noarch 0:1.0.3-2.el7ost will be updated ---> Package python-openstackclient.noarch 0:1.0.5-dev2.el7.centos will be an update ---> Package python-tuskarclient.noarch 0:0.1.18-4.el7ost will be updated ---> Package python-tuskarclient.noarch 0:0.1.19-dev16.el7.centos will be an update --> Finished Dependency Resolution Dependencies Resolved ================================================================================================================================================== Package Arch Version Repository Size ================================================================================================================================================== Updating: diskimage-builder noarch 0.1.47-dev3.el7.centos delorean-rdo-management 237 k instack noarch 0.0.8-dev1.el7.centos delorean-rdo-management 27 k openstack-heat-templates noarch 0.0.0-post350.el7.centos delorean-rdo-management 68 k openstack-tripleo-heat-templates noarch 0.8.7-dev187.el7.centos delorean-rdo-management 102 k openstack-tripleo-image-elements noarch 0.9.7-dev23.el7.centos delorean-rdo-management 438 k openstack-tripleo-puppet-elements noarch 0.0.2-dev6.el7.centos delorean-rdo-management 29 k openstack-tuskar noarch 2013.2.0-dev11.el7.centos delorean-rdo-management 280 k openstack-tuskar-ui noarch 2013.2.0-dev57.el7.centos delorean-rdo-management 241 k openstack-tuskar-ui-extras noarch 2014.1-dev22.el7.centos delorean-rdo-management 45 k os-cloud-config noarch 0.2.10-dev12.el7.centos delorean-rdo-management 92 k python-hardware noarch 0.14.1-dev1.el7.centos delorean-rdo-management 4.7 M python-ironicclient noarch 0.5.2-dev12.el7.centos delorean-rdo-management 183 k python-openstackclient noarch 1.0.5-dev2.el7.centos delorean-rdo-management 207 k python-tuskarclient noarch 0.1.19-dev16.el7.centos delorean-rdo-management 122 k Transaction Summary ================================================================================================================================================== Upgrade 14 Packages Total size: 6.7 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : diskimage-builder-0.1.47-dev3.el7.centos.noarch 1/28 Updating : python-tuskarclient-0.1.19-dev16.el7.centos.noarch 2/28 Updating : openstack-tripleo-heat-templates-0.8.7-dev187.el7.centos.noarch 3/28 Updating : os-cloud-config-0.2.10-dev12.el7.centos.noarch 4/28 Updating : python-ironicclient-0.5.2-dev12.el7.centos.noarch 5/28 Updating : openstack-tuskar-ui-2013.2.0-dev57.el7.centos.noarch 6/28 Updating : openstack-tuskar-ui-extras-2014.1-dev22.el7.centos.noarch 7/28 Updating : openstack-tuskar-2013.2.0-dev11.el7.centos.noarch 8/28 Updating : instack-0.0.8-dev1.el7.centos.noarch 9/28 Updating : openstack-tripleo-image-elements-0.9.7-dev23.el7.centos.noarch 10/28 Updating : openstack-tripleo-puppet-elements-0.0.2-dev6.el7.centos.noarch 11/28 Updating : python-openstackclient-1.0.5-dev2.el7.centos.noarch 12/28 Updating : openstack-heat-templates-0.0.0-post350.el7.centos.noarch 13/28 Updating : python-hardware-0.14.1-dev1.el7.centos.noarch 14/28 Cleanup : openstack-tripleo-image-elements-0.9.6-6.el7ost.noarch 15/28 Cleanup : openstack-tuskar-0.4.18-3.el7ost.noarch 16/28 Cleanup : openstack-tuskar-ui-extras-0.0.4-1.el7ost.noarch 17/28 Cleanup : openstack-tuskar-ui-0.3.0-15.el7ost.noarch 18/28 Cleanup : instack-0.0.7-1.el7ost.noarch 19/28 Cleanup : diskimage-builder-0.1.46-3.el7ost.noarch 20/28 Cleanup : os-cloud-config-0.2.8-5.el7ost.noarch 21/28 Cleanup : python-ironicclient-0.5.1-10.el7ost.noarch 22/28 Cleanup : python-tuskarclient-0.1.18-4.el7ost.noarch 23/28 Cleanup : openstack-tripleo-heat-templates-0.8.6-46.el7ost.noarch 24/28 Cleanup : openstack-tripleo-puppet-elements-0.0.1-4.el7ost.noarch 25/28 Cleanup : python-openstackclient-1.0.3-2.el7ost.noarch 26/28 Cleanup : openstack-heat-templates-0-0.6.20150605git.el7ost.noarch 27/28 Cleanup : python-hardware-0.14-6.el7ost.noarch 28/28 Verifying : python-hardware-0.14.1-dev1.el7.centos.noarch 1/28 Verifying : diskimage-builder-0.1.47-dev3.el7.centos.noarch 2/28 Verifying : python-ironicclient-0.5.2-dev12.el7.centos.noarch 3/28 Verifying : openstack-tuskar-ui-2013.2.0-dev57.el7.centos.noarch 4/28 Verifying : os-cloud-config-0.2.10-dev12.el7.centos.noarch 5/28 Verifying : openstack-heat-templates-0.0.0-post350.el7.centos.noarch 6/28 Verifying : instack-0.0.8-dev1.el7.centos.noarch 7/28 Verifying : openstack-tuskar-ui-extras-2014.1-dev22.el7.centos.noarch 8/28 Verifying : openstack-tripleo-heat-templates-0.8.7-dev187.el7.centos.noarch 9/28 Verifying : python-openstackclient-1.0.5-dev2.el7.centos.noarch 10/28 Verifying : python-tuskarclient-0.1.19-dev16.el7.centos.noarch 11/28 Verifying : openstack-tuskar-2013.2.0-dev11.el7.centos.noarch 12/28 Verifying : openstack-tripleo-puppet-elements-0.0.2-dev6.el7.centos.noarch 13/28 Verifying : openstack-tripleo-image-elements-0.9.7-dev23.el7.centos.noarch 14/28 Verifying : python-hardware-0.14-6.el7ost.noarch 15/28 Verifying : openstack-tuskar-ui-extras-0.0.4-1.el7ost.noarch 16/28 Verifying : instack-0.0.7-1.el7ost.noarch 17/28 Verifying : openstack-tripleo-image-elements-0.9.6-6.el7ost.noarch 18/28 Verifying : openstack-tripleo-puppet-elements-0.0.1-4.el7ost.noarch 19/28 Verifying : python-openstackclient-1.0.3-2.el7ost.noarch 20/28 Verifying : python-tuskarclient-0.1.18-4.el7ost.noarch 21/28 Verifying : python-ironicclient-0.5.1-10.el7ost.noarch 22/28 Verifying : openstack-tuskar-0.4.18-3.el7ost.noarch 23/28 Verifying : openstack-tripleo-heat-templates-0.8.6-46.el7ost.noarch 24/28 Verifying : openstack-tuskar-ui-0.3.0-15.el7ost.noarch 25/28 Verifying : diskimage-builder-0.1.46-3.el7ost.noarch 26/28 Verifying : os-cloud-config-0.2.8-5.el7ost.noarch 27/28 Verifying : openstack-heat-templates-0-0.6.20150605git.el7ost.noarch 28/28 Updated: diskimage-builder.noarch 0:0.1.47-dev3.el7.centos instack.noarch 0:0.0.8-dev1.el7.centos openstack-heat-templates.noarch 0:0.0.0-post350.el7.centos openstack-tripleo-heat-templates.noarch 0:0.8.7-dev187.el7.centos openstack-tripleo-image-elements.noarch 0:0.9.7-dev23.el7.centos openstack-tripleo-puppet-elements.noarch 0:0.0.2-dev6.el7.centos openstack-tuskar.noarch 0:2013.2.0-dev11.el7.centos openstack-tuskar-ui.noarch 0:2013.2.0-dev57.el7.centos openstack-tuskar-ui-extras.noarch 0:2014.1-dev22.el7.centos os-cloud-config.noarch 0:0.2.10-dev12.el7.centos python-hardware.noarch 0:0.14.1-dev1.el7.centos python-ironicclient.noarch 0:0.5.2-dev12.el7.centos python-openstackclient.noarch 0:1.0.5-dev2.el7.centos python-tuskarclient.noarch 0:0.1.19-dev16.el7.centos Complete! [root@instack ~]# rpm -qv openstack-tripleo-heat-templates openstack-tripleo-heat-templates-0.8.7-dev187.el7.centos.noarch [root@instack ~]# [root@instack ~]# [root@instack ~]# openstack-service status nova openstack-nova-api (pid 29250) is active openstack-nova-compute (pid 31078) is active openstack-nova-conductor (pid 29343) is active openstack-nova-consoleauth (pid 29216) is active openstack-nova-scheduler (pid 29303) is active [root@instack ~]# [root@instack ~]# [root@instack ~]# [root@instack ~]# openstack-service status ironic openstack-ironic-api (pid 27356) is active openstack-ironic-conductor (pid 30552) is active openstack-ironic-discoverd-dnsmasq (pid 31174) is active openstack-ironic-discoverd (pid 31181) is active [root@instack ~]# [root@instack ~]# su - stack Last login: Mon Sep 14 07:33:56 EDT 2015 on pts/0 [stack@instack ~]$ ssh heat-admin.2.8 [heat-admin@overcloud-controller-0 ~]$ source overcloudrc [heat-admin@overcloud-controller-0 ~]$ [heat-admin@overcloud-controller-0 ~]$ [heat-admin@overcloud-controller-0 ~]$ [heat-admin@overcloud-controller-0 ~]$ touch a.txt [heat-admin@overcloud-controller-0 ~]$ swift upload txt a.txt a.txt [heat-admin@overcloud-controller-0 ~]$ swift list txt [heat-admin@overcloud-controller-0 ~]$ swift list txt a.txt [heat-admin@overcloud-controller-0 ~]$
Mathieu, can we try removing the workaround for this from the CI script and confirm that it is fixed that way also?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2015:1862
I hit this bug again in CI, since I used the GA image to update to the latest puddle. I re-enabled the workaround which consist to stop nova-api / nova-compute / ironic-api / ironic-conductor and discoverd
Are you hitting this on the controller or the compute nodes? It definitely should not be happening on the controller, because we use Pacemaker to stop all the services before running yum update. The compute nodes use Puppet to do the update, so there may be more to investigate there.
it's during the yum update of the undercloud
Oh, the *undercloud*. We might just need to document this as one of the steps then.
(In reply to mathieu bultel from comment #8) > I hit this bug again in CI, since I used the GA image to update to the > latest puddle. > I re-enabled the workaround which consist to stop nova-api / nova-compute / > ironic-api / ironic-conductor and discoverd to be clear the workaround here is to stop those services before running yum update on the undercloud? if so, can you also let me know what the new version of rabbitmq-server is that you ended up with after the update? my suspicion is that this might the same issue as https://bugzilla.redhat.com/show_bug.cgi?id=1264083 as i've seen some openstack services hang and not get restarted gracefully if rabbitmq goes away before.
The version on the undercloud after the update is : rabbitmq-server-3.3.5-5.el7ost.noarch Seems related ...
This is identical to bug 1264083. The solution is to update rabbitmq-server first, restart rabbitmq-server, then yum update everything else. Omri has verified that this works for him and will be documented as part of the update kbase. As a separate note, bugs that are Closed->Errata should not be reopened. If they are reproducing again, please clone to a separate bug.