Bug 1257642 - yum hanged infinitely on nova-compute cleanup when do an update
yum hanged infinitely on nova-compute cleanup when do an update
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity unspecified
: y1
: 7.0 (Kilo)
Assigned To: James Slagle
Prasanth Anbalagan
: Reopened, TestOnly, Triaged
Depends On: 1259905 1261921
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-27 10:01 EDT by mathieu bultel
Modified: 2015-11-07 02:14 EST (History)
8 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-0.8.6-57.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-07 02:14:17 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description mathieu bultel 2015-08-27 10:01:37 EDT
Description of problem:

When doing a yum update from the GA release of Director to the latest poodle (or latest puddle).
yum is hanged on the cleanup of nova-compute package.

No timeout appear with ssh or yum ... still waiting infinitely.. (I let it wait for few hours...)

Reproducible : 100%

The workaround is to stop services :

        - openstack-nova-api.service
        - openstack-nova-compute.service
        - openstack-ironic-api.service
        - openstack-ironic-discoverd.service
        - openstack-ironic-conductor.service
Comment 3 Zane Bitter 2015-09-03 12:39:42 EDT
With any luck this will be fixed by the integration between yum and puppet that we already have planned. Either way, we'll need to retest once that stuff has landed.
Comment 4 Prasanth Anbalagan 2015-09-14 10:14:51 EDT
Verified as follows. No hang of yum process was observed.

Logs
=====

[stack@instack ~]$ yum list installed | grep openstack-tripleo
openstack-tripleo.noarch           0.0.7-0.1.1664e566.el7ost
openstack-tripleo-common.noarch    0.0.1.dev6-1.git49b57eb.el7ost
openstack-tripleo-heat-templates.noarch
openstack-tripleo-image-elements.noarch
openstack-tripleo-puppet-elements.noarch

[stack@instack ~]$ rpm -qv openstack-tripleo-heat-templates.noarch
openstack-tripleo-heat-templates-0.8.6-46.el7ost.noarch

[root@rhos-compute-node-02 ~]# rpm -ivh http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm
Retrieving http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm
Preparing...                          ################################# [100%]
        file /usr/bin/rhos-release from install of rhos-release-0.69-1.noarch conflicts with file from package rhos-release-0.68-1.noarch
[root@instack ~]# 
[root@instack ~]# 
[root@instack ~]# yum update -y
Resolving Dependencies
--> Running transaction check
---> Package diskimage-builder.noarch 0:0.1.46-3.el7ost will be updated
---> Package diskimage-builder.noarch 0:0.1.47-dev3.el7.centos will be an update
---> Package instack.noarch 0:0.0.7-1.el7ost will be updated
---> Package instack.noarch 0:0.0.8-dev1.el7.centos will be an update
---> Package openstack-heat-templates.noarch 0:0-0.6.20150605git.el7ost will be updated
---> Package openstack-heat-templates.noarch 0:0.0.0-post350.el7.centos will be an update
---> Package openstack-tripleo-heat-templates.noarch 0:0.8.6-46.el7ost will be updated
---> Package openstack-tripleo-heat-templates.noarch 0:0.8.7-dev187.el7.centos will be an update
---> Package openstack-tripleo-image-elements.noarch 0:0.9.6-6.el7ost will be updated
---> Package openstack-tripleo-image-elements.noarch 0:0.9.7-dev23.el7.centos will be an update
---> Package openstack-tripleo-puppet-elements.noarch 0:0.0.1-4.el7ost will be updated
---> Package openstack-tripleo-puppet-elements.noarch 0:0.0.2-dev6.el7.centos will be an update
---> Package openstack-tuskar.noarch 0:0.4.18-3.el7ost will be updated
---> Package openstack-tuskar.noarch 0:2013.2.0-dev11.el7.centos will be an update
---> Package openstack-tuskar-ui.noarch 0:0.3.0-15.el7ost will be updated
---> Package openstack-tuskar-ui.noarch 0:2013.2.0-dev57.el7.centos will be an update
---> Package openstack-tuskar-ui-extras.noarch 0:0.0.4-1.el7ost will be updated
---> Package openstack-tuskar-ui-extras.noarch 0:2014.1-dev22.el7.centos will be an update
---> Package os-cloud-config.noarch 0:0.2.8-5.el7ost will be updated
---> Package os-cloud-config.noarch 0:0.2.10-dev12.el7.centos will be an update
---> Package python-hardware.noarch 0:0.14-6.el7ost will be updated
---> Package python-hardware.noarch 0:0.14.1-dev1.el7.centos will be an update
---> Package python-ironicclient.noarch 0:0.5.1-10.el7ost will be updated
---> Package python-ironicclient.noarch 0:0.5.2-dev12.el7.centos will be an update
---> Package python-openstackclient.noarch 0:1.0.3-2.el7ost will be updated
---> Package python-openstackclient.noarch 0:1.0.5-dev2.el7.centos will be an update
---> Package python-tuskarclient.noarch 0:0.1.18-4.el7ost will be updated
---> Package python-tuskarclient.noarch 0:0.1.19-dev16.el7.centos will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================================================================================
 Package                                       Arch               Version                               Repository                           Size
==================================================================================================================================================
Updating:
 diskimage-builder                             noarch             0.1.47-dev3.el7.centos                delorean-rdo-management             237 k
 instack                                       noarch             0.0.8-dev1.el7.centos                 delorean-rdo-management              27 k
 openstack-heat-templates                      noarch             0.0.0-post350.el7.centos              delorean-rdo-management              68 k
 openstack-tripleo-heat-templates              noarch             0.8.7-dev187.el7.centos               delorean-rdo-management             102 k
 openstack-tripleo-image-elements              noarch             0.9.7-dev23.el7.centos                delorean-rdo-management             438 k
 openstack-tripleo-puppet-elements             noarch             0.0.2-dev6.el7.centos                 delorean-rdo-management              29 k
 openstack-tuskar                              noarch             2013.2.0-dev11.el7.centos             delorean-rdo-management             280 k
 openstack-tuskar-ui                           noarch             2013.2.0-dev57.el7.centos             delorean-rdo-management             241 k
 openstack-tuskar-ui-extras                    noarch             2014.1-dev22.el7.centos               delorean-rdo-management              45 k
 os-cloud-config                               noarch             0.2.10-dev12.el7.centos               delorean-rdo-management              92 k
 python-hardware                               noarch             0.14.1-dev1.el7.centos                delorean-rdo-management             4.7 M
 python-ironicclient                           noarch             0.5.2-dev12.el7.centos                delorean-rdo-management             183 k
 python-openstackclient                        noarch             1.0.5-dev2.el7.centos                 delorean-rdo-management             207 k
 python-tuskarclient                           noarch             0.1.19-dev16.el7.centos               delorean-rdo-management             122 k

Transaction Summary
==================================================================================================================================================
Upgrade  14 Packages

Total size: 6.7 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : diskimage-builder-0.1.47-dev3.el7.centos.noarch                                                                               1/28 
  Updating   : python-tuskarclient-0.1.19-dev16.el7.centos.noarch                                                                            2/28 
  Updating   : openstack-tripleo-heat-templates-0.8.7-dev187.el7.centos.noarch                                                               3/28 
  Updating   : os-cloud-config-0.2.10-dev12.el7.centos.noarch                                                                                4/28 
  Updating   : python-ironicclient-0.5.2-dev12.el7.centos.noarch                                                                             5/28 
  Updating   : openstack-tuskar-ui-2013.2.0-dev57.el7.centos.noarch                                                                          6/28 
  Updating   : openstack-tuskar-ui-extras-2014.1-dev22.el7.centos.noarch                                                                     7/28 
  Updating   : openstack-tuskar-2013.2.0-dev11.el7.centos.noarch                                                                             8/28 
  Updating   : instack-0.0.8-dev1.el7.centos.noarch                                                                                          9/28 
  Updating   : openstack-tripleo-image-elements-0.9.7-dev23.el7.centos.noarch                                                               10/28 
  Updating   : openstack-tripleo-puppet-elements-0.0.2-dev6.el7.centos.noarch                                                               11/28 
  Updating   : python-openstackclient-1.0.5-dev2.el7.centos.noarch                                                                          12/28 
  Updating   : openstack-heat-templates-0.0.0-post350.el7.centos.noarch                                                                     13/28 
  Updating   : python-hardware-0.14.1-dev1.el7.centos.noarch                                                                                14/28 
  Cleanup    : openstack-tripleo-image-elements-0.9.6-6.el7ost.noarch                                                                       15/28 
  Cleanup    : openstack-tuskar-0.4.18-3.el7ost.noarch                                                                                      16/28 
  Cleanup    : openstack-tuskar-ui-extras-0.0.4-1.el7ost.noarch                                                                             17/28 
  Cleanup    : openstack-tuskar-ui-0.3.0-15.el7ost.noarch                                                                                   18/28 
  Cleanup    : instack-0.0.7-1.el7ost.noarch                                                                                                19/28 
  Cleanup    : diskimage-builder-0.1.46-3.el7ost.noarch                                                                                     20/28 
  Cleanup    : os-cloud-config-0.2.8-5.el7ost.noarch                                                                                        21/28 
  Cleanup    : python-ironicclient-0.5.1-10.el7ost.noarch                                                                                   22/28 
  Cleanup    : python-tuskarclient-0.1.18-4.el7ost.noarch                                                                                   23/28 
  Cleanup    : openstack-tripleo-heat-templates-0.8.6-46.el7ost.noarch                                                                      24/28 
  Cleanup    : openstack-tripleo-puppet-elements-0.0.1-4.el7ost.noarch                                                                      25/28 
  Cleanup    : python-openstackclient-1.0.3-2.el7ost.noarch                                                                                 26/28 
  Cleanup    : openstack-heat-templates-0-0.6.20150605git.el7ost.noarch                                                                     27/28 
  Cleanup    : python-hardware-0.14-6.el7ost.noarch                                                                                         28/28 
  Verifying  : python-hardware-0.14.1-dev1.el7.centos.noarch                                                                                 1/28 
  Verifying  : diskimage-builder-0.1.47-dev3.el7.centos.noarch                                                                               2/28 
  Verifying  : python-ironicclient-0.5.2-dev12.el7.centos.noarch                                                                             3/28 
  Verifying  : openstack-tuskar-ui-2013.2.0-dev57.el7.centos.noarch                                                                          4/28 
  Verifying  : os-cloud-config-0.2.10-dev12.el7.centos.noarch                                                                                5/28 
  Verifying  : openstack-heat-templates-0.0.0-post350.el7.centos.noarch                                                                      6/28 
  Verifying  : instack-0.0.8-dev1.el7.centos.noarch                                                                                          7/28 
  Verifying  : openstack-tuskar-ui-extras-2014.1-dev22.el7.centos.noarch                                                                     8/28 
  Verifying  : openstack-tripleo-heat-templates-0.8.7-dev187.el7.centos.noarch                                                               9/28 
  Verifying  : python-openstackclient-1.0.5-dev2.el7.centos.noarch                                                                          10/28 
  Verifying  : python-tuskarclient-0.1.19-dev16.el7.centos.noarch                                                                           11/28 
  Verifying  : openstack-tuskar-2013.2.0-dev11.el7.centos.noarch                                                                            12/28 
  Verifying  : openstack-tripleo-puppet-elements-0.0.2-dev6.el7.centos.noarch                                                               13/28 
  Verifying  : openstack-tripleo-image-elements-0.9.7-dev23.el7.centos.noarch                                                               14/28 
  Verifying  : python-hardware-0.14-6.el7ost.noarch                                                                                         15/28 
  Verifying  : openstack-tuskar-ui-extras-0.0.4-1.el7ost.noarch                                                                             16/28 
  Verifying  : instack-0.0.7-1.el7ost.noarch                                                                                                17/28 
  Verifying  : openstack-tripleo-image-elements-0.9.6-6.el7ost.noarch                                                                       18/28 
  Verifying  : openstack-tripleo-puppet-elements-0.0.1-4.el7ost.noarch                                                                      19/28 
  Verifying  : python-openstackclient-1.0.3-2.el7ost.noarch                                                                                 20/28 
  Verifying  : python-tuskarclient-0.1.18-4.el7ost.noarch                                                                                   21/28 
  Verifying  : python-ironicclient-0.5.1-10.el7ost.noarch                                                                                   22/28 
  Verifying  : openstack-tuskar-0.4.18-3.el7ost.noarch                                                                                      23/28 
  Verifying  : openstack-tripleo-heat-templates-0.8.6-46.el7ost.noarch                                                                      24/28 
  Verifying  : openstack-tuskar-ui-0.3.0-15.el7ost.noarch                                                                                   25/28 
  Verifying  : diskimage-builder-0.1.46-3.el7ost.noarch                                                                                     26/28 
  Verifying  : os-cloud-config-0.2.8-5.el7ost.noarch                                                                                        27/28 
  Verifying  : openstack-heat-templates-0-0.6.20150605git.el7ost.noarch                                                                     28/28 

Updated:
  diskimage-builder.noarch 0:0.1.47-dev3.el7.centos                       instack.noarch 0:0.0.8-dev1.el7.centos                                  
  openstack-heat-templates.noarch 0:0.0.0-post350.el7.centos              openstack-tripleo-heat-templates.noarch 0:0.8.7-dev187.el7.centos       
  openstack-tripleo-image-elements.noarch 0:0.9.7-dev23.el7.centos        openstack-tripleo-puppet-elements.noarch 0:0.0.2-dev6.el7.centos        
  openstack-tuskar.noarch 0:2013.2.0-dev11.el7.centos                     openstack-tuskar-ui.noarch 0:2013.2.0-dev57.el7.centos                  
  openstack-tuskar-ui-extras.noarch 0:2014.1-dev22.el7.centos             os-cloud-config.noarch 0:0.2.10-dev12.el7.centos                        
  python-hardware.noarch 0:0.14.1-dev1.el7.centos                         python-ironicclient.noarch 0:0.5.2-dev12.el7.centos                     
  python-openstackclient.noarch 0:1.0.5-dev2.el7.centos                   python-tuskarclient.noarch 0:0.1.19-dev16.el7.centos                    

Complete!
[root@instack ~]# rpm -qv openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-0.8.7-dev187.el7.centos.noarch
[root@instack ~]# 

[root@instack ~]# 
[root@instack ~]# openstack-service status nova
openstack-nova-api (pid 29250) is active
openstack-nova-compute (pid 31078) is active
openstack-nova-conductor (pid 29343) is active
openstack-nova-consoleauth (pid 29216) is active
openstack-nova-scheduler (pid 29303) is active
[root@instack ~]# 
[root@instack ~]# 
[root@instack ~]# 
[root@instack ~]# openstack-service status ironic
openstack-ironic-api (pid 27356) is active
openstack-ironic-conductor (pid 30552) is active
openstack-ironic-discoverd-dnsmasq (pid 31174) is active
openstack-ironic-discoverd (pid 31181) is active
[root@instack ~]# 


[root@instack ~]# su - stack   
Last login: Mon Sep 14 07:33:56 EDT 2015 on pts/0
[stack@instack ~]$ ssh heat-admin@192.0.2.8                              
[heat-admin@overcloud-controller-0 ~]$ source overcloudrc 
[heat-admin@overcloud-controller-0 ~]$ 
[heat-admin@overcloud-controller-0 ~]$ 
[heat-admin@overcloud-controller-0 ~]$ 
[heat-admin@overcloud-controller-0 ~]$ touch a.txt
[heat-admin@overcloud-controller-0 ~]$ swift upload txt a.txt
a.txt
[heat-admin@overcloud-controller-0 ~]$ swift list 
txt
[heat-admin@overcloud-controller-0 ~]$ swift list txt
a.txt
[heat-admin@overcloud-controller-0 ~]$
Comment 5 Zane Bitter 2015-09-14 11:51:33 EDT
Mathieu, can we try removing the workaround for this from the CI script and confirm that it is fixed that way also?
Comment 7 errata-xmlrpc 2015-10-08 08:17:48 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:1862
Comment 8 mathieu bultel 2015-11-05 11:43:36 EST
I hit this bug again in CI, since I used the GA image to update to the latest puddle.
I re-enabled the workaround which consist to stop nova-api / nova-compute / ironic-api / ironic-conductor and discoverd
Comment 9 Zane Bitter 2015-11-05 12:01:42 EST
Are you hitting this on the controller or the compute nodes? It definitely should not be happening on the controller, because we use Pacemaker to stop all the services before running yum update. The compute nodes use Puppet to do the update, so there may be more to investigate there.
Comment 10 mathieu bultel 2015-11-05 12:04:24 EST
it's during the yum update of the undercloud
Comment 11 Zane Bitter 2015-11-05 13:02:43 EST
Oh, the *undercloud*. We might just need to document this as one of the steps then.
Comment 12 James Slagle 2015-11-06 10:42:52 EST
(In reply to mathieu bultel from comment #8)
> I hit this bug again in CI, since I used the GA image to update to the
> latest puddle.
> I re-enabled the workaround which consist to stop nova-api / nova-compute /
> ironic-api / ironic-conductor and discoverd

to be clear the workaround here is to stop those services before running yum update on the undercloud?

if so, can you also let me know what the new version of rabbitmq-server is that you ended up with after the update?

my suspicion is that this might the same issue as https://bugzilla.redhat.com/show_bug.cgi?id=1264083 as i've seen some openstack services hang and not get restarted gracefully if rabbitmq goes away before.
Comment 13 mathieu bultel 2015-11-06 10:47:10 EST
The version on the undercloud after the update is :
rabbitmq-server-3.3.5-5.el7ost.noarch

Seems related ...
Comment 14 Mike Burns 2015-11-07 02:14:17 EST
This is identical to bug 1264083.  The solution is to update rabbitmq-server first, restart rabbitmq-server, then yum update everything else.  Omri has verified that this works for him and will be documented as part of the update kbase.

As a separate note, bugs that are Closed->Errata should not be reopened.  If they are reproducing again, please clone to a separate bug.

Note You need to log in before you can comment on or make changes to this bug.