Bug 1271664

Summary: Undercloud update packages
Product: Red Hat OpenStack Reporter: hrosnet
Component: rhosp-directorAssignee: chris alfonso <calfonso>
Status: CLOSED DUPLICATE QA Contact: yeylon <yeylon>
Severity: unspecified Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: cylopez, dsavinea, glambert, hbrock, imouzann, mburns, rhel-osp-director-maint, srevivo, vcojot
Target Milestone: y2   
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-23 13:41:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description hrosnet 2015-10-14 13:02:23 UTC
Description of problem:
RabbitMQ is being stop (and not started) on the undercloud while we updated the packages.

It had to be manually started. We believe we wouldn't have had the issue on the overcloud, as it would be running with pacemaker.

Version-Release number of selected component (if applicable):
OSP 7.0

How reproducible:
Every time

Steps to Reproduce:
1. Deploy undercloud
2. Update packages
3. Run 'sudo systemctl start rabbitmq-server' to finish update

Actual results:
Update hanging while waiting for rabbitMq service

Expected results:
Packages being updated and services restared


Additional info:

As we ran:
sudo yum update
...
 openstack-nova-novncproxy            noarch      2015.1.1-1.el7ost                  rhel-7-server-openstack-7.0-rpms             7.8 k
 openstack-nova-scheduler             noarch      2015.1.1-1.el7ost                  rhel-7-server-openstack-7.0-rpms             7.4 k
 openstack-puppet-modules             noarch      2015.1.8-21.el7ost                 rhel-7-server-openstack-7.0-rpms             1.6 M
 openstack-tripleo-common             noarch      0.0.1.dev6-3.git49b57eb.el7ost     rhel-7-server-openstack-7.0-director-rpms     22 k
 openstack-tripleo-heat-templates     noarch      0.8.6-71.el7ost                    rhel-7-server-openstack-7.0-director-rpms    142 k
 openstack-tripleo-image-elements     noarch      0.9.6-10.el7ost                    rhel-7-server-openstack-7.0-director-rpms    441 k
 openstack-tripleo-puppet-elements    noarch      0.0.1-5.el7ost                     rhel-7-server-openstack-7.0-director-rpms     31 k
 instack-undercloud                   noarch      2.1.2-29.el7ost                    rhel-7-server-openstack-7.0-director-rpms    153 k
 python-nova                          noarch      2015.1.1-1.el7ost                  rhel-7-server-openstack-7.0-rpms             2.9 M
 python-rdomanager-oscplugin          noarch      0.0.10-8.el7ost                    rhel-7-server-openstack-7.0-director-rpms    129 k
 rabbitmq-server                      noarch      3.3.5-5.el7ost                     rhel-7-server-openstack-7.0-rpms             4.0 M

It remained blocked on:
  Nettoyage/Cleaning    : openstack-nova-compute-2015.1.0-18.el7ost.noarch                  80/124

On the nova-compute.log we were also seeing:
2015-10-13 09:47:22.583 17887 INFO oslo_messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.153.20.1:5672

Once started the update process continued properly.

Comment 2 hrosnet 2015-10-15 09:01:17 UTC
It looks like the scripts for this package don't start the service after an upgrade:

[stack@instack-prv images-7.1]$ sudo rpm -q --scripts rabbitmq-server
preinstall scriptlet (using /bin/sh):

if [ $1 -gt 1 ]; then
  # Upgrade - stop previous instance of rabbitmq-server
  systemctl stop rabbitmq-server.service
fi

# create rabbitmq group
if ! getent group rabbitmq >/dev/null; then
        groupadd -r rabbitmq
fi

# create rabbitmq user
if ! getent passwd rabbitmq >/dev/null; then
        useradd -r -g rabbitmq -d /var/lib/rabbitmq rabbitmq \
            -c "RabbitMQ messaging server"
fi
postinstall scriptlet (using /bin/sh):

if [ $1 -eq 1 ] ; then 
        # Initial installation 
        /usr/bin/systemctl preset rabbitmq-server.service >/dev/null 2>&1 || : 
fi
preuninstall scriptlet (using /bin/sh):
# We do not remove /var/log and /var/lib directories
# Leave rabbitmq user and group

if [ $1 -eq 0 ] ; then 
        # Package removal, not upgrade 
        /usr/bin/systemctl --no-reload disable rabbitmq-server.service > /dev/null 2>&1 || : 
        /usr/bin/systemctl stop rabbitmq-server.service > /dev/null 2>&1 || : 
fi 

# Clean out plugin activation state, both on uninstall and upgrade
rm -rf /var/lib/rabbitmq/plugins
for ext in rel script boot ; do
    rm -f /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/ebin/rabbit.$ext
done

In comparison to other package such as openssh-server:
[stack@instack-prv images-7.1]$ sudo rpm -q --scripts openssh-server
preinstall scriptlet (using /bin/sh):
getent group sshd >/dev/null || groupadd -g 74 -r sshd || :
getent passwd sshd >/dev/null || \
  useradd -c "Privilege-separated SSH" -u 74 -g sshd \
  -s /sbin/nologin -r -d /var/empty/sshd sshd 2> /dev/null || :
postinstall scriptlet (using /bin/sh):

if [ $1 -eq 1 ] ; then
        # Initial installation  
        /usr/bin/systemctl preset sshd.service sshd.socket >/dev/null 2>&1 || :
fi
preuninstall scriptlet (using /bin/sh):

if [ $1 -eq 0 ] ; then
        # Package removal, not upgrade
        /usr/bin/systemctl --no-reload disable sshd.service sshd.socket > /dev/null 2>&1 || :
        /usr/bin/systemctl stop sshd.service sshd.socket > /dev/null 2>&1 || :
fi
postuninstall scriptlet (using /bin/sh):

/usr/bin/systemctl daemon-reload >/dev/null 2>&1 || :
if [ $1 -ge 1 ] ; then
        # Package upgrade, not uninstall
        /usr/bin/systemctl try-restart sshd.service >/dev/null 2>&1 || : 
fi

So we believe there is something wrong here.

Comment 3 Mike Burns 2015-10-23 13:41:59 UTC

*** This bug has been marked as a duplicate of bug 1264083 ***