1326644 – When undercloud is deployed with SSL, openstack undercloud upgrade removes the SSL VIP IP addresses

Bug 1326644 - When undercloud is deployed with SSL, openstack undercloud upgrade removes the SSL VIP IP addresses

Summary: When undercloud is deployed with SSL, openstack undercloud upgrade removes th...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	instack-undercloud
Sub Component:
Version:	8.0 (Liberty)
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	async
Target Release:	8.0 (Liberty)
Assignee:	Brad P. Crochet
QA Contact:	Dan Yasny
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1334047 (view as bug list)
Depends On:
Blocks:	1325702
TreeView+	depends on / blocked

Reported:	2016-04-13 09:32 UTC by Dan Yasny
Modified:	2016-06-15 12:39 UTC (History)
CC List:	6 users (show)
Fixed In Version:	instack-undercloud-2.2.7-5.el7ost
Doc Type:	Known Issue
Doc Text:	Cause: A change in os-net-config defaults might cause networking on the undercloud to restart Consequence: If you're using SSL on the undercloud, the vips will lose connectivity because of this restart Workaround (if any): If this occurs, simply run: systemctl restart keepalived Result: the vips will be re-established.
Clone Of:
Environment:
Last Closed:	2016-06-15 12:39:09 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
OpenStack gerrit	305478	'None'	'MERGED'	'Add a reload of keepalived'	2019-11-14 21:30:24 UTC
OpenStack gerrit	307700	'None'	'MERGED'	'Add a reload of keepalived'	2019-11-14 21:30:24 UTC
Red Hat Product Errata	RHBA-2016:1229	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 8 director Bug Fix Advisory	2016-06-15 16:38:45 UTC

Description Dan Yasny 2016-04-13 09:32:45 UTC

Description of problem:
Opening a separate BZ for the root cause of https://bugzilla.redhat.com/show_bug.cgi?id=1325702

When the undercloud has SSL, running undercloud upgrade makes the IPs belonging to the SSL endpoints disappear. Restarting keepalived restores the IPs and rerunning undercloud upgrade afterwards works

Version-Release number of selected component (if applicable):
Upgrading from 7.3 GA to 8

openstack-tripleo-heat-templates-0.8.14-7.el7ost.noarch
openstack-ceilometer-polling-5.0.2-2.el7ost.noarch
openstack-tripleo-common-0.3.1-1.el7ost.noarch
openstack-heat-api-5.0.1-5.el7ost.noarch
openstack-nova-api-12.0.2-5.el7ost.noarch
openstack-swift-proxy-2.5.0-2.el7ost.noarch
openstack-swift-container-2.5.0-2.el7ost.noarch
openstack-nova-common-12.0.2-5.el7ost.noarch
openstack-nova-compute-12.0.2-5.el7ost.noarch
openstack-neutron-7.0.1-15.el7ost.noarch
openstack-ceilometer-api-5.0.2-2.el7ost.noarch
openstack-swift-account-2.5.0-2.el7ost.noarch
openstack-tuskar-ui-extras-0.0.4-2.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.5-1.el7ost.noarch
openstack-swift-plugin-swift3-1.9-1.el7ost.noarch
python-django-openstack-auth-2.0.1-1.2.el7ost.noarch
openstack-dashboard-8.0.1-2.el7ost.noarch
openstack-heat-engine-5.0.1-5.el7ost.noarch
openstack-nova-scheduler-12.0.2-5.el7ost.noarch
openstack-neutron-ml2-7.0.1-15.el7ost.noarch
openstack-ironic-api-4.2.2-4.el7ost.noarch
openstack-ceilometer-collector-5.0.2-2.el7ost.noarch
openstack-ironic-inspector-2.2.5-2.el7ost.noarch
openstack-selinux-0.6.58-1.el7ost.noarch
openstack-tuskar-0.4.18-5.el7ost.noarch
openstack-tripleo-image-elements-0.9.9-1.el7ost.noarch
openstack-swift-2.5.0-2.el7ost.noarch
openstack-ceilometer-notification-5.0.2-2.el7ost.noarch
openstack-neutron-common-7.0.1-15.el7ost.noarch
python-openstackclient-1.7.2-1.el7ost.noarch
openstack-dashboard-theme-8.0.1-2.el7ost.noarch
openstack-heat-api-cloudwatch-5.0.1-5.el7ost.noarch
openstack-tempest-liberty-20160317.1.el7ost.noarch
openstack-nova-console-12.0.2-5.el7ost.noarch
openstack-nova-novncproxy-12.0.2-5.el7ost.noarch
openstack-ironic-conductor-4.2.2-4.el7ost.noarch
openstack-glance-11.0.1-4.el7ost.noarch
openstack-keystone-8.0.1-1.el7ost.noarch
openstack-puppet-modules-7.0.17-1.el7ost.noarch
openstack-tripleo-0.0.7-1.el7ost.noarch
openstack-nova-cert-12.0.2-5.el7ost.noarch
openstack-neutron-openvswitch-7.0.1-15.el7ost.noarch
openstack-ceilometer-alarm-5.0.2-2.el7ost.noarch
openstack-swift-object-2.5.0-2.el7ost.noarch
openstack-heat-templates-0-0.8.20150605git.el7ost.noarch
openstack-tuskar-ui-0.4.0-5.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-tripleo-heat-templates-kilo-0.8.14-7.el7ost.noarch
openstack-ceilometer-common-5.0.2-2.el7ost.noarch
openstack-ironic-common-4.2.2-4.el7ost.noarch
openstack-heat-common-5.0.1-5.el7ost.noarch
openstack-heat-api-cfn-5.0.1-5.el7ost.noarch
openstack-nova-conductor-12.0.2-5.el7ost.noarch
openstack-ceilometer-central-5.0.2-2.el7ost.noarch

instack-0.0.8-2.el7ost.noarch
instack-undercloud-2.2.7-4.el7ost.noarch
keepalived-1.2.13-7.el7.x86_64




How reproducible:
always

Steps to Reproduce:
1. deploy 7.3 with UC and OC SSL
2. try to upgrade UC
3.

Actual results:
upgrade fails, until keepalived is manually restarted. for some reason, reboot didn't help

Expected results:
upgrade should work

Additional info:

keepalived.conf:
global_defs {
  notification_email {
    root@localdomain
  }
  notification_email_from keepalived@localdomain
  smtp_server localhost
  smtp_connect_timeout 30
  router_id instack
}

static_ipaddress {
  
}

vrrp_script haproxy {
  script   "systemctl status haproxy.service"
  interval 2
  weight   2
}
vrrp_instance 51 {
  virtual_router_id 51

  # Advert interval
  advert_int 1

  # for electing MASTER, highest priority wins.
  priority  101
  state     MASTER

  interface br-ctlplane

  virtual_ipaddress {

      192.0.2.3 dev br-ctlplane
  }

  track_script {
  haproxy
  }


}
vrrp_instance 52 {
  virtual_router_id 52

  # Advert interval
  advert_int 1

  # for electing MASTER, highest priority wins.
  priority  101
  state     MASTER

  interface br-ctlplane

  virtual_ipaddress {

      192.0.2.2 dev br-ctlplane
  }

  track_script {
  haproxy
  }




}

------------------------

Event flow with 'ip a' outputs after each step:

after 7.3 deployed:

[stack@instack ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:2a:69:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.224/24 brd 192.168.122.255 scope global dynamic eth0
       valid_lft 2506sec preferred_lft 2506sec
    inet6 fe80::5054:ff:fe2a:69bd/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000
    link/ether 00:96:e8:04:9a:6b brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 1a:97:f7:6e:48:d0 brd ff:ff:ff:ff:ff:ff
5: br-ctlplane: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:96:e8:04:9a:6b brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.1/24 brd 192.0.2.255 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 192.0.2.3/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 192.0.2.2/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet6 fe80::296:e8ff:fe04:9a6b/64 scope link 
       valid_lft forever preferred_lft forever
6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ce:89:04:02:0a:43 brd ff:ff:ff:ff:ff:ff
8: vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 56:7e:a0:f7:72:5c brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.1/24 scope global vlan10
       valid_lft forever preferred_lft forever
    inet6 fe80::547e:a0ff:fef7:725c/64 scope link 
       valid_lft forever preferred_lft forever
9: vlan118: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether ce:ff:96:2c:78:d5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.1/24 scope global vlan118
       valid_lft forever preferred_lft forever
    inet6 fe80::ccff:96ff:fe2c:78d5/64 scope link 
       valid_lft forever preferred_lft forever

after populate script, creating some instances, tenants, volumes, etc:

[stack@instack ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:2a:69:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.224/24 brd 192.168.122.255 scope global dynamic eth0
       valid_lft 3110sec preferred_lft 3110sec
    inet6 fe80::5054:ff:fe2a:69bd/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000
    link/ether 00:96:e8:04:9a:6b brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 1a:97:f7:6e:48:d0 brd ff:ff:ff:ff:ff:ff
5: br-ctlplane: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:96:e8:04:9a:6b brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.1/24 brd 192.0.2.255 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 192.0.2.3/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 192.0.2.2/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet6 fe80::296:e8ff:fe04:9a6b/64 scope link 
       valid_lft forever preferred_lft forever
6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ce:89:04:02:0a:43 brd ff:ff:ff:ff:ff:ff
8: vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 56:7e:a0:f7:72:5c brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.1/24 scope global vlan10
       valid_lft forever preferred_lft forever
    inet6 fe80::547e:a0ff:fef7:725c/64 scope link 
       valid_lft forever preferred_lft forever
9: vlan118: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether ce:ff:96:2c:78:d5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.1/24 scope global vlan118
       valid_lft forever preferred_lft forever
    inet6 fe80::ccff:96ff:fe2c:78d5/64 scope link 
       valid_lft forever preferred_lft forever

run sudo rhos-release -P 8-director
and sudo yum update -y

[stack@instack ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:2a:69:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.224/24 brd 192.168.122.255 scope global dynamic eth0
       valid_lft 2095sec preferred_lft 2095sec
    inet6 fe80::5054:ff:fe2a:69bd/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000
    link/ether 00:96:e8:04:9a:6b brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 1a:97:f7:6e:48:d0 brd ff:ff:ff:ff:ff:ff
5: br-ctlplane: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:96:e8:04:9a:6b brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.1/24 brd 192.0.2.255 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 192.0.2.3/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 192.0.2.2/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet6 fe80::296:e8ff:fe04:9a6b/64 scope link 
       valid_lft forever preferred_lft forever
6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ce:89:04:02:0a:43 brd ff:ff:ff:ff:ff:ff
8: vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 56:7e:a0:f7:72:5c brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.1/24 scope global vlan10
       valid_lft forever preferred_lft forever
    inet6 fe80::547e:a0ff:fef7:725c/64 scope link 
       valid_lft forever preferred_lft forever
9: vlan118: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether ce:ff:96:2c:78:d5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.1/24 scope global vlan118
       valid_lft forever preferred_lft forever
    inet6 fe80::ccff:96ff:fe2c:78d5/64 scope link 
       valid_lft forever preferred_lft forever

sudo cp cacert.pem /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust extract
openstack undercloud upgrade

Fails with:

Notice: Finished catalog run in 313.06 seconds
+ rc=6
+ set -e
+ echo 'puppet apply exited with exit code 6'
puppet apply exited with exit code 6
+ '[' 6 '!=' 2 -a 6 '!=' 0 ']'
+ exit 6
[2016-04-13 05:10:43,132] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 6]

[2016-04-13 05:10:43,133] (os-refresh-config) [ERROR] Aborting...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
Command 'instack-install-undercloud' returned non-zero exit status 1

[stack@instack ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:2a:69:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.224/24 brd 192.168.122.255 scope global dynamic eth0
       valid_lft 3067sec preferred_lft 3067sec
    inet6 fe80::5054:ff:fe2a:69bd/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000
    link/ether 00:96:e8:04:9a:6b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::296:e8ff:fe04:9a6b/64 scope link 
       valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 1a:97:f7:6e:48:d0 brd ff:ff:ff:ff:ff:ff
6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ce:89:04:02:0a:43 brd ff:ff:ff:ff:ff:ff
10: br-ctlplane: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:96:e8:04:9a:6b brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.1/24 brd 192.0.2.255 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet6 fe80::296:e8ff:fe04:9a6b/64 scope link 
       valid_lft forever preferred_lft forever

sudo systemctl restart keepalived.service
...twice + wait for IPs to show up, this is not immediate

[stack@instack ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:2a:69:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.224/24 brd 192.168.122.255 scope global dynamic eth0
       valid_lft 2905sec preferred_lft 2905sec
    inet6 fe80::5054:ff:fe2a:69bd/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000
    link/ether 00:96:e8:04:9a:6b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::296:e8ff:fe04:9a6b/64 scope link 
       valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 1a:97:f7:6e:48:d0 brd ff:ff:ff:ff:ff:ff
6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ce:89:04:02:0a:43 brd ff:ff:ff:ff:ff:ff
10: br-ctlplane: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:96:e8:04:9a:6b brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.1/24 brd 192.0.2.255 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 192.0.2.3/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 192.0.2.2/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet6 fe80::296:e8ff:fe04:9a6b/64 scope link 
       valid_lft forever preferred_lft forever

openstack undercloud upgrade
--success

Comment 2 Brad P. Crochet 2016-04-13 20:37:13 UTC

Adding a reload (not a restart) of keepalived appears to fix this. I've posted a patch to master.

Comment 5 Alexander Chuzhoy 2016-05-09 17:43:28 UTC

*** Bug 1334047 has been marked as a duplicate of this bug. ***

Comment 6 Alexander Chuzhoy 2016-06-13 17:20:23 UTC

Verified:
Environment:
instack-undercloud-2.2.7-7.el7ost.noarch


grep -q OS_AUTH_URL=https stackrc  && echo "https"
https


ran "openstack undercloud upgrade".



#############################################################################
Undercloud install complete.

The file containing this installation's passwords is at
/home/stack/undercloud-passwords.conf.

There is also a stackrc file at /home/stack/stackrc.

These files are needed to interact with the OpenStack services, and should be
secured.

#############################################################################


#############################################################################

Upgrade is a new feature in Red Hat OpenStack Platform-director (OSP-d) that
needs to be fully tested on each customer’s specific configurations before it
is performed on any live production environment.

Red Hat has tested most use cases and combinations that are offered as
standard options through the director but, due to the number of possible
combinations, this can never be a fully exhaustive list. In addition, if
the configuration has been modified from the standard deployment, either
manually or through post configuration hooks, testing the upgrade feature in
a non-production environment becomes even more critical.

Therefore, we advise you to:

1) Perform a backup of your undercloud node before starting any steps in the
 upgrade procedure.
2) Run the upgrade procedure in a test environment that includes all of the
 changes you have made before running the procedure in your production
 environment.
3) Please contact Red Hat and request guidance and assistance on the upgrade
 process before proceeding If you feel uncomfortable about performing this
 upgrade we suggest.

#############################################################################

Comment 8 errata-xmlrpc 2016-06-15 12:39:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1229

Note You need to log in before you can comment on or make changes to this bug.