Bug 1368104

Summary: rhel-osp-director: 9.0 different services failed to load after reboot of undercloud. (OPM)
Product: Red Hat OpenStack Reporter: Mike Burns <mburns>
Component: openstack-puppet-modulesAssignee: Emilien Macchi <emacchi>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: high Docs Contact:
Priority: high    
Version: 9.0 (Mitaka)CC: abeekhof, achernet, apevec, dbecker, dmacpher, emacchi, fdinitto, gfidente, jason.dobies, jcoufal, jeckersb, jguiditt, jjoyce, lhh, markmc, mburns, mcornea, michele, morazi, nlevinki, ohochman, plemenko, rhel-osp-director-maint, royoung, rscarazz, sasha, sclewis, srevivo, tvignaud
Target Milestone: asyncKeywords: ZStream
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-puppet-modules-8.1.10-1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1348700 Environment:
Last Closed: 2016-12-21 16:47:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1348700    

Description Mike Burns 2016-08-18 12:31:40 UTC
The initial fix for this issue introduced a regression that requires an OPM patch.  This bug is for that OPM patch.  For details, please see bug 1348700

+++ This bug was initially created as a clone of Bug #1348700 +++

rhel-osp-director: 9.0 rabbitmq and neutron-server failed to load after reboot of undercloud.

Environment:
instack-undercloud-4.0.0-4.el7ost.noarch
python-neutron-8.1.0-6.el7ost.noarch
python-neutron-lib-0.0.2-1.el7ost.noarch
openstack-neutron-8.1.0-6.el7ost.noarch
openstack-neutron-openvswitch-8.1.0-6.el7ost.noarch
python-neutronclient-4.1.1-2.el7ost.noarch
openstack-neutron-common-8.1.0-6.el7ost.noarch
openstack-neutron-ml2-8.1.0-6.el7ost.noarch
rabbitmq-server-3.6.2-3.el7ost.noarch


Steps to reproduce:
1. Deploy overcloud with:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --swift-storage-scale 0 --block-storage-scale 0 --neutron-tunnel-types vxlan,gre --neutron-network-type vxlan,gre --neutron-network-vlan-ranges datacentre:118:143 --neutron-bridge-mappings datacentre:br-ex --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml -e /home/stack/ssl-heat-templates/environments/enable-tls.yaml -e /home/stack/ssl-heat-templates/environments/inject-trust-anchor.yaml --ceph-storage-scale 1

2. Populate the overcloud
3. Reboot the undercloud
4. After reboot attempt to run "nova list" on the undercloud.


Result:

No handlers could be found for logger "keystoneauth.identity.generic.base"

[stack@instack ~]$ sudo systemctl|grep -i fail
● neutron-server.service  loaded failed     failed   OpenStack Neutron Server
● ovirt-guest-agent.service loaded failed     failed          oVirt Guest Agent
● postfix.service   loaded failed     failed          Postfix Mail Transport Agent
● rabbitmq-server.service   loaded failed     failed          RabbitMQ broker


Manually starting rabbitmq and neutron-server works.

Workaround:
systemctl start rabbitmq-server.service
systemctl start neutron-server.service
nova list

--- Additional comment from Alexander Chuzhoy on 2016-06-21 16:49:20 EDT ---

On another deployment/attempt, only the neutron-server was failed after rebooting the undercloud.
Manually started it.

Comment 2 Mike Burns 2016-08-18 18:18:55 UTC
After talking to Emilien, this is to fix an issue related to IPv6 but is unrelated to the original issue.  Leaving it as an async target.

Comment 3 Jaromir Coufal 2016-08-18 20:33:29 UTC
ack

Comment 8 Alexander Chuzhoy 2016-12-20 17:06:01 UTC
Verified:
Environment:                           
openstack-puppet-modules-8.1.10-1.el7ost.noarch

[stack@instack ~]$ . stackrc
[stack@instack ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 6bfae073-7359-4046-aa9e-7952d8c555b6 | overcloud-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| 386d5c45-2fa7-4651-82e3-5372f2af3c7e | overcloud-compute-0     | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| 24162ed9-2be3-4384-a4f1-8049689dde1a | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| ba8b9557-46f5-44d3-9095-f7adac8d7e82 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| 55d944dc-c717-40b5-8320-06cedab5bc72 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
[stack@instack ~]$ sudo systemctl |grep -i fail
● ovirt-guest-agent.service                                                                loaded failed failed    oVirt Guest Agent
● selinux-policy-migrate-local-changes                                    loaded failed failed    Migrate local SELinux policy changes from the old store structure to the new structure
[stack@instack ~]$ sudo reboot
Connection to 192.168.122.83 closed by remote host.
Connection to 192.168.122.83 closed.

[root@sealusa11 ~]# ssh stack.122.83
ssh: connect to host 192.168.122.83 port 22: Connection refused
[root@sealusa11 ~]# ssh stack.122.83
stack.122.83's password:
Last login: Tue Dec 20 12:01:23 2016 from gateway
[stack@instack ~]$ . stackrc
[stack@instack ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 6bfae073-7359-4046-aa9e-7952d8c555b6 | overcloud-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| 386d5c45-2fa7-4651-82e3-5372f2af3c7e | overcloud-compute-0     | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| 24162ed9-2be3-4384-a4f1-8049689dde1a | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| ba8b9557-46f5-44d3-9095-f7adac8d7e82 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| 55d944dc-c717-40b5-8320-06cedab5bc72 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
[stack@instack ~]$ uptime
 12:04:42 up 1 min,  1 user,  load average: 2.94, 0.88, 0.30
[stack@instack ~]$ sudo systemctl |grep -i fail
● ovirt-guest-agent.service                                                                loaded failed failed    oVirt Guest Agent
● selinux-policy-migrate-local-changes                                    loaded failed failed    Migrate local SELinux policy changes from the old store structure to the new structure

Comment 10 errata-xmlrpc 2016-12-21 16:47:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2987.html