Bug 1229296

Summary:

ironic services did not start up after reboot

Product:

Red Hat OpenStack

Reporter:

Ola Pavlenko <opavlenk>

Component:

instack-undercloud

Assignee:

Dmitry Tantsur <dtantsur>

Status:

CLOSED ERRATA

QA Contact:

Ola Pavlenko <opavlenk>

Severity:

unspecified

Docs Contact:

Priority:

high

Version:

Director

CC:

apevec, calfonso, dmacpher, dtantsur, lhh, mburns, ohochman, rhel-osp-director-maint, rrosa, ukalifon

Target Milestone:

Keywords:

Triaged

Target Release:

Director

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

instack-undercloud-2.1.2-2.el7ost

Doc Type:

Bug Fix

Doc Text:

The openstack-ironic-discoverd service would check for openstack-ironic-api's presence on start up. Due to start up ordering, openstack-ironic-discoverd failed to detect openstack-ironic-api on reboot. This fix removes the need to check for openstack-ironic-api's presence from openstack-ironic-discoverd start up code. All services start successfully now on reboot.

Story Points:

---

Clone Of:

Environment:

Last Closed:

2015-08-05 13:52:50 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
journal.log	none

Description Ola Pavlenko 2015-06-08 12:10:51 UTC

Description of problem:
installed poodle following  https://repos.fedorapeople.org/repos/openstack-m/docs/internal/master/installation/installing.html then rebooted the instack VM
and ironic services didn't start. 
while trying to discover node details got :
$ instack-ironic-deployment --discover-nodes
Preparing for deployment...
  Discovering nodes.
    Sending node ID 55d21860-66c6-4977-ad07-0e343dea2f5a to discoverd for discovery ... ERROR: openstack ('Connection aborted.', error(111, 'Connection refused'))

when node registration worked:
$ instack-ironic-deployment --nodes-json instackenv.json --register-nodes
Preparing for deployment...
  Registering nodes from instackenv.json
  Nodes registered.

+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID | Power State | Provision State | Maintenance |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| 55d21860-66c6-4977-ad07-0e343dea2f5a | None | None          | power off   | available       | False       |
| 9ab3df2c-4a5c-4951-b24a-3eb42f116122 | None | None          | power off   | available       | False       |
| be19bfdf-1422-4df9-bcc2-a2b875ae3b9e | None | None          | power off   | available       | False       |
| 13c64e11-1e13-4390-bcaf-4f81ae043e7e | None | None          | power off   | available       | False       |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+


Version-Release number of selected component (if applicable):
$ rpm -qa | grep openstack-ironic
openstack-ironic-api-2015.1.0-4.el7ost.noarch
openstack-ironic-conductor-2015.1.0-4.el7ost.noarch
openstack-ironic-common-2015.1.0-4.el7ost.noarch
openstack-ironic-discoverd-1.1.0-3.el7ost.noarch



How reproducible:
100%

Steps to Reproduce:
1.install the undercloud
2.reboot the VM
3. register nodes
4. discover nodes 

Actual results:

ironic services stopped
Expected results:
ironic services are up and running.
discovery process succeeded.

Additional info:
$ sudo systemctl status openstack-ironic-discoverd
openstack-ironic-discoverd.service - Hardware introspection service for OpenStack Ironic
   Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-discoverd.service; enabled)
   Active: failed (Result: exit-code) since Mon 2015-06-08 07:03:56 EDT; 50min ago
  Process: 490 ExecStart=/usr/bin/ironic-discoverd --config-file /etc/ironic-discoverd/discoverd.conf (code=exited, status=1/FAILURE)
 Main PID: 490 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/openstack-ironic-discoverd.service

Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: self.authenticate()
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: File "/usr/lib/python2.7/site-packages/keyst...er
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: return func(*args, **kwargs)
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: File "/usr/lib/python2.7/site-packages/keyst...te
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: resp = self.get_raw_token_from_identity_serv...s)
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: File "/usr/lib/python2.7/site-packages/keyst...ce
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: _("Authorization Failed: %s") % e)
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: keystoneclient.openstack.common.apiclient.ex...ns
Jun 08 07:03:56 instack.localdomain systemd[1]: openstack-ironic-discoverd.service: main process exite...URE
Jun 08 07:03:56 instack.localdomain systemd[1]: Unit openstack-ironic-discoverd.service entered failed...te.
Hint: Some lines were ellipsized, use -l to show in full.
[stack@instack ~]$ sudo systemctl status openstack-keystone
openstack-keystone.service - OpenStack Identity Service (code-named Keystone)
   Loaded: loaded (/usr/lib/systemd/system/openstack-keystone.service; enabled)
   Active: active (running) since Mon 2015-06-08 07:03:56 EDT; 50min ago
 Main PID: 1075 (keystone-all)
   CGroup: /system.slice/openstack-keystone.service
           ├─1075 /usr/bin/python /usr/bin/keystone-all
           ├─1734 /usr/bin/python /usr/bin/keystone-all
           ├─1735 /usr/bin/python /usr/bin/keystone-all
           ├─1736 /usr/bin/python /usr/bin/keystone-all
           └─1737 /usr/bin/python /usr/bin/keystone-all

Jun 08 07:03:37 instack.localdomain systemd[1]: Starting OpenStack Identity Service (code-named Keystone)...
Jun 08 07:03:56 instack.localdomain systemd[1]: Started OpenStack Identity Service (code-named Keystone).
Hint: Some lines were ellipsized, use -l to show in full.
[stack@instack ~]$ sudo systemctl status openstack-ironic-api
openstack-ironic-api.service - OpenStack Ironic API service
   Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-api.service; enabled)
   Active: active (running) since Mon 2015-06-08 07:03:37 EDT; 50min ago
 Main PID: 1042 (ironic-api)
   CGroup: /system.slice/openstack-ironic-api.service
           └─1042 /usr/bin/python2 /usr/bin/ironic-api

Jun 08 07:52:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:52:55.046 1042 DEBUG keystonemiddl...296
Jun 08 07:52:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:52:55.046 1042 DEBUG keystonemiddl...603
Jun 08 07:52:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:52:55.046 1042 DEBUG keystonemiddlewa...
Jun 08 07:52:55 instack.localdomain ironic-api[1042]: 192.0.2.1 - - [08/Jun/2015 07:52:55] "GET /v1/no...391
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:53:55.045 1042 DEBUG keystonemiddl...X-S
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:53:55.045 1042 DEBUG keystonemiddl...581
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:53:55.046 1042 DEBUG keystonemiddl...296
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:53:55.046 1042 DEBUG keystonemiddl...603
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:53:55.046 1042 DEBUG keystonemiddlewa...
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 192.0.2.1 - - [08/Jun/2015 07:53:55] "GET /v1/no...391
Hint: Some lines were ellipsized, use -l to show in full.


the output of  journalctl -b -u openstack-ironic-discoverd -u openstack-keystone -u openstack-ironic-api is attached

Comment 3 Ola Pavlenko 2015-06-08 12:11:22 UTC

Created attachment 1036348 [details]
journal.log

Comment 4 Dmitry Tantsur 2015-06-08 12:17:58 UTC

instack-undercloud patch to increase timeout: https://review.gerrithub.io/#/c/235907/

Comment 6 Udi Kalifon 2015-06-29 16:22:21 UTC

Verified in puddle 2015-06-26:
openstack-ironic-discoverd-1.1.0-4.el7ost.noarch

Comment 8 errata-xmlrpc 2015-08-05 13:52:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549