Bug 1229296 - ironic services did not start up after reboot
Summary: ironic services did not start up after reboot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: Director
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: ga
: Director
Assignee: Dmitry Tantsur
QA Contact: Ola Pavlenko
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-08 12:10 UTC by Ola Pavlenko
Modified: 2015-08-05 13:52 UTC (History)
10 users (show)

Fixed In Version: instack-undercloud-2.1.2-2.el7ost
Doc Type: Bug Fix
Doc Text:
The openstack-ironic-discoverd service would check for openstack-ironic-api's presence on start up. Due to start up ordering, openstack-ironic-discoverd failed to detect openstack-ironic-api on reboot. This fix removes the need to check for openstack-ironic-api's presence from openstack-ironic-discoverd start up code. All services start successfully now on reboot.
Clone Of:
Environment:
Last Closed: 2015-08-05 13:52:50 UTC
Target Upstream Version:


Attachments (Terms of Use)
journal.log (302.80 KB, text/x-vhdl)
2015-06-08 12:11 UTC, Ola Pavlenko
no flags Details


Links
System ID Priority Status Summary Last Updated
Gerrithub.io 235907 None None None Never
Red Hat Product Errata RHEA-2015:1549 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform director Release 2015-08-05 17:49:10 UTC

Description Ola Pavlenko 2015-06-08 12:10:51 UTC
Description of problem:
installed poodle following  https://repos.fedorapeople.org/repos/openstack-m/docs/internal/master/installation/installing.html then rebooted the instack VM
and ironic services didn't start. 
while trying to discover node details got :
$ instack-ironic-deployment --discover-nodes
Preparing for deployment...
  Discovering nodes.
    Sending node ID 55d21860-66c6-4977-ad07-0e343dea2f5a to discoverd for discovery ... ERROR: openstack ('Connection aborted.', error(111, 'Connection refused'))

when node registration worked:
$ instack-ironic-deployment --nodes-json instackenv.json --register-nodes
Preparing for deployment...
  Registering nodes from instackenv.json
  Nodes registered.

+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID | Power State | Provision State | Maintenance |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| 55d21860-66c6-4977-ad07-0e343dea2f5a | None | None          | power off   | available       | False       |
| 9ab3df2c-4a5c-4951-b24a-3eb42f116122 | None | None          | power off   | available       | False       |
| be19bfdf-1422-4df9-bcc2-a2b875ae3b9e | None | None          | power off   | available       | False       |
| 13c64e11-1e13-4390-bcaf-4f81ae043e7e | None | None          | power off   | available       | False       |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+


Version-Release number of selected component (if applicable):
$ rpm -qa | grep openstack-ironic
openstack-ironic-api-2015.1.0-4.el7ost.noarch
openstack-ironic-conductor-2015.1.0-4.el7ost.noarch
openstack-ironic-common-2015.1.0-4.el7ost.noarch
openstack-ironic-discoverd-1.1.0-3.el7ost.noarch



How reproducible:
100%

Steps to Reproduce:
1.install the undercloud
2.reboot the VM
3. register nodes
4. discover nodes 

Actual results:

ironic services stopped
Expected results:
ironic services are up and running.
discovery process succeeded.

Additional info:
$ sudo systemctl status openstack-ironic-discoverd
openstack-ironic-discoverd.service - Hardware introspection service for OpenStack Ironic
   Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-discoverd.service; enabled)
   Active: failed (Result: exit-code) since Mon 2015-06-08 07:03:56 EDT; 50min ago
  Process: 490 ExecStart=/usr/bin/ironic-discoverd --config-file /etc/ironic-discoverd/discoverd.conf (code=exited, status=1/FAILURE)
 Main PID: 490 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/openstack-ironic-discoverd.service

Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: self.authenticate()
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: File "/usr/lib/python2.7/site-packages/keyst...er
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: return func(*args, **kwargs)
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: File "/usr/lib/python2.7/site-packages/keyst...te
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: resp = self.get_raw_token_from_identity_serv...s)
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: File "/usr/lib/python2.7/site-packages/keyst...ce
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: _("Authorization Failed: %s") % e)
Jun 08 07:03:56 instack.localdomain ironic-discoverd[490]: keystoneclient.openstack.common.apiclient.ex...ns
Jun 08 07:03:56 instack.localdomain systemd[1]: openstack-ironic-discoverd.service: main process exite...URE
Jun 08 07:03:56 instack.localdomain systemd[1]: Unit openstack-ironic-discoverd.service entered failed...te.
Hint: Some lines were ellipsized, use -l to show in full.
[stack@instack ~]$ sudo systemctl status openstack-keystone
openstack-keystone.service - OpenStack Identity Service (code-named Keystone)
   Loaded: loaded (/usr/lib/systemd/system/openstack-keystone.service; enabled)
   Active: active (running) since Mon 2015-06-08 07:03:56 EDT; 50min ago
 Main PID: 1075 (keystone-all)
   CGroup: /system.slice/openstack-keystone.service
           ├─1075 /usr/bin/python /usr/bin/keystone-all
           ├─1734 /usr/bin/python /usr/bin/keystone-all
           ├─1735 /usr/bin/python /usr/bin/keystone-all
           ├─1736 /usr/bin/python /usr/bin/keystone-all
           └─1737 /usr/bin/python /usr/bin/keystone-all

Jun 08 07:03:37 instack.localdomain systemd[1]: Starting OpenStack Identity Service (code-named Keystone)...
Jun 08 07:03:56 instack.localdomain systemd[1]: Started OpenStack Identity Service (code-named Keystone).
Hint: Some lines were ellipsized, use -l to show in full.
[stack@instack ~]$ sudo systemctl status openstack-ironic-api
openstack-ironic-api.service - OpenStack Ironic API service
   Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-api.service; enabled)
   Active: active (running) since Mon 2015-06-08 07:03:37 EDT; 50min ago
 Main PID: 1042 (ironic-api)
   CGroup: /system.slice/openstack-ironic-api.service
           └─1042 /usr/bin/python2 /usr/bin/ironic-api

Jun 08 07:52:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:52:55.046 1042 DEBUG keystonemiddl...296
Jun 08 07:52:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:52:55.046 1042 DEBUG keystonemiddl...603
Jun 08 07:52:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:52:55.046 1042 DEBUG keystonemiddlewa...
Jun 08 07:52:55 instack.localdomain ironic-api[1042]: 192.0.2.1 - - [08/Jun/2015 07:52:55] "GET /v1/no...391
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:53:55.045 1042 DEBUG keystonemiddl...X-S
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:53:55.045 1042 DEBUG keystonemiddl...581
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:53:55.046 1042 DEBUG keystonemiddl...296
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:53:55.046 1042 DEBUG keystonemiddl...603
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 2015-06-08 07:53:55.046 1042 DEBUG keystonemiddlewa...
Jun 08 07:53:55 instack.localdomain ironic-api[1042]: 192.0.2.1 - - [08/Jun/2015 07:53:55] "GET /v1/no...391
Hint: Some lines were ellipsized, use -l to show in full.


the output of  journalctl -b -u openstack-ironic-discoverd -u openstack-keystone -u openstack-ironic-api is attached

Comment 3 Ola Pavlenko 2015-06-08 12:11:22 UTC
Created attachment 1036348 [details]
journal.log

Comment 4 Dmitry Tantsur 2015-06-08 12:17:58 UTC
instack-undercloud patch to increase timeout: https://review.gerrithub.io/#/c/235907/

Comment 6 Udi 2015-06-29 16:22:21 UTC
Verified in puddle 2015-06-26:
openstack-ironic-discoverd-1.1.0-4.el7ost.noarch

Comment 8 errata-xmlrpc 2015-08-05 13:52:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549


Note You need to log in before you can comment on or make changes to this bug.