Hide Forgot
[rhel-osp-director][upgrade to 8.0]: After running openstack undercloud upgrade complete the openstack-keystone.service down and failed to start. Environment: ------------- instack-0.0.8-2.el7ost.noarch instack-undercloud-2.2.7-1.el7ost.noarch openstack-tripleo-heat-templates-0.8.13-1.el7ost.noarch openstack-heat-common-5.0.1-4.el7ost.noarch openstack-heat-api-5.0.1-4.el7ost.noarch heat-cfntools-1.2.8-2.el7.noarch openstack-heat-api-cfn-5.0.1-4.el7ost.noarch openstack-heat-api-cloudwatch-5.0.1-4.el7ost.noarch openstack-tripleo-heat-templates-kilo-0.8.13-1.el7ost.noarch python-heatclient-1.0.0-1.el7ost.noarch openstack-heat-engine-5.0.1-4.el7ost.noarch openstack-heat-templates-0-0.8.20150605git.el7ost.noarch python-keystone-8.0.1-1.el7ost.noarch python-keystoneauth1-1.1.0-4.el7ost.noarch python-keystonemiddleware-2.3.1-1.el7ost.noarch openstack-keystone-8.0.1-1.el7ost.noarch python-keystoneclient-1.7.2-1.el7ost.noarch Description : -------------- Attempt to Upgrade from 7.3GA to 8.0 latest on Bare-Metal environment, After the openstack undercloud upgrade complete successfully, it seems that the keystone-service is down. --------------------- END PROFILING --------------------- [2016-02-10 19:29:05,583] (os-refresh-config) [INFO] Completed phase post-configure os-refresh-config completed successfully Not creating flavor "baremetal" because it already exists. Created flavor "control" with profile "control" Created flavor "compute" with profile "compute" Created flavor "ceph-storage" with profile "ceph-storage" Created flavor "block-storage" with profile "block-storage" Created flavor "swift-storage" with profile "swift-storage" ############################################################################# Undercloud install complete. The file containing this installation's passwords is at /home/stack/undercloud-passwords.conf. There is also a stackrc file at /home/stack/stackrc. These files are needed to interact with the OpenStack services, and should be secured. ############################################################################# [stack@undercloud72 ~]$ [stack@undercloud72 ~]$ openstack undercloud upgrade^C [stack@undercloud72 ~]$ sudo systemctl|grep -i fail ● NetworkManager-wait-online.service loaded failed failed Network Manager Wait Online ● openstack-keystone.service loaded failed failed OpenStack Identity Service (code-named Keystone) ● ovirt-guest-agent.service loaded failed failed oVirt Guest Agent [root@undercloud72 keystone]# systemctl status openstack-keystone.service openstack-keystone.service - OpenStack Identity Service (code-named Keystone) Loaded: loaded (/usr/lib/systemd/system/openstack-keystone.service; disabled; vendor preset: disabled) Active: activating (start) since Wed 2016-02-10 19:58:33 EST; 772ms ago Main PID: 11825 (keystone-all) CGroup: /system.slice/openstack-keystone.service └─11825 /usr/bin/python2 /usr/bin/keystone-all Feb 10 19:58:33 undercloud72.localdomain systemd[1]: Starting OpenStack Identity Service (code-named Keystone)... [root@undercloud72 keystone]# systemctl start openstack-keystone.service Job for openstack-keystone.service failed because the control process exited with error code. See "systemctl status openstack-keystone.service" and "journalctl -xe" for details. messages (attached): --------------------- Feb 10 19:12:43 undercloud72 ironic-api: 192.168.0.1 - - [10/Feb/2016 19:12:43] "GET / HTTP/1.0" 200 346 Feb 10 19:12:43 undercloud72 systemd: Stopping OpenStack Identity Service (code-named Keystone)... Feb 10 19:12:44 undercloud72 systemd: Starting OpenStack Identity Service (code-named Keystone)... Feb 10 19:12:44 undercloud72 ceilometer-alarm-notifier: No handlers could be found for logger "oslo_config.cfg" Feb 10 19:12:44 undercloud72 systemd: openstack-ceilometer-alarm-notifier.service: main process exited, code=exited, status=1/FAILURE Feb 10 19:12:44 undercloud72 systemd: Unit openstack-ceilometer-alarm-notifier.service entered failed state. Feb 10 19:12:44 undercloud72 systemd: openstack-ceilometer-alarm-notifier.service failed. Feb 10 19:12:44 undercloud72 ceilometer-alarm-evaluator: No handlers could be found for logger "oslo_config.cfg" Feb 10 19:12:44 undercloud72 systemd: openstack-ceilometer-alarm-evaluator.service: main process exited, code=exited, status=1/FAILURE Feb 10 19:12:44 undercloud72 systemd: Unit openstack-ceilometer-alarm-evaluator.service entered failed state. Feb 10 19:12:44 undercloud72 systemd: openstack-ceilometer-alarm-evaluator.service failed. Feb 10 19:12:45 undercloud72 keystone-all: No handlers could be found for logger "oslo_config.cfg" Feb 10 19:12:45 undercloud72 systemd: openstack-keystone.service: main process exited, code=exited, status=1/FAILURE Feb 10 19:12:45 undercloud72 systemd: Failed to start OpenStack Identity Service (code-named Keystone). Feb 10 19:12:45 undercloud72 systemd: Unit openstack-keystone.service entered failed state. Feb 10 19:12:45 undercloud72 systemd: openstack-keystone.service failed. Feb 10 19:12:45 undercloud72 ironic-api: 192.168.0.1 - - [10/Feb/2016 19:12:45] "GET / HTTP/1.0" 200 346 Feb 10 19:12:45 undercloud72 ironic-api: Traceback (most recent call last): Feb 10 19:12:45 undercloud72 ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 86, in run Feb 10 19:12:45 undercloud72 ironic-api: self.finish_response() Feb 10 19:12:45 undercloud72 ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 128, in finish_response Feb 10 19:12:45 undercloud72 ironic-api: self.write(data) Feb 10 19:12:45 undercloud72 ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 212, in write Feb 10 19:12:45 undercloud72 ironic-api: self.send_headers() Feb 10 19:12:45 undercloud72 ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 270, in send_headers Feb 10 19:12:45 undercloud72 ironic-api: self.send_preamble() Feb 10 19:12:45 undercloud72 ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 197, in send_preamble Feb 10 19:12:45 undercloud72 ironic-api: self._write('Server: %s\r\n' % self.server_software) Feb 10 19:12:45 undercloud72 ironic-api: File "/usr/lib64/python2.7/socket.py", line 324, in write Feb 10 19:12:45 undercloud72 ironic-api: self.flush() Feb 10 19:12:45 undercloud72 ironic-api: File "/usr/lib64/python2.7/socket.py", line 303, in flush Feb 10 19:12:45 undercloud72 ironic-api: self._sock.sendall(view[write_offset:write_offset+buffer_size]) Feb 10 19:12:45 undercloud72 ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in sendall Feb 10 19:12:45 undercloud72 ironic-api: tail = self.send(data, flags) Feb 10 19:12:45 undercloud72 ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send Feb 10 19:12:45 undercloud72 ironic-api: total_sent += fd.send(data[total_sent:], flags) Feb 10 19:12:45 undercloud72 ironic-api: error: [Errno 104] Connection reset by peer :
Seems like the openstack-keystone should be down. It shows failed in the output from systemctl and it's impossible to stop the service - remained failed.
Environment: openstack-tripleo-heat-templates-kilo-0.8.13-1.el7ost.noarch python-keystoneclient-1.7.2-1.el7ost.noarch python-keystoneauth1-1.1.0-4.el7ost.noarch instack-undercloud-2.2.7-1.el7ost.noarch python-keystonemiddleware-2.3.1-1.el7ost.noarch openstack-keystone-8.0.1-1.el7ost.noarch openstack-puppet-modules-7.0.16-1.el7ost.noarch openstack-tripleo-heat-templates-0.8.13-1.el7ost.noarch python-keystone-8.0.1-1.el7ost.noarch Didn't reproduce the issue.
it is expected that the openstack-keystone service will be stopped after the upgrade, as keystone now runs under httpd on the undercloud in osp-d 8. there could be something in the puppet manifest apply that is causing the service to go to the failed state instead of just saying it is stopped. is there anything else wrong? an error when you try to use keystone? are there any other symptoms that prevent actual usage? (note that the ironic-api error in the messages output you pasted is unrelated and there is already a bug open on that).
Emilien, assigning this one to you since it's related to the patch you worked on to move keystone from eventlet to httpd, and this problem is encountered during upgrading.
please provide puppet logs, we need to check if keystone is stopped before starting apache, like we want in: https://github.com/openstack/puppet-keystone/blob/stable/liberty/manifests/init.pp#L924 Thanks
I think we determined this was due to a bad environment and incorrect steps. Please reopen if it reproduces.
Reproduced with: openstack-tripleo-heat-templates-0.8.14-5.el7ost.noarch openstack-tripleo-heat-templates-kilo-0.8.14-5.el7ost.noarch openstack-puppet-modules-7.0.17-1.el7ost.noarch instack-undercloud-2.2.7-2.el7ost.noarch
(In reply to Alexander Chuzhoy from comment #7) > Reproduced with: > openstack-tripleo-heat-templates-0.8.14-5.el7ost.noarch > openstack-tripleo-heat-templates-kilo-0.8.14-5.el7ost.noarch > openstack-puppet-modules-7.0.17-1.el7ost.noarch > instack-undercloud-2.2.7-2.el7ost.noarch ok, please provide the needinfo from comment 5
Created attachment 1143468 [details] some logs from the undercloud.
This bug did not make the OSP 8.0 release. It is being deferred to OSP 10.
Not reproducable.
closed, no need for needinfo.