Bug 1356107
| Summary: | nova-compute service is down on compute nodes added post 8->9 upgrade (tripleO Heat Template) | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> | |
| Component: | openstack-tripleo-heat-templates | Assignee: | Giulio Fidente <gfidente> | |
| Status: | CLOSED ERRATA | QA Contact: | Marius Cornea <mcornea> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 9.0 (Mitaka) | CC: | dbecker, dmacpher, eglynn, gfidente, jefbrown, jjoyce, mburns, mcornea, morazi, rhel-osp-director-maint, sasha, scohen, seb, sgordon, svanders, tvignaud | |
| Target Milestone: | ga | Keywords: | Reopened | |
| Target Release: | 9.0 (Mitaka) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-tripleo-heat-templates-2.0.0-29.el7ost | Doc Type: | Bug Fix | |
| Doc Text: |
OpenStack Platform 9 deployments require an additional CephX key for the "client.openstack" user. However, the director's command line client does not generate this key for existing deployments and updates the "ceph.openstack" keyring with an empty secret. Before upgrading, generate a new CephX key and pass it to the deployment using the CephClientKey parameter in an environment file. For example:
parameter_defaults:
CephClientKey: 'my_cephx_key'
Generate the new key with the following command:
$ ceph-authtool --gen-print-key
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1363645 1363650 (view as bug list) | Environment: | ||
| Last Closed: | 2016-08-11 11:36:10 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1353971, 1363645 | |||
| Bug Blocks: | ||||
|
Description
Marius Cornea
2016-07-13 12:21:23 UTC
So, how was the health state of the underlying CEPH storage when the problem is reproduced? Comment 3 suggest that this could be the underlying issue. Is this more than a gut feeling? During my tests, before starting the upgrade process I've successfully launched an instance on the OSP8 overcloud which is still accessible post upgrade so I believe the ceph storage isn't an issue here. Unfortunately the environment is currently blocked for another BZ investigation but once it's available I can use it to reproduce the issue if the sosreports don't help. Any update here? Not from my side. Are you still going to reproduce this again in your environment? Yes, I'm preparing an environment today. I'll get back with the credentials once it's ready. Checked that the issue doesn't reproduce on clean deployment of 9 + scale out. (In reply to Alexander Chuzhoy from comment #11) > Checked that the issue doesn't reproduce on clean deployment of 9 + scale > out. @sasha: can you clarify what you mean by by "clean deployment of 9"? Specifically, did you mean a non-upgraded deployment of OSP9? (In reply to Eoghan Glynn from comment #12) > (In reply to Alexander Chuzhoy from comment #11) > > Checked that the issue doesn't reproduce on clean deployment of 9 + scale > > out. > > @sasha: can you clarify what you mean by by "clean deployment of 9"? > > Specifically, did you mean a non-upgraded deployment of OSP9? Exactly. It looks like nova-compute on the new node keeps trying to get started but it fails (segfault in librados): /var/log/messages shows the following: Jul 27 07:16:54 localhost systemd: Started OpenStack Nova Compute Server. Jul 27 07:16:54 localhost kernel: nova-compute[18765]: segfault at 0 ip 00007f1510d8217a sp 00007f150ab093b0 error 4 in librados.so.2.0.0[7f1510a54000+504000] Jul 27 07:16:54 localhost journal: End of file while reading data: Input/output error Jul 27 07:16:54 localhost systemd: openstack-nova-compute.service: main process exited, code=killed, status=11/SEGV Jul 27 07:16:54 localhost systemd: Unit openstack-nova-compute.service entered failed state. Jul 27 07:16:54 localhost systemd: openstack-nova-compute.service failed. Jul 27 07:16:54 localhost systemd: openstack-nova-compute.service holdoff time over, scheduling restart. Jul 27 07:16:54 localhost systemd: Starting OpenStack Nova Compute Server... The /etc/ceph/ceph.client.openstack.keyring on the upgraded compute node had a wrong format.
It was like so:
[client.openstack]
key = AAAAAAAAAAAAAAAA
Where a valid key is:
[client.openstack]
key = AQBHOJdXAAAAABAAod6tL8beRSB1IasVq0FywQ==
Using the proper key on the compute node and the process starts normally.
So I'm closing this since this is not a bug.
I'm reopening it as we need to figure out how the ceph.client.openstack.keyring got generated for the new node, probably it changed between releases. After passing an environment file containing: parameter_defaults: CephClientKey: 'AQBFZqRXGHWdIRAAqHuCLQgXHQYHVl3jQSFqWg==' to the overcloud deploy commands I was able to scale out with an additional compute node on which nova service got started ok: [stack@undercloud ~]$ nova service-list /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 172.16.18.25 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 172.16.18.25 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning +-----+------------------+------------------------------------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +-----+------------------+------------------------------------+----------+---------+-------+----------------------------+-----------------+ | 2 | nova-scheduler | overcloud-controller-0.localdomain | internal | enabled | up | 2016-08-05T15:47:28.000000 | - | | 5 | nova-scheduler | overcloud-controller-2.localdomain | internal | enabled | up | 2016-08-05T15:47:27.000000 | - | | 8 | nova-scheduler | overcloud-controller-1.localdomain | internal | enabled | up | 2016-08-05T15:47:29.000000 | - | | 11 | nova-consoleauth | overcloud-controller-0.localdomain | internal | enabled | up | 2016-08-05T15:47:32.000000 | - | | 14 | nova-conductor | overcloud-controller-1.localdomain | internal | enabled | up | 2016-08-05T15:47:31.000000 | - | | 23 | nova-consoleauth | overcloud-controller-2.localdomain | internal | enabled | up | 2016-08-05T15:47:33.000000 | - | | 26 | nova-consoleauth | overcloud-controller-1.localdomain | internal | enabled | up | 2016-08-05T15:47:33.000000 | - | | 29 | nova-conductor | overcloud-controller-0.localdomain | internal | enabled | up | 2016-08-05T15:47:31.000000 | - | | 41 | nova-conductor | overcloud-controller-2.localdomain | internal | enabled | up | 2016-08-05T15:47:30.000000 | - | | 44 | nova-compute | overcloud-compute-0.localdomain | nova | enabled | up | 2016-08-05T15:47:29.000000 | - | | 103 | nova-compute | overcloud-compute-1.localdomain | nova | enabled | up | 2016-08-05T15:47:29.000000 | - | +-----+------------------+------------------------------------+----------+---------+-------+----------------------------+-----------------+ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-1599.html |