1491611 – OSP11 -> OSP12 upgrade: post upgrade nova-compute services is disabled on 1/2 compute nodes with: Auto-disabled due to 10 build failures reason

Bug 1491611 - OSP11 -> OSP12 upgrade: post upgrade nova-compute services is disabled on 1/2 compute nodes with: Auto-disabled due to 10 build failures reason

Summary: OSP11 -> OSP12 upgrade: post upgrade nova-compute services is disabled on 1/2...

Keywords:
Status:	CLOSED DUPLICATE of bug 1477770
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	rhosp-director
Sub Component:
Version:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	ga
Target Release:	12.0 (Pike)
Assignee:	Ollie Walsh
QA Contact:	Amit Ugol
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-09-14 09:20 UTC by Marius Cornea
Modified:	2017-10-06 14:41 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-10-06 14:41:48 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
nova-compute log from host (2.16 MB, text/plain) 2017-09-14 09:20 UTC, Marius Cornea	no flags	Details
View All

Description Marius Cornea 2017-09-14 09:20:03 UTC

Created attachment 1325850 [details]
nova-compute log from host

Description of problem:
OSP11 -> OSP12 upgrade: post upgrade nova-compute services is disabled on 1/2 compute nodes with: Auto-disabled due to 10 build failures reason:

(overcloud) [stack@undercloud-0 ~]$ nova service-list
+--------------------------------------+------------------+--------------------------+----------+----------+-------+----------------------------+----------------------------------------+-------------+
| Id                                   | Binary           | Host                     | Zone     | Status   | State | Updated_at                 | Disabled Reason                        | Forced down |
+--------------------------------------+------------------+--------------------------+----------+----------+-------+----------------------------+----------------------------------------+-------------+
| ccbbbf4d-1932-49f9-94c0-33078409e645 | nova-conductor   | controller-1.localdomain | internal | enabled  | up    | 2017-09-14T09:08:58.000000 | -                                      | False       |
| de82e518-9db6-4241-a75a-63f099c17b82 | nova-compute     | compute-0.localdomain    | nova     | enabled  | up    | 2017-09-14T09:09:03.000000 | -                                      | False       |
| 3105036a-d12b-4d23-a09a-86c9b1feaa35 | nova-conductor   | controller-2.localdomain | internal | enabled  | up    | 2017-09-14T09:09:01.000000 | -                                      | False       |
| 50304f57-f647-46ed-9487-a678d318a3ec | nova-scheduler   | controller-1.localdomain | internal | enabled  | up    | 2017-09-14T09:09:04.000000 | -                                      | False       |
| 167a03b1-2f13-4f03-8b21-4cf410836535 | nova-consoleauth | controller-1.localdomain | internal | enabled  | up    | 2017-09-14T09:08:57.000000 | -                                      | False       |
| 12a51499-3939-4c25-b8d1-5a15897eb003 | nova-scheduler   | controller-2.localdomain | internal | enabled  | up    | 2017-09-14T09:09:03.000000 | -                                      | False       |
| deae93d6-35b0-4d20-8c6c-9228e4facfc9 | nova-consoleauth | controller-2.localdomain | internal | enabled  | up    | 2017-09-14T09:09:03.000000 | -                                      | False       |
| aa8e58b5-74d6-4b07-9949-038c9182541a | nova-compute     | compute-1.localdomain    | nova     | disabled | up    | 2017-09-14T09:09:01.000000 | Auto-disabled due to 10 build failures | False       |
| 99db0344-7dff-4337-99d6-fd0b3704a3e6 | nova-conductor   | controller-0.localdomain | internal | enabled  | up    | 2017-09-14T09:09:02.000000 | -                                      | False       |
| 5fcb35af-5c9a-4d26-8cdb-a1d014f070d2 | nova-scheduler   | controller-0.localdomain | internal | enabled  | up    | 2017-09-14T09:09:05.000000 | -                                      | False       |
| 308b805f-552e-44c8-a3d1-393679e906f8 | nova-consoleauth | controller-0.localdomain | internal | enabled  | up    | 2017-09-14T09:09:02.000000 | -                                      | False       |
| ccbbbf4d-1932-49f9-94c0-33078409e645 | nova-conductor   | controller-1.localdomain | internal | enabled  | up    | 2017-09-14T09:08:58.000000 | -                                      | False       |
| de82e518-9db6-4241-a75a-63f099c17b82 | nova-compute     | compute-0.localdomain    | nova     | enabled  | up    | 2017-09-14T09:09:03.000000 | -                                      | False       |
| 3105036a-d12b-4d23-a09a-86c9b1feaa35 | nova-conductor   | controller-2.localdomain | internal | enabled  | up    | 2017-09-14T09:09:01.000000 | -                                      | False       |
| 50304f57-f647-46ed-9487-a678d318a3ec | nova-scheduler   | controller-1.localdomain | internal | enabled  | up    | 2017-09-14T09:09:04.000000 | -                                      | False       |
| 167a03b1-2f13-4f03-8b21-4cf410836535 | nova-consoleauth | controller-1.localdomain | internal | enabled  | up    | 2017-09-14T09:08:57.000000 | -                                      | False       |
| 12a51499-3939-4c25-b8d1-5a15897eb003 | nova-scheduler   | controller-2.localdomain | internal | enabled  | up    | 2017-09-14T09:09:03.000000 | -                                      | False       |
| deae93d6-35b0-4d20-8c6c-9228e4facfc9 | nova-consoleauth | controller-2.localdomain | internal | enabled  | up    | 2017-09-14T09:09:03.000000 | -                                      | False       |
| aa8e58b5-74d6-4b07-9949-038c9182541a | nova-compute     | compute-1.localdomain    | nova     | disabled | up    | 2017-09-14T09:09:01.000000 | Auto-disabled due to 10 build failures | False       |
| 99db0344-7dff-4337-99d6-fd0b3704a3e6 | nova-conductor   | controller-0.localdomain | internal | enabled  | up    | 2017-09-14T09:09:02.000000 | -                                      | False       |
| 5fcb35af-5c9a-4d26-8cdb-a1d014f070d2 | nova-scheduler   | controller-0.localdomain | internal | enabled  | up    | 2017-09-14T09:09:05.000000 | -                                      | False       |
| 308b805f-552e-44c8-a3d1-393679e906f8 | nova-consoleauth | controller-0.localdomain | internal | enabled  | up    | 2017-09-14T09:09:02.000000 | -                                      | False       |
+--------------------------------------+------------------+--------------------------+----------+----------+-------+----------------------------+----------------------------------------+-------------+


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.0-0.20170901051303.0rc1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11 with 3 controller, 2 computes, 3 ceph nodes
2. Upgrade to OSP12
3. Check nova service-list

Actual results:
nova-compute on compute-1.localdomain is disabled with 'Auto-disabled due to 10 build failures' reason

Expected results:
nova-compute on compute-1.localdomain doesn't get disabled during upgrade

Additional info:
Attaching nova-compute logs from compute-1.localdomain:

Comment 1 Marius Cornea 2017-09-14 09:22:17 UTC

[heat-admin@compute-1 ~]$ sudo docker ps
CONTAINER ID        IMAGE                                                                  COMMAND             CREATED             STATUS                    PORTS               NAMES
c440d3c46d07        192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-09-08.3           "kolla_start"       16 hours ago        Up 16 hours                                   logrotate_crond
0cbd961b1ad8        192.168.24.1:8787/rhosp12/openstack-nova-compute-docker:2017-09-08.3   "kolla_start"       16 hours ago        Up 16 hours (unhealthy)                       nova_migration_target
5bf4128ab4e1        192.168.24.1:8787/rhosp12/openstack-nova-compute-docker:2017-09-08.3   "kolla_start"       16 hours ago        Up 16 hours (unhealthy)                       nova_compute
e735376905dd        192.168.24.1:8787/rhosp12/openstack-iscsid-docker:2017-09-08.3         "kolla_start"       16 hours ago        Up 16 hours                                   iscsid
87cca17f10e0        192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-09-08.3   "kolla_start"       16 hours ago        Up 16 hours                                   nova_libvirt
d87be782416a        192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-09-08.3   "kolla_start"       16 hours ago        Up 16 hours                                   nova_virtlogd
[heat-admin@compute-1 ~]$

Comment 2 Lee Yarwood 2017-09-28 07:46:19 UTC

I'm pretty sure this is a duplicate of BZ 1477770 right? The duplicate compute service records are the issue here, the fact they are auto-disabled is just a side-effect of there being multiple records per compute host.

Comment 3 Ollie Walsh 2017-10-06 14:39:50 UTC

Need /var/log/containers/nova/nova-compute.log for the post-upgrade logs.

Also was libvirtd started manually during the upgrade?

Comment 4 Marius Cornea 2017-10-06 14:41:48 UTC

(In reply to Ollie Walsh from comment #3)
> Need /var/log/containers/nova/nova-compute.log for the post-upgrade logs.
> 
> Also was libvirtd started manually during the upgrade?

I'll close this as a duplicate of 1477770. I wasn't able to reproduce it after applying the patches for that.

*** This bug has been marked as a duplicate of bug 1477770 ***

Note You need to log in before you can comment on or make changes to this bug.