Bug 1491611
| Summary: | OSP11 -> OSP12 upgrade: post upgrade nova-compute services is disabled on 1/2 compute nodes with: Auto-disabled due to 10 build failures reason | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> | ||||
| Component: | rhosp-director | Assignee: | Ollie Walsh <owalsh> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Amit Ugol <augol> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 12.0 (Pike) | CC: | dbecker, lyarwood, mburns, mcornea, morazi, rhel-osp-director-maint | ||||
| Target Milestone: | ga | Keywords: | Triaged | ||||
| Target Release: | 12.0 (Pike) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-10-06 14:41:48 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
[heat-admin@compute-1 ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c440d3c46d07 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-09-08.3 "kolla_start" 16 hours ago Up 16 hours logrotate_crond 0cbd961b1ad8 192.168.24.1:8787/rhosp12/openstack-nova-compute-docker:2017-09-08.3 "kolla_start" 16 hours ago Up 16 hours (unhealthy) nova_migration_target 5bf4128ab4e1 192.168.24.1:8787/rhosp12/openstack-nova-compute-docker:2017-09-08.3 "kolla_start" 16 hours ago Up 16 hours (unhealthy) nova_compute e735376905dd 192.168.24.1:8787/rhosp12/openstack-iscsid-docker:2017-09-08.3 "kolla_start" 16 hours ago Up 16 hours iscsid 87cca17f10e0 192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-09-08.3 "kolla_start" 16 hours ago Up 16 hours nova_libvirt d87be782416a 192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-09-08.3 "kolla_start" 16 hours ago Up 16 hours nova_virtlogd [heat-admin@compute-1 ~]$ I'm pretty sure this is a duplicate of BZ 1477770 right? The duplicate compute service records are the issue here, the fact they are auto-disabled is just a side-effect of there being multiple records per compute host. Need /var/log/containers/nova/nova-compute.log for the post-upgrade logs. Also was libvirtd started manually during the upgrade? (In reply to Ollie Walsh from comment #3) > Need /var/log/containers/nova/nova-compute.log for the post-upgrade logs. > > Also was libvirtd started manually during the upgrade? I'll close this as a duplicate of 1477770. I wasn't able to reproduce it after applying the patches for that. *** This bug has been marked as a duplicate of bug 1477770 *** |
Created attachment 1325850 [details] nova-compute log from host Description of problem: OSP11 -> OSP12 upgrade: post upgrade nova-compute services is disabled on 1/2 compute nodes with: Auto-disabled due to 10 build failures reason: (overcloud) [stack@undercloud-0 ~]$ nova service-list +--------------------------------------+------------------+--------------------------+----------+----------+-------+----------------------------+----------------------------------------+-------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down | +--------------------------------------+------------------+--------------------------+----------+----------+-------+----------------------------+----------------------------------------+-------------+ | ccbbbf4d-1932-49f9-94c0-33078409e645 | nova-conductor | controller-1.localdomain | internal | enabled | up | 2017-09-14T09:08:58.000000 | - | False | | de82e518-9db6-4241-a75a-63f099c17b82 | nova-compute | compute-0.localdomain | nova | enabled | up | 2017-09-14T09:09:03.000000 | - | False | | 3105036a-d12b-4d23-a09a-86c9b1feaa35 | nova-conductor | controller-2.localdomain | internal | enabled | up | 2017-09-14T09:09:01.000000 | - | False | | 50304f57-f647-46ed-9487-a678d318a3ec | nova-scheduler | controller-1.localdomain | internal | enabled | up | 2017-09-14T09:09:04.000000 | - | False | | 167a03b1-2f13-4f03-8b21-4cf410836535 | nova-consoleauth | controller-1.localdomain | internal | enabled | up | 2017-09-14T09:08:57.000000 | - | False | | 12a51499-3939-4c25-b8d1-5a15897eb003 | nova-scheduler | controller-2.localdomain | internal | enabled | up | 2017-09-14T09:09:03.000000 | - | False | | deae93d6-35b0-4d20-8c6c-9228e4facfc9 | nova-consoleauth | controller-2.localdomain | internal | enabled | up | 2017-09-14T09:09:03.000000 | - | False | | aa8e58b5-74d6-4b07-9949-038c9182541a | nova-compute | compute-1.localdomain | nova | disabled | up | 2017-09-14T09:09:01.000000 | Auto-disabled due to 10 build failures | False | | 99db0344-7dff-4337-99d6-fd0b3704a3e6 | nova-conductor | controller-0.localdomain | internal | enabled | up | 2017-09-14T09:09:02.000000 | - | False | | 5fcb35af-5c9a-4d26-8cdb-a1d014f070d2 | nova-scheduler | controller-0.localdomain | internal | enabled | up | 2017-09-14T09:09:05.000000 | - | False | | 308b805f-552e-44c8-a3d1-393679e906f8 | nova-consoleauth | controller-0.localdomain | internal | enabled | up | 2017-09-14T09:09:02.000000 | - | False | | ccbbbf4d-1932-49f9-94c0-33078409e645 | nova-conductor | controller-1.localdomain | internal | enabled | up | 2017-09-14T09:08:58.000000 | - | False | | de82e518-9db6-4241-a75a-63f099c17b82 | nova-compute | compute-0.localdomain | nova | enabled | up | 2017-09-14T09:09:03.000000 | - | False | | 3105036a-d12b-4d23-a09a-86c9b1feaa35 | nova-conductor | controller-2.localdomain | internal | enabled | up | 2017-09-14T09:09:01.000000 | - | False | | 50304f57-f647-46ed-9487-a678d318a3ec | nova-scheduler | controller-1.localdomain | internal | enabled | up | 2017-09-14T09:09:04.000000 | - | False | | 167a03b1-2f13-4f03-8b21-4cf410836535 | nova-consoleauth | controller-1.localdomain | internal | enabled | up | 2017-09-14T09:08:57.000000 | - | False | | 12a51499-3939-4c25-b8d1-5a15897eb003 | nova-scheduler | controller-2.localdomain | internal | enabled | up | 2017-09-14T09:09:03.000000 | - | False | | deae93d6-35b0-4d20-8c6c-9228e4facfc9 | nova-consoleauth | controller-2.localdomain | internal | enabled | up | 2017-09-14T09:09:03.000000 | - | False | | aa8e58b5-74d6-4b07-9949-038c9182541a | nova-compute | compute-1.localdomain | nova | disabled | up | 2017-09-14T09:09:01.000000 | Auto-disabled due to 10 build failures | False | | 99db0344-7dff-4337-99d6-fd0b3704a3e6 | nova-conductor | controller-0.localdomain | internal | enabled | up | 2017-09-14T09:09:02.000000 | - | False | | 5fcb35af-5c9a-4d26-8cdb-a1d014f070d2 | nova-scheduler | controller-0.localdomain | internal | enabled | up | 2017-09-14T09:09:05.000000 | - | False | | 308b805f-552e-44c8-a3d1-393679e906f8 | nova-consoleauth | controller-0.localdomain | internal | enabled | up | 2017-09-14T09:09:02.000000 | - | False | +--------------------------------------+------------------+--------------------------+----------+----------+-------+----------------------------+----------------------------------------+-------------+ Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-7.0.0-0.20170901051303.0rc1.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP11 with 3 controller, 2 computes, 3 ceph nodes 2. Upgrade to OSP12 3. Check nova service-list Actual results: nova-compute on compute-1.localdomain is disabled with 'Auto-disabled due to 10 build failures' reason Expected results: nova-compute on compute-1.localdomain doesn't get disabled during upgrade Additional info: Attaching nova-compute logs from compute-1.localdomain: