Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1711479

Summary: Unable to deploy an overcloud with RHOSP15
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: openstack-tripleoAssignee: James Slagle <jslagle>
Status: CLOSED DUPLICATE QA Contact: Arik Chernetsky <achernet>
Severity: low Docs Contact:
Priority: unspecified    
Version: 15.0 (Stein)CC: aschultz, mburns
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-18 03:27:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sosreport-undercloud-0-rhosp15-2019-05-17-xzsfrcd.tar.xz.0 1/3
none
sosreport-undercloud-0-rhosp15-2019-05-17-xzsfrcd.tar.xz.1 2/3
none
sosreport-undercloud-0-rhosp15-2019-05-17-xzsfrcd.tar.xz.2 3/3 none

Description David Hill 2019-05-17 22:19:08 UTC
Description of problem:
Unable to deploy an overcloud with RHOSP15 ...
(undercloud) [stack@undercloud-0-rhosp15 cloud]$ openstack overcloud profiles list

(undercloud) [stack@undercloud-0-rhosp15 cloud]$ 



(undercloud) [stack@undercloud-0-rhosp15 cloud]$ openstack flavor show control
+----------------------------+--------------------------------------------------------------------------------------------------------------------+
| Field                      | Value                                                                                                              |
+----------------------------+--------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                                                                              |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                                                                                  |
| access_project_ids         | None                                                                                                               |
| disk                       | 10                                                                                                                 |
| id                         | e140023e-8956-457e-81af-6a3e71735f37                                                                               |
| name                       | control                                                                                                            |
| os-flavor-access:is_public | True                                                                                                               |
| properties                 | capabilities:boot_mode='bios', capabilities:boot_option='local', capabilities:profile='control', cpu_arch='x86_64' |
| ram                        | 256                                                                                                                |
| rxtx_factor                | 1.0                                                                                                                |
| swap                       | 2048                                                                                                               |
| vcpus                      | 1                                                                                                                  |
+----------------------------+--------------------------------------------------------------------------------------------------------------------+
(undercloud) [stack@undercloud-0-rhosp15 cloud]$ openstack flavor show compute
+----------------------------+--------------------------------------------------------------------------------------------------------------------+
| Field                      | Value                                                                                                              |
+----------------------------+--------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                                                                              |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                                                                                  |
| access_project_ids         | None                                                                                                               |
| disk                       | 10                                                                                                                 |
| id                         | eb81d8ca-c87e-4c5e-879c-5d8401f9a380                                                                               |
| name                       | compute                                                                                                            |
| os-flavor-access:is_public | True                                                                                                               |
| properties                 | capabilities:boot_mode='bios', capabilities:boot_option='local', capabilities:profile='compute', cpu_arch='x86_64' |
| ram                        | 256                                                                                                                |
| rxtx_factor                | 1.0                                                                                                                |
| swap                       | 2048                                                                                                               |
| vcpus                      | 1                                                                                                                  |
+----------------------------+--------------------------------------------------------------------------------------------------------------------+


(undercloud) [stack@undercloud-0-rhosp15 cloud]$ for p in $(ironic node-list  | grep False | awk '{ print $2 }'); do ironic node-show $p | grep profil; done
The "ironic" CLI is deprecated and will be removed in the S* release. Please use the "openstack baremetal" CLI instead.
The "ironic" CLI is deprecated and will be removed in the S* release. Please use the "openstack baremetal" CLI instead.
|                        | 'profile:control,boot_option:local,boot_mode:bios'}                     |
The "ironic" CLI is deprecated and will be removed in the S* release. Please use the "openstack baremetal" CLI instead.
|                        | 'profile:control,boot_option:local,boot_mode:bios'}                     |
The "ironic" CLI is deprecated and will be removed in the S* release. Please use the "openstack baremetal" CLI instead.
|                        | 'profile:control,boot_option:local,boot_mode:bios'}                     |
The "ironic" CLI is deprecated and will be removed in the S* release. Please use the "openstack baremetal" CLI instead.
|                        | 'profile:compute,boot_option:local,boot_mode:bios'}                     |
The "ironic" CLI is deprecated and will be removed in the S* release. Please use the "openstack baremetal" CLI instead.
^[[O|                        | 'profile:compute,boot_option:local,boot_mode:bios'}                     |
The "ironic" CLI is deprecated and will be removed in the S* release. Please use the "openstack baremetal" CLI instead.
|                        | 'profile:compute,boot_option:local,boot_mode:bios'}                     |
The "ironic" CLI is deprecated and will be removed in the S* release. Please use the "openstack baremetal" CLI instead.
|                        | 'profile:compute,boot_option:local,boot_mode:bios'}                     |



Version-Release number of selected component (if applicable):
 [root@undercloud-0-rhosp15 ~]# podman list | egrep "nova|ironic"
cee8450f4f14  192.0.2.1:8787/rhosp15/openstack-nova-compute-ironic:15.0-35        dumb-init --singl...  7 hours ago  Up 7 hours ago         nova_compute
b1193585456b  192.0.2.1:8787/rhosp15/openstack-ironic-inspector:15.0-34           dumb-init --singl...  7 hours ago  Up 7 hours ago         ironic_inspector_dnsmasq
11a8b0693f09  192.0.2.1:8787/rhosp15/openstack-ironic-inspector:15.0-34           dumb-init --singl...  7 hours ago  Up 7 hours ago         ironic_inspector
c64a36c52b14  192.0.2.1:8787/rhosp15/openstack-ironic-pxe:15.0-31                 dumb-init --singl...  7 hours ago  Up 7 hours ago         ironic_pxe_http
5588c2ec2fa7  192.0.2.1:8787/rhosp15/openstack-ironic-pxe:15.0-31                 dumb-init --singl...  7 hours ago  Up 7 hours ago         ironic_pxe_tftp
1f5fa6e4fe7b  192.0.2.1:8787/rhosp15/openstack-ironic-neutron-agent:15.0-31       dumb-init --singl...  7 hours ago  Up 7 hours ago         ironic_neutron_agent
164e75eb6782  192.0.2.1:8787/rhosp15/openstack-ironic-conductor:15.0-33           dumb-init --singl...  7 hours ago  Up 7 hours ago         ironic_conductor
00b747ee8c0a  192.0.2.1:8787/rhosp15/openstack-ironic-api:15.0-31                 dumb-init --singl...  7 hours ago  Up 7 hours ago         ironic_api
6d2cfaed5101  192.0.2.1:8787/rhosp15/openstack-nova-api:15.0-34                   dumb-init --singl...  7 hours ago  Up 7 hours ago         nova_metadata
e75abd63751f  192.0.2.1:8787/rhosp15/openstack-nova-api:15.0-34                   dumb-init --singl...  7 hours ago  Up 7 hours ago         nova_api
a18f2ec90fbf  192.0.2.1:8787/rhosp15/openstack-nova-placement-api:15.0-37         dumb-init --singl...  7 hours ago  Up 7 hours ago         nova_placement
e1c79892b5ee  192.0.2.1:8787/rhosp15/openstack-nova-scheduler:15.0-35             dumb-init --singl...  7 hours ago  Up 7 hours ago         nova_scheduler
77e03eb5ae79  192.0.2.1:8787/rhosp15/openstack-nova-conductor:15.0-37             dumb-init --singl...  7 hours ago  Up 7 hours ago         nova_conductor
76b0133ce924  192.0.2.1:8787/rhosp15/openstack-nova-api:15.0-34                   dumb-init --singl...  7 hours ago  Up 7 hours ago         nova_api_cron


How reproducible:
Always

Steps to Reproduce:
1. Deploy an undercloud
2. Import / introspect nodes / tag them
3.

Actual results:
Unable to deploy the overcloud

Expected results:
Should be able 

Additional info:

Comment 1 David Hill 2019-05-17 22:41:26 UTC
It looks like introspection completed but the command won't return:

(undercloud) [stack@undercloud-0-rhosp15 cloud]$ openstack overcloud node introspect --all-manageable --provide 
Waiting for introspection to finish...
Waiting for messages on queue 'tripleo' with no timeout.
        
Introspection completed.
Waiting for messages on queue 'tripleo' with no timeout.

and it'll eventually timeout after 4 hours.

I see a bunch of ERRORS like this in /var/log/containers/mistral/engine.log


2019-05-17 18:38:19.096 7 INFO workflow_trace [req-43f9a144-3225-4e1f-989d-a3bdf07d43f7 c2fcadaffb9740dd9e01976190dcf06c ad6ea4a2df754e81a58ad41941ea7064 - default default] Task 'cell_v2_discover_hosts' (72708317-2e06-47dd-8ba1-71d0fe9879a3) [RUNNING -> ERROR, msg=Failure caused by error in tasks: wait_for_nova_resources

  wait_for_nova_resources [task_ex_id=3f97b3a3-18bb-4096-98c1-b5b9e5900368] -> One or more actions had failed.
    [action_ex_id=04e3455b-8630-4624-b99b-d60bb435c7ff, idx=0]: The action raised an exception [action_ex_id=04e3455b-8630-4624-b99b-d60bb435c7ff, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '98685e66-868f-47ac-8a4f-f1d0db216914'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '98685e66-868f-47ac-8a4f-f1d0db216914'}. (HTTP 404)
    [action_ex_id=9f1635a3-c86a-4580-a340-9909bf1e1643, idx=1]: The action raised an exception [action_ex_id=9f1635a3-c86a-4580-a340-9909bf1e1643, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': 'ae3869fe-c18b-4b0a-a0a5-16d3235f1b90'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': 'ae3869fe-c18b-4b0a-a0a5-16d3235f1b90'}. (HTTP 404)
    [action_ex_id=b2e8df2b-5ff5-4c9f-8910-1cdab40388e7, idx=2]: The action raised an exception [action_ex_id=b2e8df2b-5ff5-4c9f-8910-1cdab40388e7, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '7028cf41-c766-498d-b12e-23c01ce5474d'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '7028cf41-c766-498d-b12e-23c01ce5474d'}. (HTTP 404)
    [action_ex_id=bb6f4bd7-eaab-4fb4-a6e5-2c1f35bccf9c, idx=3]: The action raised an exception [action_ex_id=bb6f4bd7-eaab-4fb4-a6e5-2c1f35bccf9c, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '4c5f0118-9160-4bce-a6c2-1d7bd25718bb'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '4c5f0118-9160-4bce-a6c2-1d7bd25718bb'}. (HTTP 404)
    [action_ex_id=c46d8c62-717a-47f6-b7ac-605e6a5e4290, idx=4]: The action raised an exception [action_ex_id=c46d8c62-717a-47f6-b7ac-605e6a5e4290, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '4213a1ab-1ac0-4101-8e8e-cb5e03cf46a7'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '4213a1ab-1ac0-4101-8e8e-cb5e03cf46a7'}. (HTTP 404)
    [action_ex_id=df8f99d0-f0f1-47d2-b945-fc5bf8694daa, idx=5]: The action raised an exception [action_ex_id=df8f99d0-f0f1-47d2-b945-fc5bf8694daa, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '7508d2bf-d8cd-4291-a7d3-9c3c3c30ba63'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '7508d2bf-d8cd-4291-a7d3-9c3c3c30ba63'}. (HTTP 404)
    [action_ex_id=e7b2be84-80d9-466f-9497-384a9c241a27, idx=6]: The action raised an exception [action_ex_id=e7b2be84-80d9-466f-9497-384a9c241a27, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '57266938-ffe5-45de-852e-1105fa48c0d6'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '57266938-ffe5-45de-852e-1105fa48c0d6'}. (HTTP 404)
] (execution_id=4630877e-ddf9-40ca-9516-70d0c9aa46b5)
2019-05-17 18:38:19.104 7 INFO workflow_trace [req-43f9a144-3225-4e1f-989d-a3bdf07d43f7 c2fcadaffb9740dd9e01976190dcf06c ad6ea4a2df754e81a58ad41941ea7064 - default default] Task 'cell_v2_discover_hosts' [ERROR -> DELAYED, delay = 30 sec] (execution_id=4630877e-ddf9-40ca-9516-70d0c9aa46b5 task_id=72708317-2e06-47dd-8ba1-71d0fe9879a3)

Comment 2 David Hill 2019-05-17 22:43:14 UTC
And in mistral execution-list :

| 1f1bc248-52c6-4274-a5a7-6a39fae7557e | ae973070-fead-47e1-b81a-3a517480e4df | tripleo.baremetal.v1.set_node_state                             |                    | sub-workflow execution                                                                                                                                                                                                            | 1c7500b2-5d9b-4015-ac9e-282d4f90595f | c263b730-d809-414c-852d-b5e37f211237 | SUCCESS | None                         | 2019-05-17 22:30:46 | 2019-05-17 22:32:47 |
| 5f85f374-e1c2-4e28-99a1-98613fabe271 | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:32:52 | 2019-05-17 22:32:57 |
| e1059572-2454-49d9-acb8-4e548b71066a | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:33:27 | 2019-05-17 22:33:32 |
| 3fc8f1e2-df1d-48f6-9095-e8d573723d2d | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:34:02 | 2019-05-17 22:34:08 |
| 261094fc-17ef-4d56-a0c3-f6a4f1d13683 | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:34:38 | 2019-05-17 22:34:44 |
| 09c58a59-e2f3-4995-8d24-6a4423eaa2bf | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:35:14 | 2019-05-17 22:35:20 |
| 19838b66-bedb-4748-a757-e49deaa12d2a | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:35:50 | 2019-05-17 22:35:55 |
| 78b956cc-9cc3-4b33-9e1d-05a8e159e087 | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:36:25 | 2019-05-17 22:36:30 |
| 756e89d6-6d26-419a-99bd-e18fd5784dd2 | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:37:00 | 2019-05-17 22:37:06 |
| 3ba91826-fbc9-4ddc-889e-02cddf647050 | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:37:36 | 2019-05-17 22:37:42 |
| 7e1c07c9-eefd-4cf6-80f8-229120ca9a8e | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:38:12 | 2019-05-17 22:38:19 |
| 1aa11e79-bd97-4696-acb3-7fee6b3b9640 | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:38:49 | 2019-05-17 22:38:54 |
| 258980d6-f42d-4619-a8ed-ff5512a1be33 | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:39:24 | 2019-05-17 22:39:29 |
| 293378c9-cd76-4304-a726-070f83a07b41 | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:39:59 | 2019-05-17 22:40:05 |
| 2c9efbfb-3399-4e53-9c7e-ecebb6027c80 | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:40:35 | 2019-05-17 22:40:41 |
| af350574-8338-43eb-9dd8-54d4df9c226b | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:41:11 | 2019-05-17 22:41:17 |
| 1ef3c254-527f-4bcd-8ab5-a48030ed192a | 85733831-9250-475d-b5f5-486abb8fa61a | tripleo.baremetal.v1.cellv2_discovery                           |                    | sub-workflow execution                                                                                                                                                                                                            | 72708317-2e06-47dd-8ba1-71d0fe9879a3 | c263b730-d809-414c-852d-b5e37f211237 | ERROR   | Failure caused by error i... | 2019-05-17 22:41:47 | 2019-05-17 22:41:52 |
+--------------------------------------+--------------------------------------+-----------------------------------------------------------------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+--------------------------------------+---------+------------------------------+---------------------+---------------------+

Is this related ?  I'm generating a sosreport at this time and will attach it as soon as it's completed.

Comment 3 David Hill 2019-05-17 22:50:11 UTC
Created attachment 1570406 [details]
sosreport-undercloud-0-rhosp15-2019-05-17-xzsfrcd.tar.xz.0 1/3

Comment 4 David Hill 2019-05-17 22:50:58 UTC
Created attachment 1570407 [details]
sosreport-undercloud-0-rhosp15-2019-05-17-xzsfrcd.tar.xz.1 2/3

Comment 5 David Hill 2019-05-17 22:51:20 UTC
Created attachment 1570408 [details]
sosreport-undercloud-0-rhosp15-2019-05-17-xzsfrcd.tar.xz.2 3/3

Comment 6 David Hill 2019-05-17 23:01:25 UTC
If you look at /var/log/containers/nova/nova-compute.log, we see those errors :

2019-05-17 18:00:22.079 7 ERROR nova.virt.ironic.driver [req-c8e5ea51-1c47-4826-b0fd-f5d6ab2c8053 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: 'StrictVersion' object has n
o attribute 'version'
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task [req-c8e5ea51-1c47-4826-b0fd-f5d6ab2c8053 - - - - -] Error during ComputeManager._cleanup_running_deleted_instances: nova.exception.VirtDriverNotReady: Virt driver is not ready.
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task Traceback (most recent call last):
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 617, in _get_node_list
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     node_list = self.ironicclient.call("node.list", **kwargs)
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/nova/virt/ironic/client_wrapper.py", line 186, in call
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     return self._multi_getattr(client, method)(*args, **kwargs)
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/ironicclient/v1/node.py", line 163, in list
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     os_ironic_api_version=os_ironic_api_version)
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/ironicclient/common/base.py", line 162, in _list_pagination
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     resp, body = self.api.json_request('GET', url, **kwargs)
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/ironicclient/common/http.py", line 679, in json_request
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     resp = self._http_request(url, method, **kwargs)
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/ironicclient/common/http.py", line 287, in wrapper
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     return func(self, url, method, **kwargs)
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/ironicclient/common/http.py", line 632, in _http_request
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     self.negotiate_version(self.session, None)
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/ironicclient/common/http.py", line 161, in negotiate_version
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     if StrictVersion(max_ver) > StrictVersion(LATEST_VERSION):
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib64/python3.6/distutils/version.py", line 64, in __gt__
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     c = self._cmp(other)
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib64/python3.6/distutils/version.py", line 170, in _cmp
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     if self.version != other.version:
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task AttributeError: 'StrictVersion' object has no attribute 'version'
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task 
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task During handling of the above exception, another exception occurred:
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task 
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task Traceback (most recent call last):
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/oslo_service/periodic_task.py", line 222, in run_periodic_tasks
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     task(self, context)
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 8101, in _cleanup_running_deleted_instances
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     for instance in self._running_deleted_instances(context):
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 8155, in _running_deleted_instances
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     instances = self._get_instances_on_driver(context, filters)
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 599, in _get_instances_on_driver
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     driver_uuids = self.driver.list_instance_uuids()
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 654, in list_instance_uuids
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     fields=['instance_uuid'], limit=0)
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task   File "/usr/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 625, in _get_node_list
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task     raise exception.VirtDriverNotReady()
2019-05-17 18:00:22.079 7 ERROR oslo_service.periodic_task nova.exception.VirtDriverNotReady: Virt driver is not ready.

Comment 7 David Hill 2019-05-17 23:07:31 UTC
2019-05-17 19:06:04.424 7 ERROR nova.compute.manager [req-304d9649-92c8-40f3-9356-cb4c77aded36 - - - - -] No compute node record for host undercloud-0-rhosp15.localdomain: nova.exception_Remote.ComputeHostNotFound_Remote: Compute host undercloud-0-rhosp15.localdomain could not be found.

Comment 8 Alex Schultz 2019-05-18 03:27:29 UTC

*** This bug has been marked as a duplicate of bug 1711366 ***