Bug 1647671

Summary: Ansible Networking - introspection fails for BM guests on Overcloud
Product: Red Hat OpenStack Reporter: Arkady Shtempler <ashtempl>
Component: python-networking-ansibleAssignee: Dan Radez <dradez>
Status: CLOSED NOTABUG QA Contact: Arkady Shtempler <ashtempl>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 14.0 (Rocky)CC: bfournie, derekh, jlibosva, michapma, sasha
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-04 15:45:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Ironic Inspector logs none

Description Arkady Shtempler 2018-11-08 05:42:54 UTC 
Setup:
Ansible Neworking QE setup with inspector service enabled on Overcloud:

Scenario:
Import BM guests on Overcloud
. overcloudrc
openstack baremetal create bm_guests_env.yaml

Start introspection for BM guest
openstack overcloud node introspect --all-manageable --provide

Check introspection status
(openstack) baremetal introspection status 755e51ce-daac-4efc-84ce-13a010d59182


Expected result:
Introspection is successfully completed

Actual Result:
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| error       | Introspection timeout                |
| finished    | True                                 |
| finished_at | 2018-11-07T12:48:52                  |
| started_at  | 2018-11-07T11:48:40                  |
| state       | error                                |
| uuid        | 755e51ce-daac-4efc-84ce-13a010d59182 |
+-------------+--------------------------------------+
timeout Error is logged on ironic-inspector 
2018-11-07 12:48:52.976 1 ERROR ironic_inspector.node_cache [-] Introspection for nodes [u'755e51ce-daac-4efc-84ce-13a010d59182'] has timed out
Comment 1 Arkady Shtempler 2018-11-11 17:35:08 UTC 
Created attachment 1504427 [details]
Ironic Inspector logs
Comment 2 Arkady Shtempler 2018-11-11 17:44:13 UTC 
There is a single line error message about the introspection failure in log:

Log path: /var/log/containers/ironic-inspector/ironic-inspector.log 
Line: 675
2018-11-11 11:06:32.228 1 ERROR ironic_inspector.node_cache [-] Introspection for nodes [u'b610e51b-ded6-4cac-b972-5c210421add8'] has timed out
time out ERROR 


BTW: the ID in log is not the same as in bug description, it was changed on reproducing.
Comment 4 Bob Fournier 2018-12-03 22:04:50 UTC 
Isn't this the same as https://bugzilla.redhat.com/show_bug.cgi?id=1654648?

As there is workaround (from 1654648):
1.  When deploying new BMaaS hardware, switch ports for the new nodes must be
    manually configured to place them in the provisioning VLAN.

2.  When re-introspecting an existing node (after replacing a faulty hardware
    component, for example) the switch port must be manually recofigured to
    connect the node to the provisioning VLAN.

and the fact that 1654648 is an RFE since there is significant effort to fix and test a  change to this behavior, I don't think this should be a blocker.
Comment 5 Derek Higgins 2018-12-04 15:45:43 UTC 
(In reply to Arkady Shtempler from comment #0)
> Setup:
> Ansible Neworking QE setup with inspector service enabled on Overcloud:
> 
> Scenario:
> Import BM guests on Overcloud
> . overcloudrc
> openstack baremetal create bm_guests_env.yaml
> 
> Start introspection for BM guest
> openstack overcloud node introspect --all-manageable --provide

here is the problem, any command beginning in "openstack overcloud ..." should be run against the undercloud (for deploying the overcloud), the overcloud doesn't have the workflows required to run this command. The command needed would be

$ openstack baremetal node inspect ironic-1