Bug 1472295 - [mix-version] introspect failed from osp12 undercloud with osp 11 image with "Failed to load collector set(['numa-topology'])"
[mix-version] introspect failed from osp12 undercloud with osp 11 image with ...
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Angus Thomas
Amit Ugol
Depends On:
  Show dependency treegraph
Reported: 2017-07-18 07:55 EDT by Raviv Bar-Tal
Modified: 2017-08-16 08:31 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-08-16 08:31:55 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
sos report (15.84 MB, application/x-xz)
2017-07-18 07:55 EDT, Raviv Bar-Tal
no flags Details
ironic.conf (119.96 KB, text/plain)
2017-08-15 08:42 EDT, Raviv Bar-Tal
no flags Details
ironic-conductor.log (1.85 MB, text/plain)
2017-08-15 08:44 EDT, Raviv Bar-Tal
no flags Details
ironic-inspector.log (7.17 MB, text/plain)
2017-08-15 08:44 EDT, Raviv Bar-Tal
no flags Details
screenshot (43.71 KB, image/png)
2017-08-15 08:45 EDT, Raviv Bar-Tal
no flags Details

  None (edit)
Description Raviv Bar-Tal 2017-07-18 07:55:32 EDT
Created attachment 1300443 [details]
sos report

Description of problem:
Introspection from ops 12 undercloud with osp11 (latest) images on fails.

Version-Release number of selected component (if applicable):
undercloud  osp12 

How reproducible:

Steps to Reproduce:
1. install osp12 undercloud
2. upload osp11 images to glance and /httpboot
3. set agent.kernel and agent.ramdisk for the nodes
4. run introspection,

Actual results:
introspection fails with time out

Expected results:
introspection pass 

Additional info:
The hypervisor os is RHEL7.4
Comment 1 Dmitry Tantsur 2017-08-02 11:47:17 EDT
Could you please check what exactly happens on the node? Is it e.g. PXE timeout?  https://docs.openstack.org/ironic-inspector/latest/user/troubleshooting.html#introspection-times-out may help
Comment 3 Bob Fournier 2017-08-14 14:45:03 EDT
Ravi - any update on this one?  Thanks.
Comment 4 Raviv Bar-Tal 2017-08-15 08:42 EDT
Created attachment 1313650 [details]
Comment 5 Raviv Bar-Tal 2017-08-15 08:44 EDT
Created attachment 1313651 [details]
Comment 6 Raviv Bar-Tal 2017-08-15 08:44 EDT
Created attachment 1313652 [details]
Comment 7 Raviv Bar-Tal 2017-08-15 08:45 EDT
Created attachment 1313653 [details]
Comment 8 Raviv Bar-Tal 2017-08-15 08:49:54 EDT
Hi Bob, Dmitry
This BZ is easy to reproduce and it is still happens.
The ramdisk log are not collected to my system, please check attached ironic.conf and advice about deploy_logs_* configuration

also attached:
screen shot form inspected node

Comment 9 Bob Fournier 2017-08-15 14:37:03 EDT
Thanks Ravi.

Looks like this error in IPA:
2017-08-15 07:28:11.837 27198 DEBUG ironic_inspector.main [-] [unidentified node] Received data from the ramdisk: {u'error': u"The following errors were encountered:\n* Failed to load collector set(['numa-topology'])"} api_continue /usr/lib/python2.7/site-packages/ironic_inspector/main.py:193
2017-08-15 07:28:11.837 27198 DEBUG ironic_inspector.process [-] [unidentified node] Running pre-processing hook ramdisk_error _run_pre_hooks /usr/lib/python2.7/site-packages/ironic_inspector/process.py:117
2017-08-15 07:28:11.837 27198 ERROR ironic_inspector.utils [-] [unidentified node] Ramdisk reported error: The following errors were encountered:
* Failed to load collector set(['numa-topology'])
2017-08-15 07:28:11.838 27198 ERROR ironic_inspector.process [-] [unidentified node] Hook ramdisk_error failed, delaying error report until node look up: Ramdisk reported error: The following errors were encountered:

The full stack trace can be seen in the screenshot.
Comment 10 Bob Fournier 2017-08-15 15:12:55 EDT
So what appears to be happening is this instack-undercloud change [1] adds numa-topology as a default collector in OSP-12.  This is run in the OSP-12 undercloud which installs OSP-11 images.  So the default is there in the undercloud but the installed OSP-11 IPA does not support numa-topology, i.e. doesn't have [2], so the collector can't be found.

[1] https://review.openstack.org/#/c/474120/
[2] https://review.openstack.org/#/c/424729/
Comment 11 Bob Fournier 2017-08-16 08:31:55 EDT
Closing this as this isn't a supported configuration.  The problem is with running  older IPA images with an updated undercloud.  Although mixed undercloud/overcloud configurations are supported, the IPA image is really part of the undercloud, and as such, an older IPA image will not work for introspection with a newer undercloud.

Note You need to log in before you can comment on or make changes to this bug.