Bug 1541942
| Summary: | Introspection failed to load collector set numa-topology | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Raoul Scarazzini <rscarazz> |
| Component: | openstack-ironic-python-agent | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED DUPLICATE | QA Contact: | mlammon |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 10.0 (Newton) | CC: | agurenko, bfournie, dbecker, mburns, morazi, rhel-osp-director-maint, rscarazz, skatlapa, slinaber, ssbarnea |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-02-07 15:15:58 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
One important additional note, all the nodes are affected indistinctly: +--------------------------------------+----------+-----------------------+ | Node UUID | Finished | Error | +--------------------------------------+----------+-----------------------+ | a6aef726-be90-48e8-a733-75df0a2d062e | True | Introspection timeout | | bad3bed0-c6a6-49cf-aa9e-254b26f7edeb | True | Introspection timeout | | cc1c50f2-470f-46f4-8494-0357fd8fe3ad | True | Introspection timeout | | 4988ec47-a879-4757-8a55-43a8c46d56e2 | True | Introspection timeout | | 76613f5d-c73a-4f01-830f-b5d56ef4b41f | True | Introspection timeout | | 07df2b67-1213-49e8-a5c0-46e8432f1e31 | True | Introspection timeout | | 497b298b-c76f-4c7b-afa4-c387ca334ed3 | True | Introspection timeout | | 566ab447-41de-4106-9137-28272bf776ea | True | Introspection timeout | | a1fdf5da-943f-4b98-8007-af8510f234b4 | True | Introspection timeout | | 1bb3283d-fad1-43af-89d8-37a3cbd97b59 | True | Introspection timeout | +--------------------------------------+----------+-----------------------+ I've uploaded here [1] the sosreport coming from the undercloud. [1] http://file.rdu.redhat.com/~rscarazz/BZ1541942/sosreport-undercloud-20180205045157.tar.xz There are 3 downstream patches associated with the numa_topology backport into OSP-10 ( https://bugzilla.redhat.com/show_bug.cgi?id=1501875) - https://code.engineering.redhat.com/gerrit/#/c/125939/ - https://code.engineering.redhat.com/gerrit/#/c/124833/ - https://code.engineering.redhat.com/gerrit/#/c/121837/ The first one is in instack-undercloud, the next 2 are in ironic-python-agent. It looks like the first one has been brought into the osp-10 instack-undercloud image but the other 2 weren't brought into IPA, which is what is causing the introspection problem. numa_topology is set as one of the collectors by instack-undercloud, but IPA doesn't have the patch to handle it. [ 30.382587] iconic-python-agent[1124]: 2B18-02-05 03:51:22.585 1124 CRITICAL ironic-python-agent [-] InspectionError: failed to load collector set(['numa-topology']) As a temporary workaround, you can edit /httpboot/inspector.ipxe and remove the "numa-topology" field and restart ironic inspector. This will allow IPA to boot and introspection to succeed without the numa-topology data. [stack@undercloud-0 ~]$ sudo cat /httpboot/inspector.ipxe #!ipxe :retry_boot imgfree kernel --timeout 60000 http://192.168.24.1:8088/agent.kernel ipa-inspection-callback-url=http://192.168.24.1:5050/v1/continue ipa-inspection-collectors=default,extra-hardware,numa-topology,logs systemd.journald.forward_to_console=yes BOOTIF=${mac} ipa-debug=1 ipa-inspection-dhcp-all-interfaces=1 ipa-collect-lldp=1 initrd=agent.ramdisk || goto retry_boot initrd --timeout 60000 http://192.168.24.1:8088/agent.ramdisk || goto retry_boot boot Just as a confirmation the workaround worked as expected. Marking this as a duplicate. The fix has already been backported to OSP-10 but ironic-python-agent had not been build for the puddle. This has been fixed. *** This bug has been marked as a duplicate of bug 1501875 *** |
Description of problem: With the latest Puddle introspection is constantly failing. The message appearing on the nodes console is this one: [ 30.366464] ironic-python-agent[1124]: }, proceeding with lookup [ 30.382587] iconic-python-agent[1124]: 2B18-02-05 03:51:22.585 1124 CRITICAL ironic-python-agent [-] InspectionError: failed to load collector set(['numa-topology']) [ 30.414556] ironic-python-agent[1124]: 20l8-02-05 03:5l:22.585 1124 ERROR ironic-python-agent Traceback (most recent call last): [ 30.446234] systemd[1]: Unit openstack-iconic-python-agent.secvice entered Failed state. [ 30.462394] iconic-python-agent[1124]: 2018-02-05 03:51:22.585 1124 ERROR iconic-python-agent File "/usc/bin/ironic-python-agent", line 10, in <module> [ 30.493819] iconic-python-agentl1124]: 2018-02-05 03:51:22.585 1124 ERROR iconic-python-agent sys.exit(run()) ... ... [ 30.9764l4] ironic-python-agent[1124]: 2018-02-05 03:51:22.585 1124 ERROR ironic-python-agent File "/usr/lib/python2.7/site-packages/ironic-python-agent/inspectoc.py", line 46, in _extension_manager_err_callback [ 31.B06l56] ironic-python-agent[1124]: 2018-02-05 B3:51:22.585 1124 ERROR ironic-python-agent raise errors.InspectionError('Failed to load collector %s' % names) [ 31.036202] ironic-python-agent[1124]: 2018-02-05 03:51:22.585 1124 ERROR ironic-python-agent InspectionError: Failed to load collector set(['numa-topology']) [ 31.066217] ironic-python-agent[1124]: 2018-02-05 03:51:22.585 1124 ERROR ironic-python-agent This might be related to [1] which was closed as NOTABUG that's why I'm opening this new one. In addition this is related to OSP10 and the testing environment is always the same, the last known [2] working puddle was 2018-01-03.2. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1472295 [2] https://thirdparty.logs.rdoproject.org/jenkins-oooq-rhos-10-puddle-bmu-hac01-lab-float_nic_with_vlans-23/hac-01.ha.lab.eng.bos.redhat.com/etc/yum.repos.d/rhos-release-10.repo.txt.gz Version-Release number of selected component (if applicable): 2018-02-02.1 How reproducible: source /home/stack/stackrc openstack overcloud image upload openstack overcloud node import instackenv.json openstack overcloud node introspect --all-manageable Actual results: Introspection times out, on the nodes console the error is the one above. Expected results: Introspection succeeds. Additional info: The failed test comes from the promotion pipeline, but I've reproduced it also on a dedicated environment as well, so again it can't be something related to the hardware.