Bug 2251309
| Summary: | Introspection fails for newly added nodes at scale | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Asma Syed Hameed <asyedham> |
| Component: | openstack-ironic | Assignee: | OSP Team <rhos-maint> |
| Status: | CLOSED UPSTREAM | QA Contact: | |
| Severity: | unspecified | Docs Contact: | |
| Priority: | medium | ||
| Version: | 17.1 (Wallaby) | CC: | hjensas, jkreger, sbaker |
| Target Milestone: | --- | Keywords: | Documentation |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-03-05 01:08:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Description of problem: We currently have 1K nodes and while performing bulk introspection for the newly added 100 nodes it fails with he last error is ironic-inspector inspection failed: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition)", "2023-11-24 07:11:38,725 ERROR os_tripleo_baremetal_node_introspection: Introspection of node 59a114f6-c832-4250-b1ae-ba0aee2d584f failed on attempt 2: Node 59a114f6-c832-4250-b1ae-ba0aee2d584f reached failure state \"inspect failed\"; $he last error is ironic-inspector inspection failed: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition)", "" ], } 2023-11-24 07:11:38.965733 | ac1f6b0f-8e5c-0b9d-7f58-000000000011 | TIMING | Introspection log | localhost | 0:09:41.158224 | 0.08s 2023-11-24 07:11:38.970230 | ac1f6b0f-8e5c-0b9d-7f58-000000000012 | TASK | Node introspection summary 2023-11-24 07:11:39.036232 | ac1f6b0f-8e5c-0b9d-7f58-000000000012 | OK | Node introspection summary | localhost | result={ "changed": false, "msg": "Passed: []\nFailed: [b25fd891-5be1-4fab-be25-f37813429eec]\nNo nodes passed introspection\n" } 2023-11-24 07:11:39.037125 | ac1f6b0f-8e5c-0b9d-7f58-000000000012 | TIMING | Node introspection summary | localhost | 0:09:41.229620 | 0.07s 2023-11-24 07:11:39.040642 | ac1f6b0f-8e5c-0b9d-7f58-000000000013 | TASK | Node introspection failed and no results are provided 2023-11-24 07:11:39.096518 | ac1f6b0f-8e5c-0b9d-7f58-000000000013 | FATAL | Node introspection failed and no results are provided | localhost | error={"changed": false, "msg": "Nodes failed introspection"} 2023-11-24 07:11:39.097384 | ac1f6b0f-8e5c-0b9d-7f58-000000000013 | TIMING | Node introspection failed and no results are provided | localhost | 0:09:41.289876 | 0.06s NO MORE HOSTS LEFT ************************************************************* Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20230802.n.1 How reproducible: 100% Steps to Reproduce: 1. Deploy 850 nodes 2. Register 100 new nodes 3. Perform bulk introspection Actual results: Introspection failed Expected results: Introspection successful Additional info: From the logs it seems pxefilter driver not able to initialize for the DNSmasq