Bug 2251309 - Introspection fails for newly added nodes at scale
Summary: Introspection fails for newly added nodes at scale
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ironic
Version: 17.1 (Wallaby)
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: ---
: ---
Assignee: OSP Team
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-11-24 07:27 UTC by Asma Syed Hameed
Modified: 2024-03-05 01:08 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-03-05 01:08:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-30549 0 None None None 2023-11-24 07:28:02 UTC

Description Asma Syed Hameed 2023-11-24 07:27:47 UTC
Description of problem:
We currently have 1K nodes and while performing bulk introspection for the newly added 100 nodes it fails with 

he last error is ironic-inspector inspection failed: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition)",              
        "2023-11-24 07:11:38,725 ERROR os_tripleo_baremetal_node_introspection: Introspection of node 59a114f6-c832-4250-b1ae-ba0aee2d584f failed on attempt 2: Node 59a114f6-c832-4250-b1ae-ba0aee2d584f reached failure state \"inspect failed\"; $he last error is ironic-inspector inspection failed: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition)",             
        ""
    ],
}
2023-11-24 07:11:38.965733 | ac1f6b0f-8e5c-0b9d-7f58-000000000011 |     TIMING | Introspection log | localhost | 0:09:41.158224 | 0.08s                                                                                                             
2023-11-24 07:11:38.970230 | ac1f6b0f-8e5c-0b9d-7f58-000000000012 |       TASK | Node introspection summary
2023-11-24 07:11:39.036232 | ac1f6b0f-8e5c-0b9d-7f58-000000000012 |         OK | Node introspection summary | localhost | result={                                                                                                                  
    "changed": false,
    "msg": "Passed: []\nFailed: [b25fd891-5be1-4fab-be25-f37813429eec]\nNo nodes passed introspection\n"
}
2023-11-24 07:11:39.037125 | ac1f6b0f-8e5c-0b9d-7f58-000000000012 |     TIMING | Node introspection summary | localhost | 0:09:41.229620 | 0.07s                                                                                                    
2023-11-24 07:11:39.040642 | ac1f6b0f-8e5c-0b9d-7f58-000000000013 |       TASK | Node introspection failed and no results are provided                                                                                                              
2023-11-24 07:11:39.096518 | ac1f6b0f-8e5c-0b9d-7f58-000000000013 |      FATAL | Node introspection failed and no results are provided | localhost | error={"changed": false, "msg": "Nodes failed introspection"}                                  
2023-11-24 07:11:39.097384 | ac1f6b0f-8e5c-0b9d-7f58-000000000013 |     TIMING | Node introspection failed and no results are provided | localhost | 0:09:41.289876 | 0.06s                                                                         

NO MORE HOSTS LEFT *************************************************************



Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230802.n.1

How reproducible:
100%

Steps to Reproduce:
1. Deploy 850 nodes
2. Register 100 new nodes
3. Perform bulk introspection

Actual results:
Introspection failed

Expected results:
Introspection successful

Additional info:

From the logs it seems pxefilter driver not able to initialize for the DNSmasq


Note You need to log in before you can comment on or make changes to this bug.