Description of problem: Attempting to introspect and deploy w/ OSP10 on multiple supermicro nodes. iPXE load process would consistently fail to complete on 1-5 nodes, but never the same ones, on each attempt. We later determined that the upstream switch was not correctly using its switching table and was flooding some unicast traffic unrelated to the target NICs, as well as the usual broadcast traffic. Apparently this irrelevant traffic fills up an ipxe buffer somewhere. As soon as we removed the irrelevant flooded unicast traffic from the network, ipxe works fine from then on. Console screenshots will be attached. Version-Release number of selected component (if applicable): OSP10 and Director How reproducible: Very reproducible, just flood an interface with irrelevant (towards MAC addresses that are not the NICs) traffic and try to introspect or deploy a node. PXE boots, chainloads iPXE, then iPXE starts to work and fails in various ways Steps to Reproduce: 1. add nodes to director 2. reproduce unicast traffic flooding on the switchport facing the pxe boot nic (one way to do this would be to just set it as a span port similarly used for packet captures) 3. attempt to introspect the node Actual results: ipxe fails to load the rest of the ramdisk/agent/kernel Expected results: ipxe should complete, ignoring all irrelevant unicast traffic, and continue loading the OS Additional info:
Created attachment 1458484 [details] node console screenshot of ipxe failing to load the next image file
Created attachment 1458485 [details] node console screenshot of ipxe failing to load the next image file (second failure type)
Hi! What could we do to improve the situation, considering that it's way out of control of ironic itself. Maybe you could publish a KB article explaining the issue? I'm really not sure what we can do in the code to fix it.
I agree, this is not an ironic issue, this is an issue with the ipxe image. The ipxe network stack should operate like a standard network stack and ignore all unicast traffic received on the nic that is not destined for its mac address. Receiving flooded unicast traffic (just like being on a hub vs a switch) should not break functionality of ipxe. This isn't a broadcast storm instance, this is just normal ethernet unknown unicast flooding that occurs on hubs or when a switch hasn't identified where a mac address is.
Are we able to fix an issue in ipxe?
We get the iPXE image from RHEL, redirecting to RHEL folks for their input.
can you mirror a port on the peer switch to a failing ipxe node and capture the traffic during a failure please? To be perfectly honest, the problem description doesn't make any sense to me. If the incorrectly flooded traffic is unicast, then it should be squashed by the NIC itself via the mac filter table which should only let in traffic from its own mac, or the broadcast mac (unless you have a very very old NIC that has no filtering capabilities). Getting a network trace (in pcap format), would help me root cause this issue and develop an appropriate fix
I'll do my best to get that capture, but it can be done by doing a pxe>ipxe boot via a switch port that is configured as a span port with a lot of extra unicast traffic You are exactly correct, the nic -should- be filtering that out, but for some reason it doesn't seem to be when it's running the ipxe image. PXE works fine to chainload ipxe, but once ipxe is up that's when it fails Perhaps ipxe is doing something to the NIC to make it process and forward all traffic to the ipxe network stack? (promiscuous mode?)
Its certainly possible, but I don't see anything in any drivers that explicitly force promisc mode. theres some higher layer cases where that appears to happen, but nothing that you would hit in normal operation. Any luck on the capture? also, what is the pci id tuple of the nic in question here?
This has gone unanswered for over two months now. Closing for lack of response