Description of problem:

This is a preemptive bug for an upstream issue we've just recently started to see in Centos 9-stream, which may end up impacting RHEL9 based builds at some point in the near future given the cycle of package updates.

It appears that an issue is has appeared in NetworkManager where the dhcp client in the ramdisk, specifically dhclient launched by NetworkManager, is not settling, and retries every sixty seconds while the ramdisk runs. Eventually, the dnsmasq dhcp server says "You already have a lease, I'm going to ignore you until your lease runs down." Unfortunately, ramdisk connectivity eventually breaks.

On a positive side, we see this in CI which is generally slower than real hardware, but my worry right now is this suddenly breaks us.

The path we've determined upstream is to set the dhcp client setting for NetworkManager to "internal", which we've been unable to reproduce this issue as of yet using that as the client default. Unfortunately, this requires a diskimage-builder change to take effect.

Upstream we're able to reproduce this with any ramdisk that is online for greater than about 250 seconds. After which the network configuration disappears from within the ramdisk, and we're no longer able to ping the ramdisk which continues to run.

Steps to Reproduce:

The only way to reproduce this is to have an artificially long deploy, which might mean a BMaaS deployment with a larger than normal image. That being said, if this appears in the product, we should likely just rely upon a code check if we've not figured out a better way to test this, given we basically race physical system performance versus a clock.

Actual results:

Deployments with a full ramdisk fail upstream.


Expected results:

Deployments succeed.

Additional info:

https://storyboard.openstack.org/#!/story/2010109