Description of problem: On a setup which has been left online for a couple of days we can notice that some of the nodes in the cluster have lost their IP addresses on the provisioning network interface: for node in $(oc get nodes --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'); do echo $node; ssh core@$node.ocp-edge.lab.eng.tlv2.redhat.com 'ip a s dev ens1f0';done openshift-master-0 2: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 48:df:37:c7:e7:50 brd ff:ff:ff:ff:ff:ff inet6 fe80::c1c6:1a61:a442:d9bf/64 scope link noprefixroute valid_lft forever preferred_lft forever openshift-master-1 2: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 48:df:37:b0:65:50 brd ff:ff:ff:ff:ff:ff inet6 fd00:1101::3/64 scope global dynamic valid_lft 10sec preferred_lft 10sec inet6 fd00:1101::d/128 scope global dynamic noprefixroute valid_lft 1980sec preferred_lft 1980sec inet6 fe80::926c:73b9:e90f:5631/64 scope link noprefixroute valid_lft forever preferred_lft forever openshift-master-2 2: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 48:df:37:b0:8a:a0 brd ff:ff:ff:ff:ff:ff inet6 fd00:1101::c/128 scope global dynamic noprefixroute valid_lft 3040sec preferred_lft 3040sec inet6 fe80::effa:9769:ed1e:e7b8/64 scope link noprefixroute valid_lft forever preferred_lft forever openshift-worker-0 2: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 48:df:37:c7:f7:b0 brd ff:ff:ff:ff:ff:ff inet6 fe80::bb34:bea5:e05a:28ae/64 scope link noprefixroute valid_lft forever preferred_lft forever openshift-worker-1 2: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 48:df:37:b0:79:30 brd ff:ff:ff:ff:ff:ff inet6 fd00:1101::e/128 scope global dynamic noprefixroute valid_lft 2753sec preferred_lft 2753sec inet6 fe80::dbfa:f33c:83d1:5793/64 scope link noprefixroute valid_lft forever preferred_lft forever openshift-worker-2 2: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 48:df:37:c8:04:a0 brd ff:ff:ff:ff:ff:ff inet6 fd00:1101::47/128 scope global dynamic noprefixroute valid_lft 2663sec preferred_lft 2663sec inet6 fe80::2ad1:8ce7:bc4f:690a/64 scope link noprefixroute valid_lft forever preferred_lft forever openshift-worker-3 2: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 48:df:37:c8:00:b0 brd ff:ff:ff:ff:ff:ff inet6 fe80::2d29:be18:de1f:73ad/64 scope link noprefixroute valid_lft forever preferred_lft forever Looking at the failing nodes we can see the error below: Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: PRC: Address fd00:1101::1c depreferred. Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: XMT: Rebind on ens1f0, interval 00ms. Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: Impossible condition at dhc6.c:273. Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: This version of ISC DHCP is based on the release available Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: on ftp.isc.org. Features have been added and other changes Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: have been made to the base software release in order to make Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: it work better with this distribution. Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: Please report issues with this software via: Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: https://bugzilla.redhat.com/ Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: Jun 12 02:59:07 openshift-worker-0 dhclient[2350]: exiting. Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: PRC: Address fd00:1101::34 depreferred. Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: XMT: Rebind on ens1f0, interval 00ms. Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: Impossible condition at dhc6.c:273. Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: This version of ISC DHCP is based on the release available Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: on ftp.isc.org. Features have been added and other changes Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: have been made to the base software release in order to make Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: it work better with this distribution. Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: Please report issues with this software via: Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: https://bugzilla.redhat.com/ Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: Jun 12 03:02:08 openshift-worker-3 dhclient[2256]: exiting. Version-Release number of selected component (if applicable): 4.4.7 How reproducible: 100% Steps to Reproduce: 1. Deploy bare metal IPI cluster with IPv6 provisioning netowkr 2. Leave the environment online for some time 3. Check the provisioning network interface on the cluster vnodes Actual results: Some of the nodes do not have an IP address on the provisioning network interface Expected results: The nodes keep their IP address on the provisioning network interface. Additional info: Attaching must-gather.
So, during initial deployment all masters get a DHCP lease from the bootstrap VM, but subsequently the static-ip-manager container configures the provisioning IP statically, but only on the master where the metal3 pod is scheduled (in this case fd00:1101::3) What we're seeing here I think is just the lease expiry on the other masters, which is expected because NetworkManager brought up the nics, then the DHCP server (on the bootstrap VM) went away. Arguably they should get a new lease from the dnsmasq running in the metal3 pod, I'm not sure why that doesn't happen rather than the failed rebind in the errors. However it raises the question of whether we want to support any use of the provisioning nic on nodes where the metal3 pod is not scheduled - IMHO we don't, so perhaps we can either consider this not a bug or explicitly disable the nics on nodes not running the metal3 pod? There is one potential corner-case, which is related to the order of running static-ip-set here: https://github.com/openshift/machine-api-operator/blob/master/pkg/operator/baremetal_pod.go#L209 We run the machine-os-downloader container, then after we run static-ip-set - this works fine provided the image download happens via the baremetal network. However dev-scripts we actually configure the local mirror such that it downloads via the provisioning network - in that case our current testing works only because of these leases from the bootstrap VM, and if they expire then the metal3 pod gets rescheduled, the download will fail. One solution would be to move the static-ip-set before the machine-os-downloader, but if we do that we may hit this hard-coded lft timeout and image download could still fail. https://github.com/openshift/ironic-static-ip-manager/blob/master/set-static-ip#L20 I think it's reasonable to expect image download in all real non-virt environments to happen via the baremetal/external network (regardless of whether we're downloading from the internet or a local mirror), but we may want to add some validation to prevent the corner case above causing potential problems?
I've deployed a 4.4.7 cluster and let it run for two weeks. Tested with IPv6, with DHCP range of fd00:1101::a - fd00:1101::64. During that time I checked the lease renewal logs on a daily basis and never seen an IP address change. Capturing the IP addresses via: for node in $(oc get nodes --template '{{range .items}}{{index .status.addresses 0 "address"}}{{"\n"}}{{end}}'); do ssh core@$node 'journalctl -u NetworkManager | grep "dhcp6 (enp2s0): address"';done > lease_times.txt Had the same values for the entire lifetime of the cluster: master-0: fd2e:6f44:5dd8:c956::14 master-1: fd2e:6f44:5dd8:c956::15 master-2: fd2e:6f44:5dd8:c956::16 worker-1: fd2e:6f44:5dd8:c956::18 As an interesting side note, average time before lease renewal: master-0: 9 sec. master-1: 23 sec. master-2: 24 sec. worker-1: 25 sec.