Description of problem: We used AI ocm-2.3-20210603 to install ocp 4.8.0-fc.7, and found 16/1081 clusters are showing clusterdeployment installed=true, but failed to connect to hub (RHACM). After we did a scan of the clusters, all the SNO nodes' `/etc/resolve.conf` files are empty, which we expect they should include the DNS we setup in the nmstateconfig CR. The cluster can be installed which means during bootstrapping, the cluster has correct DNS settings, otherwise it won't be able to download the rootfs. Something happened after bootstrap, and may be after AI installed the cluster. Version-Release number of selected component (if applicable): How reproducible: 16/1081 => about 1.5% Steps to Reproduce: 1. Install AI with ACM 2. Zero touch provisioning SNOs with DNS settings in nmstateconfig 3. See some of the clusters are not in ready status in ACM, and `/etc/resolve.conf` is empty. Actual results: `/etc/resolve.conf` is empty. Expected results: `/etc/resolve.conf` should have nameservers in nmstateconfig Additional info:
It can be related to the forcedns sed command not working: ``` # cat /etc/NetworkManager/dispatcher.d/forcedns export IP="198.18.8.147" if [ "$2" = "dhcp4-change" ] || [ "$2" = "dhcp6-change" ] || [ "$2" = "up" ] || [ "$2" = "connectivity-change" ]; then if ! grep -q "$IP" /etc/resolv.conf; then sed -i "s/sno00030.rdu2.scalelab.redhat.com//" /etc/resolv.conf sed -i "s/search /search sno00030.rdu2.scalelab.redhat.com /" /etc/resolv.conf sed -i "0,/nameserver/s/nameserver/nameserver $IP\nnameserver/" /etc/resolv.conf fi fi # journalctl | grep -C10 "/etc/NetworkManager/dispatcher.d/forcedns" ... Jun 04 23:29:54 sno00030 kernel: IPv6: ADDRCONF(NETDEV_UP): enp1s0: link is not ready Jun 04 23:29:54 sno00030 dbus-daemon[1360]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' Jun 04 23:29:54 sno00030 systemd[1]: Started Network Manager Script Dispatcher Service. Jun 04 23:29:54 sno00030 kernel: IPv6: ADDRCONF(NETDEV_UP): enp1s0: link is not ready Jun 04 23:29:54 sno00030 nm-dispatcher[1605]: Error: Device '' not found. Jun 04 23:29:54 sno00030 nm-dispatcher[1605]: Error: Device '' not found. Jun 04 23:29:54 sno00030 nm-dispatcher[1605]: grep: /etc/resolv.conf: No such file or directory Jun 04 23:29:54 sno00030 nm-dispatcher[1605]: sed: can't read /etc/resolv.conf: No such file or directory Jun 04 23:29:54 sno00030 nm-dispatcher[1605]: sed: can't read /etc/resolv.conf: No such file or directory Jun 04 23:29:54 sno00030 nm-dispatcher[1605]: sed: can't read /etc/resolv.conf: No such file or directory Jun 04 23:29:54 sno00030 nm-dispatcher[1605]: req:2 'connectivity-change', "/etc/NetworkManager/dispatcher.d/forcedns": complete: failed with Script '/etc/NetworkManager/dispatcher.d/forcedns' exited wit$ error status 2. Jun 04 23:29:54 sno00030 NetworkManager[1593]: <warn> [1622849394.1499] dispatcher: (2) /etc/NetworkManager/dispatcher.d/forcedns failed (failed): Script '/etc/NetworkManager/dispatcher.d/forcedns' exit$d with error status 2. ```
*** Bug 1969574 has been marked as a duplicate of this bug. ***
Verified with latest 1k cluster provisioning test, not seeing this issue any more.
*** Bug 2042378 has been marked as a duplicate of this bug. ***