Bug 2151504

Summary: kdump to NFS gets stuck during boot in eternal "Bad kdump network destination" loop
Product: Red Hat Enterprise Linux 9 Reporter: Martin Pitt <mpitt>
Component: kexec-toolsAssignee: Coiby <coxu>
Status: CLOSED ERRATA QA Contact: xiaoying yan <yiyan>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.2CC: bgalvani, coxu, jieli, lichliu, lrintel, ltao, rkhan, ruyang, sfaye, sukulkar, till, xiawu
Target Milestone: rcKeywords: Regression, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: CockpitTest
Fixed In Version: kexec-tools-2.0.26-2.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-07 08:51:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin Pitt 2022-12-07 11:45:56 UTC 
Description of problem: Our most recent RHEL 9.2 image refresh in Cockpit's CI shows that configuring kdump to NFS is now broken.


Version-Release number of selected component (if applicable):

kexec-tools-2.0.25-7.el9.x86_64
dracut-057-13.git20220816.el9.x86_64
NetworkManager-1.41.6-1.el9.x86_64
kernel-5.14.0-205.el9.x86_64

How reproducible: Always


Steps to Reproduce:
1. You need to have an NFS server set up. Verify that you can mount it, e.g. with

   mount -t nfs 10.111.113.2:/srv/kdump /mnt
   umount /mnt

2. Configure kdump for NFS:
   echo "nfs 10.111.113.2:/srv/kdump" >> /etc/kdump.conf

3. Restart kdump to pick up the change/rebuild initrd:
   systemctl restart kdump.service

This succeeded (unlike SSH, see bug 2151500), but the journal for kdump.service has some relevant error messages:

kdumpctl[5439]: cp: cannot stat '/run/NetworkManager/system-connections/static-52\\\:54\\\:01.nmconnection': No such file or directory
kdumpctl[5440]: sed: can't read /tmp/1977-DRACUT_KDUMP_NM/ifcfg-static-52-54-01: No such file or directory
kdumpctl[5449]: dracut-install: ERROR: installing '/tmp/1977-DRACUT_KDUMP_NM/ifcfg-static-52-54-01' to '/etc/NetworkManager/system-connections/ifcfg-static-52-54-01'

4. Provoke a kernel crash:
  echo 1 > /proc/sys/kernel/sysrq
  echo c > /proc/sysrq-trigger

Actual results: reboot fails. First there is this (virsh console only, can't login to get journal):

[    1.369764] mount[374]: mount.nfs: Network is unreachable
[FAILED] Failed to mount /kdumproot.
See 'systemctl status kdumproot.mount' for details.
[DEPEND] Dependency failed for Remote File Systems.
         Starting dracut pre-pivot and cleanup hook...
[    1.374169] systemd[1]: kdumproot.mount: Mount process exited, code=exited, status=32/n/a
[    1.375175] systemd[1]: kdumproot.mount: Failed with result 'exit-code'.
[    1.375892] systemd[1]: Failed to mount /kdumproot.
[    1.376491] systemd[1]: Dependency failed for Remote File Systems.
[    1.377139] systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.
[    1.378037] systemd[1]: Starting dracut pre-pivot and cleanup hook...
[    1.388945] rpc.idmapd[345]: exiting on signal 15

And then there is an endless stream of:

[    2.464521] kdump[486]: Bad kdump network destination: 10.111.113.2
[    3.486197] kdump[492]: Bad kdump network destination: 10.111.113.2
[...]

this never stops, and thus one can never log in. This is a rather bad failure mode.

Expected results: initrd configures the network correctly and NFS works, as until two weeks ago. Also, if network configuration fails, it should time out trying to dump, and not get stuck indefinitely.

Additional info:
Comment 1 Coiby 2022-12-13 06:41:42 UTC 
Hi Martin,

Could you paste the output of "debug=1 kdumpctl rebuild 2>&1 | grep -B150 kdump_setup_znet" as I failed to reproduce this issue? Thanks!
Comment 2 Coiby 2022-12-13 06:46:01 UTC 
Btw, printing "Bad kdump network destination: 10.111.113.2" would be stopped after 10 minutes so it wouldn't get stuck forever. Of course, it's better to improve current error message.
Comment 3 Martin Pitt 2022-12-13 09:23:21 UTC 
Sure!

# debug=1 kdumpctl rebuild 2>&1 | grep -B150 kdump_setup_znet
dracut-install: get_real_file('/usr/sbin/ip')
dracut-install: get_real_file('/usr/bin/ip')
dracut-install: get_real_file(/usr/bin/ip) not found
dracut-install: find_binary(ip) == /usr/sbin/ip
dracut-install: dracut_install '/usr/sbin/ip'
dracut-install: dracut_install('/usr/sbin/ip', '/usr/sbin/ip', 0, 0, 1)
dracut-install: '/var/tmp/dracut.FagmuW/initramfs/usr/sbin/ip' already exists
dracut-install: dracut_install ret = 0
dracut-install: dracut_install '/usr/sbin/ip' OK
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@1049(install): kdump_install_net
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@567(kdump_install_net): local _netifs
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@569(kdump_install_net): _get_kdump_netifs
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@10(_get_kdump_netifs): echo -n eth1
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@569(kdump_install_net): _netifs=eth1
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@570(kdump_install_net): [[ -n eth1 ]]
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@571(kdump_install_net): kdump_install_nmconnections
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@346(kdump_install_nmconnections): local _netif _nm_conn_path _cloned_nm_path
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@345(kdump_install_nmconnections): nmcli -t -f device,filename connection show --active
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@348(kdump_install_nmconnections): IFS=:
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@348(kdump_install_nmconnections): read -r _netif _nm_conn_path
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@349(kdump_install_nmconnections): [[ -v unique_netifs[eth0] ]]
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@349(kdump_install_nmconnections): continue
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@348(kdump_install_nmconnections): IFS=:
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@348(kdump_install_nmconnections): read -r _netif _nm_conn_path
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@349(kdump_install_nmconnections): [[ -v unique_netifs[lo] ]]
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@349(kdump_install_nmconnections): continue
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@348(kdump_install_nmconnections): IFS=:
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@348(kdump_install_nmconnections): read -r _netif _nm_conn_path
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@349(kdump_install_nmconnections): [[ -v unique_netifs[eth1] ]]
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@350(kdump_install_nmconnections): clone_and_modify_nmconnection eth1 /etc/sysconfig/network-scripts/ifcfg-static-52-54-01
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@306(clone_and_modify_nmconnection): local _dev _cloned_nmconnection_file_path _tmp_nmconnection_file_path _old_uuid _uuid
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@308(clone_and_modify_nmconnection): _dev=eth1
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@309(clone_and_modify_nmconnection): _nmconnection_file_path=/etc/sysconfig/network-scripts/ifcfg-static-52-54-01
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@311(clone_and_modify_nmconnection): nmcli --get-values connection.uuid connection show filename /etc/sysconfig/network-scripts/ifcfg-static-52-54-01
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@311(clone_and_modify_nmconnection): _old_uuid=07f56f14-f46a-4106-bee9-036bf610fc8e
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@313(clone_and_modify_nmconnection): _clone_nmconnection 07f56f14-f46a-4106-bee9-036bf610fc8e
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@275(_clone_nmconnection): local _clone_output _name _unique_id
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@277(_clone_nmconnection): _unique_id=07f56f14-f46a-4106-bee9-036bf610fc8e
////usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@278(_clone_nmconnection): nmcli --get-values connection.id connection show 07f56f14-f46a-4106-bee9-036bf610fc8e
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@278(_clone_nmconnection): _name='static-52\:54\:01'
////usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@279(_clone_nmconnection): nmcli connection clone --temporary uuid 07f56f14-f46a-4106-bee9-036bf610fc8e 'static-52\:54\:01'
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@279(_clone_nmconnection): _clone_output='static-52:54:01 (07f56f14-f46a-4106-bee9-036bf610fc8e) cloned as static-52\:54\:01 (4b3d597b-9f35-451d-9006-6881f0c4d1f3).'
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@280(_clone_nmconnection): sed -E -n 's/.* \(.*\) cloned as.*\((.*)\)\.$/\1/p'
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@281(_clone_nmconnection): return 0
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@313(clone_and_modify_nmconnection): _uuid=4b3d597b-9f35-451d-9006-6881f0c4d1f3
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@318(clone_and_modify_nmconnection): use_ipv4_or_ipv6 eth1 4b3d597b-9f35-451d-9006-6881f0c4d1f3
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@257(use_ipv4_or_ipv6): local _netif=eth1 _uuid=4b3d597b-9f35-451d-9006-6881f0c4d1f3
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@259(use_ipv4_or_ipv6): [[ -v ipv4_usage[eth1] ]]
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@260(use_ipv4_or_ipv6): nmcli connection modify --temporary 4b3d597b-9f35-451d-9006-6881f0c4d1f3 ipv4.may-fail no
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@260(use_ipv4_or_ipv6): ddebug
///usr/lib/dracut/dracut-logger.sh@390(ddebug): set +x
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@263(use_ipv4_or_ipv6): [[ -v ipv6_usage[eth1] ]]
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@267(use_ipv4_or_ipv6): [[ -v ipv4_usage[eth1] ]]
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@267(use_ipv4_or_ipv6): [[ ! -v ipv6_usage[eth1] ]]
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@268(use_ipv4_or_ipv6): nmcli connection modify --temporary 4b3d597b-9f35-451d-9006-6881f0c4d1f3 ipv6.method disabled
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@268(use_ipv4_or_ipv6): ddebug
///usr/lib/dracut/dracut-logger.sh@390(ddebug): set +x
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@320(clone_and_modify_nmconnection): nmcli connection modify --temporary uuid 4b3d597b-9f35-451d-9006-6881f0c4d1f3 connection.wait-device-timeout 60000
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@320(clone_and_modify_nmconnection): ddebug
///usr/lib/dracut/dracut-logger.sh@390(ddebug): set +x
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@323(clone_and_modify_nmconnection): _match_nmconnection_by_mac 4b3d597b-9f35-451d-9006-6881f0c4d1f3 eth1
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@288(_match_nmconnection_by_mac): local _unique_id _dev _mac _mac_field
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@290(_match_nmconnection_by_mac): _unique_id=4b3d597b-9f35-451d-9006-6881f0c4d1f3
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@291(_match_nmconnection_by_mac): _dev=eth1
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@293(_match_nmconnection_by_mac): kdump_get_perm_addr eth1
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@231(kdump_get_perm_addr): local addr
////usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@232(kdump_get_perm_addr): sed -e 's/Permanent address: //'
////usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@232(kdump_get_perm_addr): ethtool -P eth1
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@232(kdump_get_perm_addr): addr=52:54:01:00:00:01
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@233(kdump_get_perm_addr): [[ -z 52:54:01:00:00:01 ]]
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@233(kdump_get_perm_addr): [[ 52:54:01:00:00:01 == \0\0\:\0\0\:\0\0\:\0\0\:\0\0\:\0\0 ]]
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@236(kdump_get_perm_addr): echo 52:54:01:00:00:01
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@293(_match_nmconnection_by_mac): _mac=52:54:01:00:00:01
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@294(_match_nmconnection_by_mac): [[ 52:54:01:00:00:01 != \n\o\t\ \s\e\t ]]
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@295(_match_nmconnection_by_mac): nmcli --get-values connection.type connection show 4b3d597b-9f35-451d-9006-6881f0c4d1f3
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@295(_match_nmconnection_by_mac): _mac_field=802-3-ethernet.mac-address
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@296(_match_nmconnection_by_mac): nmcli connection modify --temporary 4b3d597b-9f35-451d-9006-6881f0c4d1f3 802-3-ethernet.mac-address 52:54:01:00:00:01
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@296(_match_nmconnection_by_mac): ddebug
///usr/lib/dracut/dracut-logger.sh@390(ddebug): set +x
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@297(_match_nmconnection_by_mac): nmcli connection modify --temporary 4b3d597b-9f35-451d-9006-6881f0c4d1f3 connection.interface-name ''
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@297(_match_nmconnection_by_mac): ddebug
///usr/lib/dracut/dracut-logger.sh@390(ddebug): set +x
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@325(clone_and_modify_nmconnection): sed -n 's/^4b3d597b-9f35-451d-9006-6881f0c4d1f3://p'
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@325(clone_and_modify_nmconnection): nmcli --get-values UUID,FILENAME connection show
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@325(clone_and_modify_nmconnection): _cloned_nmconnection_file_path='/run/NetworkManager/system-connections/static-52\\\:54\\\:01.nmconnection'
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@326(clone_and_modify_nmconnection): basename /etc/sysconfig/network-scripts/ifcfg-static-52-54-01
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@326(clone_and_modify_nmconnection): _tmp_nmconnection_file_path=/tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@327(clone_and_modify_nmconnection): cp '/run/NetworkManager/system-connections/static-52\\\:54\\\:01.nmconnection' /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01
cp: cannot stat '/run/NetworkManager/system-connections/static-52\\\:54\\\:01.nmconnection': No such file or directory
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@331(clone_and_modify_nmconnection): sed -i -E 's/(^uuid=).*$/\107f56f14-f46a-4106-bee9-036bf610fc8e/g' /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01
sed: can't read /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01: No such file or directory
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@332(clone_and_modify_nmconnection): nmcli connection del 4b3d597b-9f35-451d-9006-6881f0c4d1f3
///usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@332(clone_and_modify_nmconnection): ddebug
///usr/lib/dracut/dracut-logger.sh@390(ddebug): set +x
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@333(clone_and_modify_nmconnection): echo -n /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@350(kdump_install_nmconnections): _cloned_nm_path=/tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@351(kdump_install_nmconnections): _install_nmconnection /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@337(_install_nmconnection): local _src _nmconnection_name _dst
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@339(_install_nmconnection): _src=/tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@340(_install_nmconnection): basename /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@340(_install_nmconnection): _nmconnection_name=ifcfg-static-52-54-01
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@341(_install_nmconnection): _dst=/etc/NetworkManager/system-connections/ifcfg-static-52-54-01
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@342(_install_nmconnection): inst /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01 /etc/NetworkManager/system-connections/ifcfg-static-52-54-01
/usr/lib/dracut/dracut-init.sh@219(inst): local _hostonly_install
/usr/lib/dracut/dracut-init.sh@220(inst): [[ /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01 == \-\H ]]
/usr/lib/dracut/dracut-init.sh@224(inst): [[ -e /var/tmp/dracut.FagmuW/initramfs//etc/NetworkManager/system-connections/ifcfg-static-52-54-01 ]]
/usr/lib/dracut/dracut-init.sh@226(inst): /usr/lib/dracut/dracut-install -D /var/tmp/dracut.FagmuW/initramfs /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01 /etc/NetworkManager/system-connections/ifcfg-static-52-54-01
dracut-install: Program arguments:
dracut-install: /usr/lib/dracut/dracut-install
dracut-install: -D
dracut-install: /var/tmp/dracut.FagmuW/initramfs
dracut-install: /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01
dracut-install: /etc/NetworkManager/system-connections/ifcfg-static-52-54-01
dracut-install: PATH=/usr/sbin:/usr/bin
dracut-install: LDD=ldd
dracut-install: dracut_install('/tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01', '/etc/NetworkManager/system-connections/ifcfg-static-52-54-01', 0, 0, 1)
dracut-install: ERROR: installing '/tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01' to '/etc/NetworkManager/system-connections/ifcfg-static-52-54-01'
/usr/lib/dracut/dracut-init.sh@227(inst): derror FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.FagmuW/initramfs /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01 /etc/NetworkManager/system-connections/ifcfg-static-52-54-01
/usr/lib/dracut/dracut-logger.sh@438(derror): set +x
dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.FagmuW/initramfs /tmp/3391-DRACUT_KDUMP_NM/ifcfg-static-52-54-01 /etc/NetworkManager/system-connections/ifcfg-static-52-54-01
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@348(kdump_install_nmconnections): IFS=:
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@348(kdump_install_nmconnections): read -r _netif _nm_conn_path
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@361(kdump_install_nmconnections): echo
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@572(kdump_install_net): apply_nm_initrd_generator_timeouts
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@241(apply_nm_initrd_generator_timeouts): local _timeout_conf
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@243(apply_nm_initrd_generator_timeouts): _timeout_conf=/tmp/3391-DRACUT_KDUMP_NM/timeout_conf
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@244(apply_nm_initrd_generator_timeouts): cat
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@253(apply_nm_initrd_generator_timeouts): inst /tmp/3391-DRACUT_KDUMP_NM/timeout_conf /etc/NetworkManager/conf.d/95-kdump-timeouts.conf
/usr/lib/dracut/dracut-init.sh@219(inst): local _hostonly_install
/usr/lib/dracut/dracut-init.sh@220(inst): [[ /tmp/3391-DRACUT_KDUMP_NM/timeout_conf == \-\H ]]
/usr/lib/dracut/dracut-init.sh@224(inst): [[ -e /var/tmp/dracut.FagmuW/initramfs//etc/NetworkManager/conf.d/95-kdump-timeouts.conf ]]
/usr/lib/dracut/dracut-init.sh@226(inst): /usr/lib/dracut/dracut-install -D /var/tmp/dracut.FagmuW/initramfs /tmp/3391-DRACUT_KDUMP_NM/timeout_conf /etc/NetworkManager/conf.d/95-kdump-timeouts.conf
dracut-install: Program arguments:
dracut-install: /usr/lib/dracut/dracut-install
dracut-install: -D
dracut-install: /var/tmp/dracut.FagmuW/initramfs
dracut-install: /tmp/3391-DRACUT_KDUMP_NM/timeout_conf
dracut-install: /etc/NetworkManager/conf.d/95-kdump-timeouts.conf
dracut-install: PATH=/usr/sbin:/usr/bin
dracut-install: LDD=ldd
dracut-install: dracut_install('/tmp/3391-DRACUT_KDUMP_NM/timeout_conf', '/etc/NetworkManager/conf.d/95-kdump-timeouts.conf', 0, 0, 1)
dracut-install: dest dir '/var/tmp/dracut.FagmuW/initramfs/etc/NetworkManager/conf.d' does not exist
dracut-install: dracut_install('/etc/NetworkManager/conf.d', '/etc/NetworkManager/conf.d', 1, 0, 1)
dracut-install: dest dir '/var/tmp/dracut.FagmuW/initramfs/etc/NetworkManager' does not exist
dracut-install: dracut_install('/etc/NetworkManager', '/etc/NetworkManager', 1, 0, 1)
dracut-install: mkdir '/var/tmp/dracut.FagmuW/initramfs/etc/NetworkManager'
dracut-install: mkdir '/var/tmp/dracut.FagmuW/initramfs/etc/NetworkManager/conf.d'
dracut-install: dracut_install ret = 0
dracut-install: cp '/tmp/3391-DRACUT_KDUMP_NM/timeout_conf' '/var/tmp/dracut.FagmuW/initramfs/etc/NetworkManager/conf.d/95-kdump-timeouts.conf'
dracut-install: dracut_install ret = 0
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@573(kdump_install_net): kdump_setup_znet
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@480(kdump_setup_znet): local _config_file _unique_name _NM_conf_dir
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@481(kdump_setup_znet): local '__sed_discard_ignored_files=/\(~\|\.bak\|\.old\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d'
//usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@483(kdump_setup_znet): uname -m
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@483(kdump_setup_znet): [[ x86_64 != \s\3\9\0\x ]]
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh@484(kdump_setup_znet): return


Just in case it helps:

# nmcli c
NAME             UUID                                  TYPE      DEVICE 
System eth0      5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03  ethernet  eth0   
lo               d32ad476-cec4-4e39-ad30-50b489e44ee2  loopback  lo     
static-52:54:01  07f56f14-f46a-4106-bee9-036bf610fc8e  ethernet  eth1   
System eth1      9c92fad9-6ecb-3e6c-eb4d-8a47c6f50c04  ethernet  --     

# nmcli d
DEVICE  TYPE      STATE                   CONNECTION      
eth0    ethernet  connected               System eth0     
eth1    ethernet  connected               static-52:54:01 
lo      loopback  connected (externally)  lo     

# ip route
10.111.113.0/24 dev eth1 proto kernel scope link src 10.111.113.1 metric 101 
172.27.0.0/24 dev eth0 proto kernel scope link src 172.27.0.15 metric 100 

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    altname enp0s14
    altname ens14
    inet 172.27.0.15/24 brd 172.27.0.255 scope global dynamic noprefixroute eth0
       valid_lft 86139sec preferred_lft 86139sec
    inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr 
       valid_lft 86141sec preferred_lft 14141sec
    inet6 fe80::5054:ff:fe12:3456/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:01:00:00:01 brd ff:ff:ff:ff:ff:ff
    altname enp0s15
    altname ens15
    inet 10.111.113.1/24 brd 10.111.113.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5d95:fc6e:e731:605/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever


> would be stopped after 10 minutes so it wouldn't get stuck forever

Thanks, good to know. I suppose contemporary developer patience is shorter than that 😀
Comment 4 Coiby 2022-12-14 07:53:22 UTC 
Thanks for sharing the debugging output! I'll discuss with NetworkManager team about whether it should be fixed in NetworkManager or in kexec-tools. Meanwhile, as a workaround, you can avoid using ":" when specifying the value for the NAME filed in /etc/sysconfig/network-scripts/ifcfg-static-52-54-01. Currently this file has "NAME=static-52:54:01", you can remove this field or change it to "NAME=static-52-54-01".
Comment 5 Coiby 2022-12-14 08:04:07 UTC 
Hi NM team,

The root cause of this bug is when the name of a connection profile contains ":", "nmcli --get-values UUID,FILENAME connection show" will somehow try to escape ":", 

  $ nmcli --get-values UUID,FILENAME connection show
  592abf2e-d0d2-4622-87cf-ccb0a8cd8a10:/run/NetworkManager/system-connections/static-52\:54\:01.nmconnection

and we can't access this connection profile based on the output of "nmcli --get-values FILENAME"
  $ ls -l "/run/NetworkManager/system-connections/static-52\:54\:01.nmconnection"
  ls: cannot access '/run/NetworkManager/system-connections/static-52\:54\:01.nmconnection': No such file or directory

IMHO, ":" is a valid character so there is no need to escape it,

  $ ls -l /run/NetworkManager/system-connections/static-52:54:01.nmconnection
  -rw-------. 1 root root 284 Dec 14 02:34 /run/NetworkManager/system-connections/static-52:54:01.nmconnection
  $ ls -l "/run/NetworkManager/system-connections/static-52:54:01.nmconnection"
  -rw-------. 1 root root 284 Dec 14 02:34 /run/NetworkManager/system-connections/static-52:54:01.nmconnection

So I think this bug should be fixed in NetworkManager instead. Please free to reassign it back to kexec-tools if you don't think so.
Comment 6 Martin Pitt 2022-12-14 10:39:00 UTC 
Thanks Coiby! I confirm that changing NAME= that way fixes NFS dumps again. Neither our image scripts nor our test actually touches /etc/sysconfig/network-scripts (hasn't that been legacy for a *looong* time already?). That ifcfg-static-52-54-01 file doesn't even exist in our base images, NM generates it dynamically once we connect the second interface eth1 to a virtual network with another VM:

    $ nmcli c | grep static
    static-52:54:01  21515f38-98c3-4782-848b-f0c68893368a  ethernet  eth1 

So it sounds plausible to fix that in NM as well, thank you for your investigation!
Comment 10 Coiby 2022-12-19 10:38:07 UTC 
(In reply to Martin Pitt from comment #6)
> Thanks Coiby! I confirm that changing NAME= that way fixes NFS dumps again.

Good to know that!

> Neither our image scripts nor our test actually touches
> /etc/sysconfig/network-scripts (hasn't that been legacy for a *looong* time
> already?). That ifcfg-static-52-54-01 file doesn't even exist in our base
> images, NM generates it dynamically once we connect the second interface
> eth1 to a virtual network with another VM:
> 
>     $ nmcli c | grep static
>     static-52:54:01  21515f38-98c3-4782-848b-f0c68893368a  ethernet  eth1 
> 
> So it sounds plausible to fix that in NM as well, thank you for your
> investigation!

This seems to be a second issue from the one I described in comment 5 and I tend to think it is not a NM issue.

As said by you, ifcfg has been legacy and it has been already announced in RHEL9 release notes [1],

> By default, NetworkManager now uses the key files to store new connection profiles. Note that the ifcfg format is still supported.

So NM generates new connection profile as a key file (.nmconnection) and according to my experience it's always generated under/run/NetworkManager/system-connections.

Could you run "nmcli c | grep static" before you connect the second interface eth1 to a virtual network?

[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/9.0_release_notes/overview#overview-major-changes
Comment 11 Martin Pitt 2022-12-20 11:21:58 UTC 
> Could you run "nmcli c | grep static" before you connect the second interface eth1 to a virtual network?

Hm, tricky -- the interface is already connected when the VM starts, and NM apparently does that during boot. Would a journal help instead?
Comment 12 Coiby 2022-12-21 05:49:21 UTC 
(In reply to Martin Pitt from comment #11)
> > Could you run "nmcli c | grep static" before you connect the second interface eth1 to a virtual network?
> 
> Hm, tricky -- the interface is already connected when the VM starts, and NM
> apparently does that during boot. Would a journal help instead?

If the file is created by NM, we shall be able to find text like "Dec 21 00:46:02 ... NetworkManager[4990]: <info>  [1671601562.7247] settings: (eth0): created default wired connection 'Wired connection 1'" in `journal -U NetworkManager".
Comment 13 Coiby 2022-12-21 05:56:24 UTC 
Another way is first delete that connection profile for eth1 and restart NetworkManager, i.e.
1. nmcli c del static-52:54:01
2. systemctl restart NetworkManager

NM should create a connection profile for eth1. We can then run "nmcli -f device,filename c show" to check if /etc/sysconfig/network-scripts/ifcfg-static-52-54-01 is created.
Comment 14 Martin Pitt 2022-12-25 18:21:33 UTC 
Thanks Coiby. Indeed NM does not seem to auto-create the connection after all, sorry -- our test infra does with some nmcli commands. After `nmcli c del static-52:54:01` I start from a clean slate wrt. eth1:

# nmcli c
NAME         UUID                                  TYPE      DEVICE 
System eth0  5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03  ethernet  eth0   
lo           1625aaed-a7f9-4613-8f6a-f495f4aa1f04  loopback  lo     
System eth1  9c92fad9-6ecb-3e6c-eb4d-8a47c6f50c04  ethernet  --     

# nmcli d
DEVICE  TYPE      STATE                   CONNECTION  
eth0    ethernet  connected               System eth0 
lo      loopback  connected (externally)  lo          
eth1    ethernet  disconnected            --     

# cat /etc/sysconfig/network-scripts/ifcfg-static*
cat: '/etc/sysconfig/network-scripts/ifcfg-static*': No such file or directory
   
# ip -c a show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:01:00:00:01 brd ff:ff:ff:ff:ff:ff
    altname enp0s15
    altname ens15

Now I add it like our tests do:

# nmcli con add type ethernet autoconnect yes con-name static-52:54:01 ifname eth1 ip4 10.111.113.1/24
Connection 'static-52:54:01' (f645fbc3-6425-423e-a617-08fa39759854) successfully added.
# nmcli con up static-52:54:01
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/6)


This creates the sysconfig file (although this is deprecated):

# cat /etc/sysconfig/network-scripts/ifcfg-static-52-54-01 
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=10.111.113.1
PREFIX=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=default
NAME=static-52:54:01
UUID=f645fbc3-6425-423e-a617-08fa39759854
DEVICE=eth1
ONBOOT=yes


I don't seem to get this quoting bug, though:

# nmcli --get-values UUID,FILENAME connection show
5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03:/etc/sysconfig/network-scripts/ifcfg-eth0
1625aaed-a7f9-4613-8f6a-f495f4aa1f04:/run/NetworkManager/system-connections/lo.nmconnection
4d60832e-36a8-4c11-b227-4bd02639ebc6:/etc/sysconfig/network-scripts/ifcfg-static-52-54-01
9c92fad9-6ecb-3e6c-eb4d-8a47c6f50c04:/etc/sysconfig/network-scripts/ifcfg-eth1

I.e. the file is already named with dashes instead of colons.

But I do confirm that replacing the ':' with '-' in the "con-name" argument of "nmcli con add" works around this issue. I tested/submitted that as https://github.com/cockpit-project/bots/pull/4203
Comment 15 Coiby 2022-12-28 03:05:17 UTC 
(In reply to Martin Pitt from comment #14)
> Thanks Coiby. Indeed NM does not seem to auto-create the connection after
> all, sorry -- our test infra does with some nmcli commands. After `nmcli c
> del static-52:54:01` I start from a clean slate wrt. eth1:
> 
...
> 
> # nmcli con add type ethernet autoconnect yes con-name static-52:54:01
> ifname eth1 ip4 10.111.113.1/24
> Connection 'static-52:54:01' (f645fbc3-6425-423e-a617-08fa39759854)
> successfully added.
> # nmcli con up static-52:54:01
> Connection successfully activated (D-Bus active path:
> /org/freedesktop/NetworkManager/ActiveConnection/6)

Oh, you use static IP. This may be the reason that NM doesn't automatically a connection profile for you.

> 
> 
> This creates the sysconfig file (although this is deprecated):
> 
> # cat /etc/sysconfig/network-scripts/ifcfg-static-52-54-01 
> ...
> ONBOOT=yes

You must have configured NM to use ifcfg by defaut.

For newly installed system, NM creates a .nmconnection by default

    [root@dell-per7425-08 ~]# grep -A1 "\[main" /etc/NetworkManager/NetworkManager.conf
    [main]
    #plugins=keyfile,ifcfg-rh
    
    [root@dell-per7425-08 ~]# nmcli con add type ethernet autoconnect yes con-name static-52:54:01 ifname eth1 ip4 10.111.113.1/24
    
    [root@dell-per7425-08 ~]# nmcli -f name,filename c show
    NAME                FILENAME                                                               
    static-52:54:01     /etc/NetworkManager/system-connections/static-52:54:01.nmconnection    

unless you choose to use ifcfg by default,

    [root@ci-vm-10-0-138-233 test]# grep -A1 "\[main" /etc/NetworkManager/NetworkManager.conf
    [main]
    plugins = ifcfg-rh,
    
    [root@ci-vm-10-0-138-233 test]# nmcli con add type ethernet autoconnect yes con-name static-52:54:01 ifname eth1 ip4 10.111.113.1/24
    Connection 'static-52:54:01' (30f33f23-1ff4-4f1e-a575-1eed4545ea4d) successfully added.
    
    [root@ci-vm-10-0-138-233 test]# nmcli -f name,filename c show
    NAME             FILENAME                                                            
    static-52:54:01  /etc/sysconfig/network-scripts/ifcfg-static-52-54-01                


> 
> But I do confirm that replacing the ':' with '-' in the "con-name" argument
> of "nmcli con add" works around this issue. I tested/submitted that as
> https://github.com/cockpit-project/bots/pull/4203

Good to know that!
Comment 16 Coiby 2022-12-28 03:13:27 UTC 
> I don't seem to get this quoting bug, though:
> 
> # nmcli --get-values UUID,FILENAME connection show
> 4d60832e-36a8-4c11-b227-4bd02639ebc6:/etc/sysconfig/network-scripts/ifcfg-
> static-52-54-01
> I.e. the file is already named with dashes instead of colons.

The problem is that created or cloned connection profile has colons in the filename. Today I further narrowed the scope of this bug following comment 5. It turns out there is something wrong with "nmlic -t | --terse" ("nmcli -g | --get-values" is a shortcut for --mode tabular --terse --fields),
    [root@ci-vm-10-0-139-157 t2]# nmcli --terse -f NAME,FILENAME connection show
    static-52\:54\:01:/run/NetworkManager/system-connections/static-52\:54\:01.nmconnection
    [root@ci-vm-10-0-139-157 t2]# nmcli  -f NAME,FILENAME connection show
    NAME                FILENAME                                                               
    static-52:54:01     /run/NetworkManager/system-connections/static-52:54:01.nmconnection    

    [root@ci-vm-10-0-139-157 t2]# nmcli  -f NAME connection show
    NAME               
    static-52:54:01    
    [root@ci-vm-10-0-139-157 t2]# nmcli --terse -f NAME connection show
    static-52\:54\:01
Comment 17 Martin Pitt 2023-01-02 15:22:33 UTC 
(In reply to Coiby from comment #15)
> You must have configured NM to use ifcfg by defaut.

Not deliberately, but indeed..

>     [root@ci-vm-10-0-138-233 test]# grep -A1 "\[main" /etc/NetworkManager/NetworkManager.conf
>     [main]
>     plugins = ifcfg-rh,

... I do have that on our image. When I drop it and restart NM, then the connection gets saved as a keyfile:

    static-52:54:01     /etc/NetworkManager/system-connections/static-52:54:01.nmconnection  

I just tested that on a pristine cloud image:

curl -o rhel.qcow2 http://download.eng.brq.redhat.com/rhel-9/nightly/RHEL-9/latest-RHEL-9.2/compose/BaseOS/x86_64/images/rhel-guest-image-9.2-20230102.0.x86_64.qcow2
# nothing fancy, just admin:foobar and root:foobar
curl -L -O https://github.com/cockpit-project/bots/raw/main/machine/cloud-init.iso
qemu-system-x86_64 -cpu host -enable-kvm -nographic -m 2048 -drive file=rhel.qcow2,if=virtio -snapshot -cdrom cloud-init.iso 

.. logging in with root:foobar, I see

# grep ^plugins /etc/NetworkManager/NetworkManager.conf
plugins = ifcfg-rh,

So it's not actually true that the old network-scripts is deprecated -- it's still the default in RHEL. I'd rather not change such a fundamental thing in our CI images, we want to stay as close as possible to what real users will see.

FTR, Fedora's cloud images are fine, they don't use sysconfig any more.
Comment 18 Martin Pitt 2023-01-02 15:26:06 UTC 
> it's still the default in RHEL.

A-ha! Seems it's cloud-init:

/usr/lib/python3.9/site-packages/cloudinit/net/sysconfig.py:    """Add ifcfg-rh to NetworkManager.cfg plugins if main section is present"
/usr/lib/python3.9/site-packages/cloudinit/net/sysconfig.py:        if "plugins" in config["main"]:

and a few more. So Anaconda installs won't have it, but cloud instances do. Our cloud-init does not mention any network/sysconfig stuff, so that must be a default module:
https://github.com/cockpit-project/bots/blob/main/machine/make-cloud-init-iso
Comment 19 Beniamino Galvani 2023-03-13 16:27:24 UTC 
Hi,

in the output of "nmcli --get-values", fields are separated by a colon and therefore other colons must be escaped. I see that the issue is about:

  nmcli --get-values UUID,FILENAME connection show

that returns something like:

  dc8eaa87-4984-481f-bc84-e1d0d817927c:/etc/NetworkManager/system-connections/test-00\:11\:22.nmconnection

In this case, since the UUID has a fixed format without colons, you can pass "--escape no" to get the value without escape sequences for ':' and '\`:

  $ nmcli --escape no --get-values UUID,FILENAME connection show
  dc8eaa87-4984-481f-bc84-e1d0d817927c:/etc/NetworkManager/system-connections/test-00:11:22.nmconnection

and trim the part up to the first colon to get the file name.

In the long term, a better solution would be that NM implements a JSON output mode like many other tools, which would allow a easier usage in scripts; but that is not planned for now.

So, I think this bz should be reassigned to kexec-tools, which should use the no-escape option in the nmcli command in dracut-module-setup.sh. Coiby, what do you think?
Comment 21 Beniamino Galvani 2023-03-23 12:26:30 UTC 
Reassigning to kexec-tools as per comment 19.
Comment 24 Coiby 2023-03-29 09:07:41 UTC 
(In reply to Beniamino Galvani from comment #19)
> Hi,
> 
> in the output of "nmcli --get-values", fields are separated by a colon and
> therefore other colons must be escaped. I see that the issue is about:
> 
>   nmcli --get-values UUID,FILENAME connection show
> 
> that returns something like:
> 
>  
> dc8eaa87-4984-481f-bc84-e1d0d817927c:/etc/NetworkManager/system-connections/
> test-00\:11\:22.nmconnection
> 
> In this case, since the UUID has a fixed format without colons, you can pass
> "--escape no" to get the value without escape sequences for ':' and '\`:
> 
>   $ nmcli --escape no --get-values UUID,FILENAME connection show
>  
> dc8eaa87-4984-481f-bc84-e1d0d817927c:/etc/NetworkManager/system-connections/
> test-00:11:22.nmconnection
> 
> and trim the part up to the first colon to get the file name.
> 
> In the long term, a better solution would be that NM implements a JSON
> output mode like many other tools, which would allow a easier usage in
> scripts; but that is not planned for now.
> 
> So, I think this bz should be reassigned to kexec-tools, which should use
> the no-escape option in the nmcli command in dracut-module-setup.sh. Coiby,
> what do you think?


Hi Beniamino,

Thanks for the clarification! I've made a change in dracut-module-setup.sh following your suggestion.
Comment 31 errata-xmlrpc 2023-11-07 08:51:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (kexec-tools bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6594