Is removing everything matching
/etc/NetworkManager/system-connections/*.connection
a valid thing to do here?

I can not reproduce this issue, maybe some details are different from what Alex said.

Steps:

-----------------------------------
1. Use virt-install to create a CentOS Stream 9 image (https://github.com/mwhahaha/cloud-images/blob/main/centos-9.stream.virt-install-cmd)

2. Set a password for CentOS Stream 9 image with virt-customize command
virt-customize --root-password password:redhat --uninstall cloud-init --selinux-relabel -a centos-9-stream.img

3. Run virt-sysprep on CentOS Stream 9 image
virt-sysprep -a centos-9-stream.img

4. Reboot the CentOS Stream 9 image
virt-install --import --vcpus 2 --ram 2048 --disk format=qcow2,path=centos-9-stream.img  --network network=default,model=virtio -noautoconsole --nographics --name test1
...
CentOS Stream 9
Kernel 5.13.0-1.el9.x86_64 on an x86_64

localhost login: root
Password: 
[root@localhost ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:f2:b3:3b brd ff:ff:ff:ff:ff:ff
    altname enp0s2
    inet 192.168.122.103/24 brd 192.168.122.255 scope global dynamic noprefixroute ens2
       valid_lft 3589sec preferred_lft 3589sec
    inet6 fe80::621b:8145:4e5e:4a03/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[root@localhost ~]# nmcli device
DEVICE  TYPE      STATE      CONNECTION         
ens2    ethernet  connected  Wired connection 1 
lo      loopback  unmanaged  --  
               
[root@localhost ~]# nmcli connection show
NAME                UUID                                  TYPE      DEVICE 
Wired connection 1  63b29e4d-3049-3583-81f8-26d4ae46bf51  ethernet  ens2   
enp1s0              936d6a28-fe0a-4643-b8e1-948302fd0400  ethernet  -- 

[root@localhost ~]# ll /etc/NetworkManager/system-connections/
total 4
-rw-------. 1 root root 242 Jul 20 05:15 enp1s0.nmconnection

-----------------------------------

The /etc/NetworkManager/system-connections/enp1s0.nmconnection has not been deleted by virt-sysprep, but seems it doesn't affect the reboot process and network function. In RHEL8, virt-sysprep also won't delete the /etc/sysconfig/network-scripts/ifcfg-*.

Is there anything I missed?

I think i'm running into a bug in cloud-init that's conflicting with network manager.  We can ignore that part for now as I need to write that up.  In terms of leaving the config around, I don't think we should. We've run into issues with network-scripts historically because the images have left ifcfg-ens3 around and restarting the network services fails.  Any interfaces used during the image build should be purged via sysprep so that we don't up with latent issues later.


https://bugs.launchpad.net/tripleo/+bug/1866202
https://bugs.centos.org/view.php?id=17133

While I agree that virt-sysprep should do it, I'll note that you
can do this already with:

  virt-sysprep ... --delete "/etc/NetworkManager/system-connections/*.connection"

When build ec2 image from ks file, there is a boot time issue bz1862930 because of dirty data generated by NM in '/etc/resolv.conf'.

I am wondering if virt-sysprep can clean it if others do not know this point.

Below is '/etc/resolv.conf' content which impacts boot time on aws.
# guestfish -a rhel-ec2-8.5-919.x86_64.raw -i sh 'cat  /etc/resolv.conf'
# Generated by NetworkManager
nameserver 192.168.122.1

(In reply to Frank Liang from comment #6)
> When build ec2 image from ks file, there is a boot time issue bz1862930
> because of dirty data generated by NM in '/etc/resolv.conf'.
> 
> I am wondering if virt-sysprep can clean it if others do not know this point.
> 
> Below is '/etc/resolv.conf' content which impacts boot time on aws.
> # guestfish -a rhel-ec2-8.5-919.x86_64.raw -i sh 'cat  /etc/resolv.conf'
> # Generated by NetworkManager
> nameserver 192.168.122.1

This does not seem necessary:

https://code.engineering.redhat.com/gerrit/c/spin-kickstarts/+/266458
superseded by:
https://code.engineering.redhat.com/gerrit/c/spin-kickstarts/+/266717

(In reply to YongkuiGuo from comment #2)
> In RHEL8, virt-sysprep also won't delete the
> /etc/sysconfig/network-scripts/ifcfg-*.

(In reply to Alex Schultz from comment #3)
> In terms of leaving the config around, I don't think we should. We've
> run into issues with network-scripts historically because the images
> have left ifcfg-ens3 around and restarting the network services
> fails.  Any interfaces used during the image build should be purged
> via sysprep so that we don't up with latent issues later.

virt-sysprep already has two actions related to this, namely
"net-hostname" and "net-hwaddr". Both actions update the files matching

  /etc/sysconfig/network-scripts/ifcfg-*

The former removes HOSTNAME=... and DHCP_HOSTNAME=... entries, the
latter removes HWADDR=... lines.

The first action comes originally from commit 5311c50f52cd ("sysprep:
remove hostname from ifcfg-*", 2012-09-03; v1.45.1); the second action
from commit 44e04a6fca6a ("Rewrite virt-sysprep.", 2012-03-31; v1.45.1).

Both operations are enabled by default. And neither seems compatible
with the idea of removing the ifcfg-* files altogether.

By extension, why is it right to remove

  /etc/NetworkManager/system-connections/*.nmconnection

whole-sale? (Note BTW: the file suffix is not ".connection", but
".nmconnection".)

Should we retire "net-hostname" and "net-hwaddr", replacing them with an
operation that removes these files entirely?

(In reply to Laszlo Ersek from comment #8)
> (In reply to YongkuiGuo from comment #2)
> > In RHEL8, virt-sysprep also won't delete the
> > /etc/sysconfig/network-scripts/ifcfg-*.
> 
> (In reply to Alex Schultz from comment #3)
> > In terms of leaving the config around, I don't think we should. We've
> > run into issues with network-scripts historically because the images
> > have left ifcfg-ens3 around and restarting the network services
> > fails.  Any interfaces used during the image build should be purged
> > via sysprep so that we don't up with latent issues later.
> 
> virt-sysprep already has two actions related to this, namely
> "net-hostname" and "net-hwaddr". Both actions update the files matching
> 
>   /etc/sysconfig/network-scripts/ifcfg-*
> 
> The former removes HOSTNAME=... and DHCP_HOSTNAME=... entries, the
> latter removes HWADDR=... lines.
> 
> The first action comes originally from commit 5311c50f52cd ("sysprep:
> remove hostname from ifcfg-*", 2012-09-03; v1.45.1); the second action
> from commit 44e04a6fca6a ("Rewrite virt-sysprep.", 2012-03-31; v1.45.1).
> 
> Both operations are enabled by default. And neither seems compatible
> with the idea of removing the ifcfg-* files altogether.
> 
> By extension, why is it right to remove
> 
>   /etc/NetworkManager/system-connections/*.nmconnection
> 
> whole-sale? (Note BTW: the file suffix is not ".connection", but
> ".nmconnection".)
> 

It's actually problematic to leave the files in place at all. Just because when creating an image an ens3 exist, doesn't mean ens3 exists on startup of the image. This has lead to networking (and by extension also NetworkManager) failures.  This existing cleanup is insufficient.  

Related https://bugs.centos.org/view.php?id=15475 https://bugs.centos.org/view.php?id=17133
  
> Should we retire "net-hostname" and "net-hwaddr", replacing them with an
> operation that removes these files entirely?

IMHO yes but I'm just an end user.

I'd prefer to leave the old operations around.  Removing them would break
scripts and also people processing old guests.

Adding a new operation which deletes *.nmconnection sounds like a good idea
from reading the comments, but I'm not an expert.

According to 'virt-sysprep --list-operations':

net-hostname * Remove HOSTNAME and DHCP_HOSTNAME in network interface configuration
net-hwaddr * Remove HWADDR (hard-coded MAC address) configuration

The asterisks indicate that these are default actions. (As I noted earlier.)

Should we make the removal of *.nmconnection default as well?

Here's why I'm asking:

(1) If we delete *.nmconnection by default, then that could be considered inconsistent with virt-sysprep *not* removing ifcfg-* files by default. (And deleting ifcfg-* by default would conflict with the above, default, net-hostname and net-hwaddr actions.)

(2) If we do *not* delete *.nmconnection by default, then we'll require the user to spell it out manually, with either "--enable" or "--operations". But with the exact same effort, the user could just use the "--delete" option on *.nmconnection, like Rich suggested in comment 4.

I'm fine either way (remove *.nmconnection implicitly or explicitly), but I can't decide it myself. Please advise. Thanks!

It does sound like a default operation.  It seems reasonable to have virt-sysprep
clean up NetworkManager connections by default, since they are likely to contain
details about the environment where a template is built.

[guestfs-tools PATCH] sysprep: remove system-local NetworkManager connection profiles (keyfiles)
Message-Id: <20211202131006.12774-1-lersek>
https://listman.redhat.com/archives/libguestfs/2021-December/msg00021.html

(In reply to Laszlo Ersek from comment #13)
> [guestfs-tools PATCH] sysprep: remove system-local NetworkManager connection profiles (keyfiles)
> Message-Id: <20211202131006.12774-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2021-December/msg00021.html

Upstream commit 903819ecf480.

Test with package:
guestfs-tools-1.46.1-6.el9.x86_64


Steps:

1. On rhel9 host
# virt-ls -l -a centos-9-stream.img /etc/NetworkManager/system-connections/
total 4
drwxr-xr-x. 2 root root  33 Jul 20 09:15 .
drwxr-xr-x. 7 root root 134 Jul 20 09:14 ..
-rw-------. 1 root root 242 Jul 20 09:15 enp1s0.nmconnection

2. 
# virt-sysprep -a centos-9-stream.img
...
[   5.5] Performing "net-nmconn" ...
...

3.
# virt-ls -l -a centos-9-stream.img /etc/NetworkManager/system-connections/
total 0
drwxr-xr-x. 2 root root   6 Dec  6 06:29 .
drwxr-xr-x. 7 root root 134 Jul 20 09:14 ..

enp1s0.nmconnection file was deleted by virt-sysprep command.

Verified this bug as this issue has fixed according to the RHEL9 nightly compose test result.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: guestfs-tools), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2461