Quick update:

Seems this is related with the network configuration change in RHEL-9, using NetworkManager by default instead of file ifcfg-xx in rhel-9, so cloud-init failed to config network using ifcfg-xx. The ifcfg-xx will be deprecated in rhel-9, so maybe cloud-init need to fix the methods to configure network.

Other tests:
1. /etc/NetworkManager/system-connections/ens192.nmconnection is effective by default (using nmcli also can configure network)
2. The ifcfg-xx is effective(can configure the static IP successfully) after set: #nmcli con down ens192

Cloud-init enables ifcfg-rh plugin and writes network configuration to network-script file ifcfg-xx, but this fails to load network configuration correctly in rhel-9. See cloud-init related code here: https://github.com/canonical/cloud-init/blob/main/cloudinit/net/sysconfig.py#L68

There is a solution I verified locally which is adding 'AUTOCONNECT_PRIORITY: 999' to network-scripts ifcfg-xx files, NetworkManager will activate network-scripts(ifcfg-rh) instead of keyfile since network-scripts has higher autoconnect priority.

Modified cloud-init file /usr/lib/python3.9/site-packages/cloudinit/net/sysconfig.py
L293 - 'BOOTPROTO': 'none'
     + 'BOOTPROTO': 'none', 'AUTOCONNECT_PRIORITY': 999


Please RedHat team help to review and confirm if this is a workable solution.

(In reply to Pengpeng Sun from comment #12)
> There is a solution I verified locally which is adding
> 'AUTOCONNECT_PRIORITY: 999' to network-scripts ifcfg-xx files,
> NetworkManager will activate network-scripts(ifcfg-rh) instead of keyfile
> since network-scripts has higher autoconnect priority.
> 
> Modified cloud-init file
> /usr/lib/python3.9/site-packages/cloudinit/net/sysconfig.py
> L293 - 'BOOTPROTO': 'none'
>      + 'BOOTPROTO': 'none', 'AUTOCONNECT_PRIORITY': 999
> 
> 
> Please RedHat team help to review and confirm if this is a workable solution.

Thanks Pengpeng, this solution works well in my testing.

(In reply to Pengpeng Sun from comment #12)
> There is a solution I verified locally which is adding
> 'AUTOCONNECT_PRIORITY: 999' to network-scripts ifcfg-xx files,
> NetworkManager will activate network-scripts(ifcfg-rh) instead of keyfile
> since network-scripts has higher autoconnect priority.
> 
> Modified cloud-init file
> /usr/lib/python3.9/site-packages/cloudinit/net/sysconfig.py
> L293 - 'BOOTPROTO': 'none'
>      + 'BOOTPROTO': 'none', 'AUTOCONNECT_PRIORITY': 999
> 
> 
> Please RedHat team help to review and confirm if this is a workable solution.

There's another solution pending rework on github: https://github.com/canonical/cloud-init/pull/1180
But perhaps this one could also be a fix for the problem. Are you sending a pull request for this?

(In reply to Eduardo Otubo from comment #14)
> (In reply to Pengpeng Sun from comment #12)
> > There is a solution I verified locally which is adding
> > 'AUTOCONNECT_PRIORITY: 999' to network-scripts ifcfg-xx files,
> > NetworkManager will activate network-scripts(ifcfg-rh) instead of keyfile
> > since network-scripts has higher autoconnect priority.
> > 
> > Modified cloud-init file
> > /usr/lib/python3.9/site-packages/cloudinit/net/sysconfig.py
> > L293 - 'BOOTPROTO': 'none'
> >      + 'BOOTPROTO': 'none', 'AUTOCONNECT_PRIORITY': 999
> > 
> > 
> > Please RedHat team help to review and confirm if this is a workable solution.
> 
> There's another solution pending rework on github:
> https://github.com/canonical/cloud-init/pull/1180
> But perhaps this one could also be a fix for the problem. Are you sending a
> pull request for this?

Thanks for the info, I've just sent a pull request on the 'AUTOCONNECT_PRIORITY' solution: https://github.com/canonical/cloud-init/pull/1212

@huzhao since you're the reporter, do you think you can set this BZ to public? Cloud-init maintainers would to take a look at least at the public comments on this BZ.

Thanks!

This issue is being discussed in 3 different PR: 

 - Set the highest autoconnect priority for network-scripts #1212
   https://github.com/canonical/cloud-init/pull/1212

 - Add native NetworkManager support #1224
   https://github.com/canonical/cloud-init/pull/1224

 - Fix NetworkManagerActivator error #1180
   https://github.com/canonical/cloud-init/pull/1180    

I'll be working with Lubomir to make sure we merge the NetworkManager support as it seems to be the best fit for the case.

(In reply to Eduardo Otubo from comment #16)
> @huzhao since you're the reporter, do you think you can set this
> BZ to public? Cloud-init maintainers would to take a look at least at the
> public comments on this BZ.
> 
> Thanks!

Yes, I think so, setting this BZ as public

(In reply to Eduardo Otubo from comment #17)
> This issue is being discussed in 3 different PR: 
> 
>  - Set the highest autoconnect priority for network-scripts #1212
>    https://github.com/canonical/cloud-init/pull/1212

What is being done in this pull request seems incorrect to me. If the system operator creates /etc/NetworkManager/system-connections/ens192.nmconnection, the cloud-init shouldn't just beat it with a higher autoconnect priority. Quite the contrary, it makes every sense for it to extend/override configuration applied on cloud instance bootstrap.

The question here is, why does ens192.nmconnection exists if it is not to be activated?

Does the installer create it? Is there some tooling specific to ESXi involved?

(In reply to Lubomir Rintel from comment #19)
> (In reply to Eduardo Otubo from comment #17)
> > This issue is being discussed in 3 different PR: 
> > 
> >  - Set the highest autoconnect priority for network-scripts #1212
> >    https://github.com/canonical/cloud-init/pull/1212
> 
> What is being done in this pull request seems incorrect to me. If the system
> operator creates /etc/NetworkManager/system-connections/ens192.nmconnection,
> the cloud-init shouldn't just beat it with a higher autoconnect priority.
> Quite the contrary, it makes every sense for it to extend/override
> configuration applied on cloud instance bootstrap.
> 
> The question here is, why does ens192.nmconnection exists if it is not to be
> activated?
> 
> Does the installer create it? Is there some tooling specific to ESXi
> involved?

No, ESXi is not involved here. According to my experiment with RHEL9 Beta image, this issue doesn't reproduce on RHEL9 VM installed from RHEL9 Beta ISO step by step follows installation wizard, this issue reproduces on RHEL9 VM installed by kickstart. The reason is there is 'autoconnect-priority=-999' set in keyfile on RHEL9 VM installed from ISO, there is no such setting in keyfile installed by kickstart, so 'autoconnect-priority' is default to 0.
The /etc/NetworkManager/system-connections/ens192.nmconnection exists on both cases, I think it's created by RHEL9 itself. The 'autoconnect-priority=-999' looks like a backward compatible setting to me, since once network-scripts file created, it will have default 'autoconnect-priority' 0, then network-scripts will be activated since 0>-999.  The idea of PR #1212 came from this point.

For cloud-init, I agree that PR #1224 is the best choice for RHEL9 or Fedora33 which prefer keyfile by default.

(In reply to Pengpeng Sun from comment #20)
> (In reply to Lubomir Rintel from comment #19)
> > (In reply to Eduardo Otubo from comment #17)
> > > This issue is being discussed in 3 different PR: 
> > > 
> > >  - Set the highest autoconnect priority for network-scripts #1212
> > >    https://github.com/canonical/cloud-init/pull/1212
> > 
> > What is being done in this pull request seems incorrect to me. If the system
> > operator creates /etc/NetworkManager/system-connections/ens192.nmconnection,
> > the cloud-init shouldn't just beat it with a higher autoconnect priority.
> > Quite the contrary, it makes every sense for it to extend/override
> > configuration applied on cloud instance bootstrap.
> > 
> > The question here is, why does ens192.nmconnection exists if it is not to be
> > activated?
> > 
> > Does the installer create it? Is there some tooling specific to ESXi
> > involved?
> 
> No, ESXi is not involved here. According to my experiment with RHEL9 Beta
> image, this issue doesn't reproduce on RHEL9 VM installed from RHEL9 Beta
> ISO step by step follows installation wizard, this issue reproduces on RHEL9
> VM installed by kickstart. The reason is there is
> 'autoconnect-priority=-999' set in keyfile on RHEL9 VM installed from ISO,
> there is no such setting in keyfile installed by kickstart, so
> 'autoconnect-priority' is default to 0.
> The /etc/NetworkManager/system-connections/ens192.nmconnection exists on
> both cases, I think it's created by RHEL9 itself. The
> 'autoconnect-priority=-999' looks like a backward compatible setting to me,
> since once network-scripts file created, it will have default
> 'autoconnect-priority' 0, then network-scripts will be activated since
> 0>-999.  The idea of PR #1212 came from this point.
> 
> For cloud-init, I agree that PR #1224 is the best choice for RHEL9 or
> Fedora33 which prefer keyfile by default.

One concern on PR #1224, will RHEL8.x cloud-init be aligned with upstream, will PR #1223 create problem on RHEL8 since keyfile is not default choice there.

So the real problem is that installation with kickstart creates /etc/NetworkManager/system-connections/ens192.nmconnection when it shouldn't, right?

I'm wondering if you could share the .ks file you're using?

In my understanding, from RHEL9 /etc/NetworkManager/system-connections/ens192.nmconnection but /etc/sysconfig/network-scripts/ifcfg-ens192 file will be created, no matter how the OS is installed.

For my testing, Ansible helps to deploy the OS, here is the network related commands:
# Network information
network --bootproto=dhcp --ipv6=auto
network --hostname=localhost.localdomain

The whole ks.cfg file is https://github.com/vmware/ansible-vsphere-gos-validation/blob/main/autoinstall/RHEL/8/server_with_GUI/ks.cfg

Thanks for the response. Quick check of the installer source suggests that you're indeed correct about the installer always copying the configuration used for installation into the target system.

I suggest you wipe /etc/NetworkManager/system-connections/ clean in a %post scriptlet.

Thanks Lubomir for the confirmation. We will face issue for customer who want to customize a RHEL9 VM in vSphere. Let me explain this with below points:
1. When customizing a Linux VM, there are 2 engines can be used. One is cloud-init, the other is perl scripts.
2. Now both cloud-init and perl scripts write network configuration to network-scripts(ex: /etc/sysconfig/network-scripts/ifcfg-ens192), if customer's RHEL9 VM has keyfile exists, network-scripts won't be activated. Customer will find the expected network settings do NOT take effect after customization.
3. With PR #1224, cloud-init will write network configuration to keyfile directly which will fix the cloud-init side problem. One concern is for customer who uses RHEL8.x, and RHEL8.x's cloud-init picks up PR #1224, if cloud-init also writes network configuration to keyfile, will it take effect? Any backward compatibility issue for PR#1224?

Thanks Pengpeng for the quick response. 
QE's test results are same with Pengpeng's comment 20 and comment 23(although the kickstart is different).

(In reply to Huijuan Zhao from comment #26)
> Thanks Pengpeng for the quick response. 
> QE's test results are same with Pengpeng's comment 20 and comment
> 23(although the kickstart is different).

Did you guys run tests on other platforms as well? I'm asking because we might want to apply this fix downstream-only (Pengpeng's fix) - since it's a major blocker - and leave the NetworkManager support (Lubomir's fix) for the next release and we'll have more time to have a stable fix.

I believe it is safe to have this fix for the other platforms as well.
As soon as I have a green from QA I'll send a Merge Request.

Thanks!

(In reply to Eduardo Otubo from comment #27)
> (In reply to Huijuan Zhao from comment #26)
> > Thanks Pengpeng for the quick response. 
> > QE's test results are same with Pengpeng's comment 20 and comment
> > 23(although the kickstart is different).
> 
> Did you guys run tests on other platforms as well? I'm asking because we
> might want to apply this fix downstream-only (Pengpeng's fix) - since it's a
> major blocker - and leave the NetworkManager support (Lubomir's fix) for the
> next release and we'll have more time to have a stable fix.
> 
> I believe it is safe to have this fix for the other platforms as well.
> As soon as I have a green from QA I'll send a Merge Request.
> 
> Thanks!

Currently we covered tests in AWS/Azure/OpenStack which install rhel-9 via image, there is only ifcfg-rh file after rhel-9 installation, so did not meet this issue.
Seems only when install rhel-9 via ISO+kickstart can meet the issue per the tests above.

Thanks!

(In reply to Eduardo Otubo from comment #27)
> (In reply to Huijuan Zhao from comment #26)
> > Thanks Pengpeng for the quick response. 
> > QE's test results are same with Pengpeng's comment 20 and comment
> > 23(although the kickstart is different).
> 
> Did you guys run tests on other platforms as well? I'm asking because we
> might want to apply this fix downstream-only (Pengpeng's fix) - since it's a
> major blocker - and leave the NetworkManager support (Lubomir's fix) for the
> next release and we'll have more time to have a stable fix.

This makes sense. The proper way to address this is for the images to *never* contain network configuration and only create it when necessary. That way there will be no conflict. I don't understand why is not done already.

Nevertheless, if the Pengpeng's fix addresses this for now, it's perfectly okay to apply it.

Move to VERIFIED per comment 34

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: cloud-init), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2308