Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2098501

Summary: [RHEL-9.1] IPv6 not workable when cloud-init configure network using NM keyfiles
Product: Red Hat Enterprise Linux 9 Reporter: xiachen
Component: cloud-initAssignee: Emanuele Giuseppe Esposito <eesposit>
Status: CLOSED ERRATA QA Contact: xiachen
Severity: high Docs Contact:
Priority: high    
Version: 9.1CC: eesposit, eterrell, huzhao, jgreguske, lrintel, xiachen, xiliang, yacao, yuxisun
Target Milestone: rcKeywords: Regression, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: cloud-init-22.1-5.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2098624 (view as bug list) Environment:
Last Closed: 2022-11-15 10:13:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2056964    
Bug Blocks:    

Description xiachen 2022-06-20 02:08:17 UTC
Description of problem:

This problem is found after below bug fix, ipv6 is not workable after launching a new instance.

Bug 2056964 - [RHEL-9]Rebase cloud-init from Fedora so it can configure networking using NM keyfiles

we can see log "Not bringing up newly configured network interfaces" in /var/log/cloud-init.log

2022-06-17 02:15:22,191 - __init__.py[DEBUG]: Selected renderer 'network-manager' from priority list: ['netplan', 'network-manager', 'networkd', 'sysconfig', 'eni']
2022-06-17 02:15:22,192 - util.py[DEBUG]: Writing to /etc/NetworkManager/system-connections/cloud-init-eth0.nmconnection - wb: [600] 376 bytes
2022-06-17 02:15:22,193 - util.py[DEBUG]: Restoring selinux mode for /etc/NetworkManager/system-connections/cloud-init-eth0.nmconnection (recursive=False)
2022-06-17 02:15:22,194 - util.py[DEBUG]: Restoring selinux mode for /etc/NetworkManager/system-connections/cloud-init-eth0.nmconnection (recursive=False)
2022-06-17 02:15:22,194 - __init__.py[DEBUG]: Not bringing up newly configured network interfaces


On Azure, ping6 gets Network is unreachable.

$ ping6 ace:cab:deca::fe -c 1
ping6: connect: Network is unreachable

# ip -6 route
::1 dev lo proto kernel metric 256 pref medium
ace:cab:deca::4 dev eth0 proto kernel metric 100 pref medium
fe80::/64 dev eth0 proto kernel metric 1024 pref medium


cat /etc/NetworkManager/system-connections/cloud-init-eth0.nmconnection
# Generated by cloud-init. Changes will be lost.

[connection]
id=cloud-init eth0
uuid=1dd9a779-d327-56e1-8454-c65e2556c12c
type=ethernet

[user]
org.freedesktop.NetworkManager.origin=cloud-init

[ethernet]
mac-address=00:0D:3A:58:0F:B3

[ipv4]
method=auto
may-fail=false

[ipv6]
method=dhcp
may-fail=false
addr-gen-mode=stable-privacy


On Openstack, ping6 error is a little bit different, but unreachable either.

#  ping6 2620:52:0:88:f816:3eff:fe23:e623 -c 1
PING 2620:52:0:88:f816:3eff:fe23:e623(2620:52:0:88:f816:3eff:fe23:e623) 56 data bytes
From 2620:52:0:88:ef7e:b597:1cc8:32df icmp_seq=1 Destination unreachable: Address unreachable

--- 2620:52:0:88:f816:3eff:fe23:e623 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms


$ ip -6 route
::1 dev lo proto kernel metric 256 pref medium
2620:52:0:88::/64 dev eth0 proto ra metric 100 pref medium
fe80::/64 dev eth0 proto kernel metric 1024 pref medium
default proto static metric 100 pref medium
	nexthop via 2620:52:0:88::3fe dev eth0 weight 1 
	nexthop via fe80::4e16:fc01:d8c:98ed dev eth0 weight 1 
	nexthop via fe80::4e16:fc01:d83:88aa dev eth0 weight 1 

cat /etc/NetworkManager/system-connections/cloud-init-eth0.nmconnection
# Generated by cloud-init. Changes will be lost.

[connection]
id=cloud-init eth0
uuid=1dd9a779-d327-56e1-8454-c65e2556c12c
type=ethernet

[user]
org.freedesktop.NetworkManager.origin=cloud-init

[ethernet]
mtu=1500
mac-address=FA:16:3E:CE:97:C9

[ipv4]
method=auto
may-fail=false

[ipv6]
method=auto
may-fail=false
addr-gen-mode=stable-privacy
route1=::/0,2620:52:0:88::3fe


Version-Release number of selected component (if applicable):
cloud-init-22.1-2.el9 +


Expected results:
IPv6 connection should be workable after first boot.

Comment 1 xiachen 2022-06-21 08:36:01 UTC
I Modified the keyfile "/etc/NetworkManager/system-connections/cloud-init-eth0.nmconnection",

[ipv6]
method=auto
may-fail=false

and then run below nmcli commands, IPv6 connection both works on OpenStack and Azure.
nmcli conn reload
nmcli conn up "cloud-init eth0"


that changes are,
on OpenStack VM instance,
removing the line addr-gen-mode=stable-privacy

on Azure VM instance,
removing the line addr-gen-mode=stable-privacy, and changing ipv6 method=auto

Comment 2 Emanuele Giuseppe Esposito 2022-06-21 11:32:53 UTC
@Ludomir I would like to ask your opinion here: I see from logs that you introduced cloudinit/net/network_manager.py, is there a specific reason on why we have add-gen-mode set?
It seems that other renderers like sysconfig do not set it.
Also not sure why on Azure ipv6 method is dhcp and not auto.

Thank you,
Emanuele

Comment 3 xiachen 2022-06-24 03:30:19 UTC
AWS instance has the same issue with Azure.

/etc/NetworkManager/system-connections/cloud-init-eth0.nmconnection
# Generated by cloud-init. Changes will be lost.
...
[ipv6]
method=dhcp
may-fail=false
addr-gen-mode=stable-privacy


# ping6 2001:4860:4860::8888 -c 1
connect: Network is unreachable


While below configuration IP connection works.
[ipv6] 
method=auto
may-fail=false

# ping6 2001:4860:4860::8888 -c 1
PING 2001:4860:4860::8888(2001:4860:4860::8888) 56 data bytes
64 bytes from 2001:4860:4860::8888: icmp_seq=1 ttl=109 time=7.93 ms

--- 2001:4860:4860::8888 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 7.928/7.928/7.928/0.000 ms

Comment 4 Eduardo Otubo 2022-07-14 08:06:04 UTC
I believe the add-gen-mode point is not a blocker and not intimately related to the solution (please correct me if I'm wrong here). We know for a fact that changing method from dhcp to auto fixes the problem. I think we can move forward with this solution.

Comment 5 Eduardo Otubo 2022-07-14 08:26:58 UTC
Raising priority as no network has a big impact. Should be solved on the current release. As this affects 9.1 (after rebase) I assume it also can be reproduced on 8.7. Please clone if necessary and set NEEDINFO on me to set zstream flag.

Thanks!

Comment 6 Eduardo Otubo 2022-07-14 14:06:27 UTC
(In reply to Eduardo Otubo from comment #4)
> I believe the add-gen-mode point is not a blocker and not intimately related
> to the solution (please correct me if I'm wrong here). We know for a fact
> that changing method from dhcp to auto fixes the problem. I think we can
> move forward with this solution.

I'm sorry I misinterpreted the tests. Since no other renderer uses add-gen-mode I assume it's not a crucial option and I believe we can remove it from the configuration from now *as well as* change ipv6.method to auto.

Comment 9 xiachen 2022-07-15 02:31:53 UTC
8.7 bug is BZ#2098624.

I think we don't need zstream fix, because this problem was brought into 22.1 after this bug fix, it does not affect zstream.(Correct me if I'm wrong)
Bug 2056964 - [RHEL-9]Rebase cloud-init from Fedora so it can configure networking using NM keyfiles


(In reply to Eduardo Otubo from comment #5)
> Raising priority as no network has a big impact. Should be solved on the
> current release. As this affects 9.1 (after rebase) I assume it also can be
> reproduced on 8.7. Please clone if necessary and set NEEDINFO on me to set
> zstream flag.
> 
> Thanks!

Comment 10 Eduardo Otubo 2022-07-15 10:01:25 UTC
(In reply to xiachen from comment #9)
> 8.7 bug is BZ#2098624.
> 
> I think we don't need zstream fix, because this problem was brought into
> 22.1 after this bug fix, it does not affect zstream.(Correct me if I'm wrong)
> Bug 2056964 - [RHEL-9]Rebase cloud-init from Fedora so it can configure
> networking using NM keyfiles

Thanks for the info, Amy!

Also, if this solution fixes the problem, can you run some quick test on bz#2104393 with the same package? Wondering if it's the same root cause.

> 
> 
> (In reply to Eduardo Otubo from comment #5)
> > Raising priority as no network has a big impact. Should be solved on the
> > current release. As this affects 9.1 (after rebase) I assume it also can be
> > reproduced on 8.7. Please clone if necessary and set NEEDINFO on me to set
> > zstream flag.
> > 
> > Thanks!

Comment 11 Huijuan Zhao 2022-07-19 02:04:27 UTC
(In reply to Eduardo Otubo from comment #10)
> 
> Also, if this solution fixes the problem, can you run some quick test on
> bz#2104393 with the same package? Wondering if it's the same root cause.
> 

Eduardo, the scratch build in comment 8 does NOT fix bz#2104393, I think the root cause of bz#2104393 is not same with this BZ.

I added the root cause of bz#2104393 in the description: The default NM keyfile is active instead cloud-init generated keyfile, so the configurations in cloud-init-ens192.nmconnection can not take effect. 

Could I know why cloud-init set keyfile name as cloud-init-ens192.nmconnection instead of ens192.nmconnection? Thanks!

# nmcli -f NAME,FILENAME,DBUS-PATH connection show
NAME               FILENAME                                                               DBUS-PATH                                  
ens192             /etc/NetworkManager/system-connections/ens192.nmconnection             /org/freedesktop/NetworkManager/Settings/1 
cloud-init ens192  /etc/NetworkManager/system-connections/cloud-init-ens192.nmconnection  /org/freedesktop/NetworkManager/Settings/2

Comment 13 Eduardo Otubo 2022-07-20 08:09:44 UTC
(In reply to Huijuan Zhao from comment #11)
> (In reply to Eduardo Otubo from comment #10)
> > 
> > Also, if this solution fixes the problem, can you run some quick test on
> > bz#2104393 with the same package? Wondering if it's the same root cause.
> > 
> 
> Eduardo, the scratch build in comment 8 does NOT fix bz#2104393, I think the
> root cause of bz#2104393 is not same with this BZ.
> 
> I added the root cause of bz#2104393 in the description: The default NM
> keyfile is active instead cloud-init generated keyfile, so the
> configurations in cloud-init-ens192.nmconnection can not take effect. 
> 
> Could I know why cloud-init set keyfile name as
> cloud-init-ens192.nmconnection instead of ens192.nmconnection? Thanks!

Thanks for looking into that!
I don't know why that happens but I'll investigate.

> 
> # nmcli -f NAME,FILENAME,DBUS-PATH connection show
> NAME               FILENAME                                                 
> DBUS-PATH                                  
> ens192            
> /etc/NetworkManager/system-connections/ens192.nmconnection            
> /org/freedesktop/NetworkManager/Settings/1 
> cloud-init ens192 
> /etc/NetworkManager/system-connections/cloud-init-ens192.nmconnection 
> /org/freedesktop/NetworkManager/Settings/2

Comment 22 errata-xmlrpc 2022-11-15 10:13:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (cloud-init bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:8056