RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1873021 - Change of eui64 to stable-privacy broke our ipv6 networking
Summary: Change of eui64 to stable-privacy broke our ipv6 networking
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: anaconda
Version: 8.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.4
Assignee: Radek Vykydal
QA Contact: Release Test Team
Sagar Dubewar
URL:
Whiteboard:
: 1940501 (view as bug list)
Depends On:
Blocks: 1897893 1940501
TreeView+ depends on / blocked
 
Reported: 2020-08-27 07:51 UTC by michal novacek
Modified: 2021-09-07 10:09 UTC (History)
11 users (show)

Fixed In Version: anaconda-33.16.4.15-1.el8
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1940501 (view as bug list)
Environment:
Last Closed: 2021-05-18 15:47:06 UTC
Type: Bug
Target Upstream Version:
Embargoed:
anikandr: needinfo+


Attachments (Terms of Use)
ks file after installation (29.60 KB, text/plain)
2020-08-27 07:51 UTC, michal novacek
no flags Details

Description michal novacek 2020-08-27 07:51:08 UTC
Created attachment 1712778 [details]
ks file after installation

Description of problem:
We found out in our lab that ipv6 stopped working about three weeks ago where it previously worked nicely.

We traced the problem to IPV6_ADDR_GEN_MODE=stable-privacy being put into /etc/sysconfig/network-scripts/ifcfg-* files.

I'm not really sure how it got there or whether it was there before this problem appeared.

In our case the solution to the problem is to replace s/stable-privacy/eui64/g in /etc/sysconfig/network*.


Version-Release number of selected component (if applicable):
RHEL8.3
NetworkManager-1.26.0-6.el8.x86_64

How reproducible: always in our environment

Steps to Reproduce:
1. install virt in clusterqe beaker 
2. observe IPV6_ADDR_GEN_MODE=stable-privacy in sysconfig file for default routing interface

Actual results:
stable-privacy used for ipv6 local address generation without easy way to
specify it at the time of installation (ks file)

Expected results:
eui64 used as before unless explicitely changed and/or have clear way of
specifying which method do you want to use in kickstart or by global env
variable.


Additional info:

This breaks ipv6 networks relying on eui64 ipv6 addresses. I'm putting this on anaconda as it seems to be the component talking to network manager (which might be the actual component changing the behaviour).

Comment 1 Radek Vykydal 2020-08-27 08:32:27 UTC
Could you please attach log from the installation
/var/log/anaconda/journal.log

Ideally also one from installation where the problem does not occur.

Comment 2 Radek Vykydal 2020-08-27 10:09:06 UTC
Anaconda is not setting the value of addr-gen-mode explicitly anywhere.

There has been a change between RHEL 8.2 and RHEL 8.3 that could be related - the ifcfg file created for initramfs connection contains IPV6_ADDR_GEN_MODE=stable-privacy in 8.3 (it is created in Anaconda, using NM.SimpleConnection and setting ipv6.method to auto) while in RHEL 8.2 where the ifcfg file is created already in initramfs by dracut - it does not contain the value set explicitly.

It seems that when there is no IPV6_ADDR_GEN_MODE setting in ifcfg file, the connection gets eui64 default, but when the connection is created without setting the value it has stable-privacy set. Is it expected ?

Or should we set the value explicitly when creating the default profile?

Was there any recent change in NM in this area that could explain the BZ issue (from the Description: "worked about three weeks ago" but we do not have exact versions) ?

Comment 3 Thomas Haller 2020-08-27 10:40:13 UTC
>  It seems that when there is no IPV6_ADDR_GEN_MODE setting in ifcfg file, the connection gets eui64 default, but when the connection is created without setting the value it has stable-privacy set. Is it expected?

Yes. NetworkManager defaults to addr-gen-mode=stable-privacy (if you create a profile with nmcli/D-Bus/libnm that doesn't set it otherwise). If you want eui64, you need to explicitly set it (if you create a profile with nmcli/D-Bus/libnm).

However, for the disk formats (ifcfg-rh and keyfile), an unspecified addr-gen-mode means eui64 (to preserve previous behavior from before addr-gen-mode property was introduced).


> Was there any recent change in NM in this area that could explain the BZ issue (from the Description: "worked about three weeks ago" but we do not have exact versions) ?

I don't think that anything changed in this regard for a while. But of course, there is a change on rhel-8.3, since now NM is running in initrd.



Note that "addr-gen-mode=stable-privacy" requires a stable /etc/machine-id and /var/lib/NetworkManager/secret_key. Because it generates an address by hashing these files. It's not clear to me that these files are the same for initrd as they will be on real-root. If they are not the same, it makes little sense to even do "addr-gen-mode=stable-privacy" in initrd.

The nm-initrd-generator already should create all profiles with addr-gen-mode=eui64. If that's not the case, that would be a bug in NetworkManager.

Comment 4 Radek Vykydal 2020-08-27 12:40:12 UTC
https://github.com/rhinstaller/anaconda/pull/2813

Comment 5 Radek Vykydal 2020-08-27 12:43:44 UTC
Here is updates image for latest RHEL 8.3 compose with the patch from comment #4.

It can be applied either by using boot option

inst.udpates=https://rvykydal.fedorapeople.org/updates.rhbz1873021_eui64.img

or by using kickstart command

updates https://rvykydal.fedorapeople.org/updates.rhbz1873021_eui64.img

Comment 6 Jan Stodola 2020-08-31 13:27:37 UTC
Michal, can you please confirm that the updates image from comment 5 fixes the problem for you?

Comment 7 Jan Stodola 2020-09-01 20:20:43 UTC
I was able to reproduce it by running manual and kickstart installations, which activate network in initrd, so for example inst.repo=http:// ... or ks=http://... on the kernel command line.

With updates image from comment 5 applied, IPV6_ADDR_GEN_MODE is no longer present in the ifcfg file and eui64 mode is used on the installed system as expected:

[root@localhost ~]# grep IPV6_ADDR_GEN_MODE /etc/sysconfig/network-scripts/ifcfg-enp1s0
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:68:47:23 brd ff:ff:ff:ff:ff:ff
    inet 10.43.136.122/24 brd 10.43.136.255 scope global dynamic noprefixroute enp1s0
       valid_lft 3542sec preferred_lft 3542sec
    inet6 2620:52:0:2b88:5054:ff:fe68:4723/64 scope global dynamic noprefixroute 
       valid_lft 2591945sec preferred_lft 604745sec
    inet6 fe80::5054:ff:fe68:4723/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost ~]#

Radek, can you please confirm that the missing IPV6_ADDR_GEN_MODE line in ifcfg file is intended? Based on the patch I thought that IPV6_ADDR_GEN_MODE=EUI64 would be present in ifcfg files.

Note there is a typo in inst.udpates=... in comment 5, it should be:
inst.updates=https://rvykydal.fedorapeople.org/updates.rhbz1873021_eui64.img

Comment 8 Radek Vykydal 2020-09-02 08:31:14 UTC
(In reply to Jan Stodola from comment #7)

> 
> Radek, can you please confirm that the missing IPV6_ADDR_GEN_MODE line in
> ifcfg file is intended? Based on the patch I thought that
> IPV6_ADDR_GEN_MODE=EUI64 would be present in ifcfg files.

It is decision of NM if it dumps the value in the config file or not explicitly (we are creating it via libnm API), no value means eui64 for historical reasons (comment #3).

I am going to update the BZ with differences between 8.2 and 8.3 for various configuration methods to see how big / serious the change is.

Comment 9 Radek Vykydal 2020-09-02 09:30:09 UTC
So here are my observations for various configuration methods:
X   - no IPV6_ADDR_GEN_MODE in config file (which means eui64 for NM)
s-p - IPV6_ADDR_GEN_MODE=stable-privacy in config file

1) ip=ens3:dhcp - boot option with device specified
  - 8.2            : X
  - 8.3            : X
  - 8.3 with patch : X
2) ip=dhcp (or no ip= boot option at all  but network required/activated in initramfs) - boot option for all devices this - case of this BZ ?
  - 8.2            : X
  - 8.3            : s-p
  - 8.3 with patch : X 
3) kickstart (ie applied by NM from ifcfg created by anaconda in initramfs)
  - 8.2            : X
  - 8.3            : X
  - 8.3 with patch : X 
4) kickstart with network command created in %pre section (ie applied by anaconda after switch root) - less used than 3)
  - 8.2            : s-p
  - 8.3            : s-p
  - 8.3 with patch : X
5) default ifcfg file for iface which was not configured by kickstart or boot options
  - 8.2            : s-p
  - 8.3            : s-p
  - 8.3 with patch : X
   - 5a) it will be passed to installed system (with ONBOOT=no though so not used)
   - it will be used for configuration in UI:
     - 5b) in GUI the default can be modified
     - 5c) in TUI we don't have an option to modify the default, moreover
       it will be always s-p, even for 8.3 with patch - which should be maybe fixed as well

I am working with the assumption that eui64 is correct default for installer. According to comment #3 it should be true for 2) where the patch fixes a regression.
It fixes also 4) and 5) though changing the behaviour compared to 8.2. I am not quite sure about 5b) - config in GUI defaults to eui64 with the patch. I'll ask NM / Thomas Haller for guidance.

Comment 10 Radek Vykydal 2020-09-02 09:33:24 UTC
(In reply to Radek Vykydal from comment #9)
 
> I am working with the assumption that eui64 is correct default for
> installer. 

Thomas could you please advise what should be the defaults for the installer wrt IPV6_ADDR_GEN_MODE ?
For the cases listed in comment #9

> According to comment #3 it should be true for 2) where the patch
> fixes a regression.
> It fixes also 4) and 5) though changing the behaviour compared to 8.2. I am
> not quite sure about 5b) - config in GUI defaults to eui64 with the patch.
> I'll ask NM / Thomas Haller for guidance.

Comment 12 michal novacek 2020-09-02 14:19:45 UTC
With the patch from comment 5 there are no IPV6_ADDR_GEN_MODE lines generated to /etc/sysconfig/network-scripts/* files and ipv6 address generated is the eui64 based instead of being stable-privacy based.

Comment 13 Thomas Haller 2020-09-03 12:18:37 UTC
the main problem with stable-privacy is that it depends on /etc/machine-id and /var/lib/NetworkManager/secret_key. That means, if you configure networking in initrd, and those files are not the same as they will be during real-root, then that is wrong.

In general, it is desirable that (a) during installer time and (b) during initrd network boot, these files are the same as during the installed system/real-root. If that's the case, then "stable-privacy" makes some sense. Possibly, the installer anyway should ensure that (by ensuring that /etc/machine-id and /var/lib/NetworkManager/secret_key get freshly generated, but copied over to the installed system).

Still, in the past the default was eui64. And it's also what people might expect (they get the well known host identifier). So yes, I think at least for now, in all cases we want to set ipv6.addr-gen-mode=eui64 -- both for profiles generated during installation and during network-boot/initrd.

Comment 14 Radek Vykydal 2020-09-10 10:58:41 UTC
(In reply to Radek Vykydal from comment #9)

>      - 5c) in TUI we don't have an option to modify the default, moreover
>        it will be always s-p, even for 8.3 with patch - which should be
> maybe fixed as well

Upstream fix: https://github.com/rhinstaller/anaconda/pull/2845

Comment 15 Radek Vykydal 2020-09-10 11:11:25 UTC
Upstream PR changing the default to eui64: https://github.com/rhinstaller/anaconda/pull/2846

Comment 17 Radek Vykydal 2020-11-18 12:23:18 UTC
Fixing this BZ would fix also RHV bug 1897893.

Comment 18 Dominik Holler 2021-02-04 13:17:26 UTC
Which version of anaconda is expected in RHEL 8.4? anaconda-34.5 which contains https://github.com/rhinstaller/anaconda/commit/5b833ac3beac88b38ff378ae4877dfe027d65505 ?

Comment 19 Radek Vykydal 2021-02-16 09:30:08 UTC
(In reply to Dominik Holler from comment #18)
> Which version of anaconda is expected in RHEL 8.4? anaconda-34.5 which
> contains
> https://github.com/rhinstaller/anaconda/commit/
> 5b833ac3beac88b38ff378ae4877dfe027d65505 ?

anaconda-33.16.4.X (https://github.com/rhinstaller/anaconda/tree/rhel-8)
The BZ has not been approved for 8.4.
Do you expect this BZ to be fixed in RHEL 8.4 ?

Comment 20 Sandro Bonazzola 2021-03-10 14:17:28 UTC
(In reply to Radek Vykydal from comment #19)
> (In reply to Dominik Holler from comment #18)
> > Which version of anaconda is expected in RHEL 8.4? anaconda-34.5 which
> > contains
> > https://github.com/rhinstaller/anaconda/commit/
> > 5b833ac3beac88b38ff378ae4877dfe027d65505 ?
> 
> anaconda-33.16.4.X (https://github.com/rhinstaller/anaconda/tree/rhel-8)
> The BZ has not been approved for 8.4.
> Do you expect this BZ to be fixed in RHEL 8.4 ?

It would be nice to have in 8.4 yes.

Comment 21 Radek Vykydal 2021-03-15 12:18:40 UTC
(In reply to Sandro Bonazzola from comment #20)
> (In reply to Radek Vykydal from comment #19)
> > (In reply to Dominik Holler from comment #18)
> > > Which version of anaconda is expected in RHEL 8.4? anaconda-34.5 which
> > > contains
> > > https://github.com/rhinstaller/anaconda/commit/
> > > 5b833ac3beac88b38ff378ae4877dfe027d65505 ?
> > 
> > anaconda-33.16.4.X (https://github.com/rhinstaller/anaconda/tree/rhel-8)
> > The BZ has not been approved for 8.4.
> > Do you expect this BZ to be fixed in RHEL 8.4 ?
> 
> It would be nice to have in 8.4 yes.

I am afraid that at this point you have to request an exception for 8.4.

Comment 30 Radek Vykydal 2021-03-23 06:30:24 UTC
PR with kickstart tests: https://github.com/rhinstaller/kickstart-tests/pull/496

Comment 31 Radek Vykydal 2021-03-23 06:42:45 UTC
PR with fix: https://github.com/rhinstaller/anaconda/pull/3257

Comment 37 Radek Vykydal 2021-03-30 15:43:04 UTC
I am adding a draft of the Doc Text.

I am not 100% sure though if we should release it as an installer note given the issue have been reported externally on RHEVM product originally (bug 1897893). It is possible that for RHEL installer there is only this (internal ?) report. It was a temporary change of behavior in RHEL 8.3 which was not mentioned in Known Issues. Nevertheless I am inclined to include it in RHEL 8.4 installer release notes to be transparent.

Comment 39 Radek Vykydal 2021-04-14 06:00:12 UTC
*** Bug 1940501 has been marked as a duplicate of this bug. ***

Comment 41 errata-xmlrpc 2021-05-18 15:47:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (anaconda bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1844


Note You need to log in before you can comment on or make changes to this bug.