Created attachment 1729478 [details] logs Created attachment 1729478 [details] logs Description of problem: RHV-host gets a different IPv6 address on each reprovision This is because the management network NIC is configured with 'ipv6.addr-gen-mode'='stable-privacy' instead of 'eui64' Version-Release number of selected component (if applicable): rhvh-4.4.3.1-0.20201112.0+1 How reproducible: 100% Steps to Reproduce: 1. provision host with RHVH image 2. Verify ipv6-address is equal on each provision 3. Actual results: Got different IPv6-address on each provision Expected results: Get the same IPv6-address on each provision Additional info: 1. Note: after provisioning of RHEL-8.3 image the default configured of this key is 'eui64' 2. This is a regression issue because we use dual resolving for the hosts meaning the host IPv6 address is kept on the DNS and it resolved when trying to access the host via IPv6 (RHEL default connection is IPv6 when host has dual resolving) 3. See attached logs + nmcli output of the RHVH image and also of RHEL
Dominik can you please have a look?
(In reply to Sandro Bonazzola from comment #4) > Dominik can you please have a look? Looks like the host was not yet added to oVirt, so this seems to be the default behavior of RHEL 8. If this is not the desired behavior, we could discuss with NetworkManager and Anaconda to change the default
(In reply to Roni from comment #0) > > RHV-host gets a different IPv6 address on each boot > > This is because the management network NIC is configured with > 'ipv6.addr-gen-mode'='stable-privacy' instead of 'eui64' > > Additional info: > 1. Note: after provisioning of RHEL-8.3 image the default configured of this > key is 'eui64' Does this indicate that the behavior of RHVH differentiates from RHEL?
Hi(In reply to Dominik Holler from comment #6) > (In reply to Roni from comment #0) > > > > RHV-host gets a different IPv6 address on each boot > > > > This is because the management network NIC is configured with > > 'ipv6.addr-gen-mode'='stable-privacy' instead of 'eui64' > > > > > Additional info: > > 1. Note: after provisioning of RHEL-8.3 image the default configured of this > > key is 'eui64' > > Does this indicate that the behavior of RHVH differentiates from RHEL? Yes, we didn't encounter the issue with RHEL, please see attached nmcli dumps
(In reply to Roni from comment #7) > Hi(In reply to Dominik Holler from comment #6) > > (In reply to Roni from comment #0) > > > > > > RHV-host gets a different IPv6 address on each boot > > > > > > This is because the management network NIC is configured with > > > 'ipv6.addr-gen-mode'='stable-privacy' instead of 'eui64' > > > > > > > > Additional info: > > > 1. Note: after provisioning of RHEL-8.3 image the default configured of this > > > key is 'eui64' > > > > Does this indicate that the behavior of RHVH differentiates from RHEL? > > Yes, we didn't encounter the issue with RHEL, please see attached nmcli dumps Does the difference reproduces outside QE infrastructure?
I tested the integration of RHVH Networking/IPv6 on cockpit and did not encounter this issue. But, in RHVH-4.4.3.1, the NIC is indeed configured as 'ipv6.addr-gen-mode'='stable-privacy' Version-Release number of selected component (if applicable): rhvh-4.4.3.1-0.20201112.0+1 Test Steps: 1. Install RHVH-4.4-20201112.0-RHVH-x86_64-dvd1.iso and choose to open one public NIC(IPv4 and IPv6 are the default modes) 2. Login to cockpit 3. Check the IPv6 address of the public NIC 4. Reboot the system and check whether the IPv6 address is the same as before Actual results: Got the same IPv6 address on each boot Additional info: In our laboratory, a DHCP server is set up to assign IPv6 addresses to VLANs. During the test, I did not find any difference in IPv6 addresses of the VLAN before and after rebooting. But as you can see, the default configuration of NIC is indeed 'ipv6.addr-gen-mode'='stable-privacy' ~~~~~~ # nmcli c show eno3.50 ...... ipv6.addresses: -- ipv6.gateway: -- ipv6.routes: -- ipv6.route-metric: -1 ipv6.route-table: 0 (unspec) ipv6.routing-rules: -- ipv6.ignore-auto-routes: no ipv6.ignore-auto-dns: no ipv6.never-default: no ipv6.may-fail: yes ipv6.ip6-privacy: -1 (unknown) ipv6.addr-gen-mode: stable-privacy ipv6.ra-timeout: 0 (default) ipv6.dhcp-duid: -- ipv6.dhcp-iaid: -- ipv6.dhcp-timeout: 0 (default) ipv6.dhcp-send-hostname: yes ipv6.dhcp-hostname: -- ipv6.dhcp-hostname-flags: 0x0 (none) ipv6.token: -- vlan.parent: eno3 vlan.id: 50 vlan.flags: 1 (REORDER_HEADERS) ...... ~~~~~~ I didn’t find this issue. Could you please give me some details and does my test steps or test environment need to be improved? Thank you.
Actually, there are two issues: First issue: ------------ 'ipv6.addr-gen-mode' configured to 'stable-privacy' instead of 'eui64' with the RHVH image only, this is 100% reproducible. We know that with old RHVH versions the IPv6 address was generated from the MAC address because we save this address at the DNS server. So we can assume that it was configured with 'eui64' mode. But now the address is changed on each provision because the 'stable-privacy' mode generating the IPv6 address based on the '/etc/machine-id' Second issue: ------------- When 'ipv6.addr-gen-mode' configured to 'stable-privacy' then IPv6 address changed on each boot. This issue indeed does not reproduce again, I need to keep tracking and get some proof. But until then I will change the tile: "RHVH does not keep the same IPv6 address on each reprovision"
(In reply to Roni from comment #11) > Actually, there are two issues: > > First issue: > ------------ > 'ipv6.addr-gen-mode' configured to 'stable-privacy' instead of 'eui64' with > the RHVH image only, this is 100% reproducible. > We know that with old RHVH versions the IPv6 address was generated from the > MAC address because we save this address at the DNS server. > So we can assume that it was configured with 'eui64' mode. > But now the address is changed on each provision because the > 'stable-privacy' mode generating the IPv6 address based on the > '/etc/machine-id' > > Second issue: > ------------- > When 'ipv6.addr-gen-mode' configured to 'stable-privacy' then IPv6 address > changed on each boot. > This issue indeed does not reproduce again, I need to keep tracking and get > some proof. > But until then I will change the tile: "RHVH does not keep the same IPv6 > address on each reprovision" I investigated and debugged it a bit. The change, as far as we're concerned happened in RHEL 8.x. The specific change itself is due to the modification in NetworkManager, that modified the default setting back in 2015: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/e9dfdfe9fe586f3fcfaebbf8d6a786b6f6bec03a I got the same configuration in both CentOS 8.2 and 8.3, so unless I'm missing something here, it's doesn't look like something that got suddenly changed in the last version of RHV-H. What is the version of RHV-H that worked for you, and configured the interfaces in a EUI64 mode?
Hi Lev, I mention it at the Additional info: copy it again... "1. Note: after provisioning of RHEL-8.3 image the default configured of this key is 'eui64'"
Hi Lev, With the 'rhvh-4.3.13.2-0.20210127' image, 'ipv6.addr-gen-mode' is 'eui64' (as expected) Note that we have different Foreman script for rhvh-4.3 and rhvh-4.4 (see attached) but I don't see something that can affect this value, so I guess it's related to the image
https://developer.gnome.org/NetworkManager/stable/settings-ipv6.html I found the following text here ^ "On D-Bus, the absence of an addr-gen-mode setting equals enabling stable-privacy. For keyfile plugin, the absence of the setting on disk means EUI64 so that the property doesn't change on upgrade from older versions" maybe it can explain why sometimes we see 'stable-privacy' and sometimes 'EUI64'
Since Dominik asked me to comment... :) - the profile's setting for ipv6.addr-gen-mode may be eui64 or stable-privacy. Which one it is, depends on how you create the profile! E.g. if you create a ifcfg file and don't `IPV6_ADDR_GEN_MODE=stable-privacy`, then it's eui64. But *it depends* and if you see that a profile has the undesired setting (`nmcli connection show $PROFILE`), then check how the profile was created and create it accordingly. - if stable-privacy is selected, then this happens by hashing both `/var/lib/NetworkManager/secret_key` and `/etc/machine-id`. If you want that across multiple boots the same identifier is used, then these files must be preserved. You can wipe /var/lib/NetworkManager/secret_key, then NetworkManager will generate a new one (and persist it). For example, if you clone a VM, then usually you'd want that the clone uses a different IP address. Hence you would wipe /etc/machine-id (and let it get regenerated). You don't also need to wipe `/var/lib/NetworkManager/secret_key`, because if you change /etc/machine-id, then that suffices to change the generated addresses. But it wouldn't hurt either. In short: the machine-id and the secret-key are together the ID of the host. You need to be clear about whether the machine is the same or not. And consequently the files `/var/lib/NetworkManager/secret_key` and `/etc/machine-id` must be preserved or wiped/regenerated. - even if you select ipv6.addr-gen-mode=eui64, then there might still be other identifiers that depend on the machine id. For example, the settings ethernet.cloned-mac-address=stable, ipv4.dhcp-client-id=stable, ipv6.dhcp-duid=stable-{llt,ll,uuid}, ipv6.dhcp-iaid=stable all depend on the machine-id.
I think that https://bugzilla.redhat.com/show_bug.cgi?id=1873021#c9 describes the difference in the results.
Ok so we need bug #1873021 to be fixed in order to get this one fixed too.
Verified on redhat-virtualization-host-4.4.6-20210409.0.el8_4 Test Steps: 1. Install RHVH-4.4-20210413.0-RHVH-x86_64-dvd1.iso and choose to open one public NIC(IPv4 and IPv6 are the default modes) 2. Login to cockpit 3. Check the IPv6 address of the public NIC 4. Reboot the system and check whether the IPv6 address is the same as before 5. Check the default configuration of NIC Actual results: Got the same IPv6 address on each boot and IPV6_ADDR_GEN_MODE=eui64 in sysconfig file for default routing interface ~~~~~~ # nmcli c show eno1 connection.id: eno1 connection.uuid: 4f35e7fb-7c94-42f2-a434-046946009e3d connection.stable-id: -- connection.type: 802-3-ethernet connection.interface-name: eno1 connection.autoconnect: yes connection.autoconnect-priority: 0 connection.autoconnect-retries: -1 (default) connection.multi-connect: 0 (default) connection.auth-retries: -1 connection.timestamp: 1618797291 connection.read-only: no connection.permissions: -- connection.zone: -- connection.master: -- connection.slave-type: -- connection.autoconnect-slaves: -1 (default) connection.secondaries: -- connection.gateway-ping-timeout: 0 connection.metered: unknown connection.lldp: default connection.mdns: -1 (default) connection.llmnr: -1 (default) connection.wait-device-timeout: -1 802-3-ethernet.port: -- 802-3-ethernet.speed: 0 802-3-ethernet.duplex: -- 802-3-ethernet.auto-negotiate: no 802-3-ethernet.mac-address: -- 802-3-ethernet.cloned-mac-address: -- 802-3-ethernet.generate-mac-address-mask:-- 802-3-ethernet.mac-address-blacklist: -- 802-3-ethernet.mtu: auto 802-3-ethernet.s390-subchannels: -- 802-3-ethernet.s390-nettype: -- 802-3-ethernet.s390-options: -- 802-3-ethernet.wake-on-lan: default 802-3-ethernet.wake-on-lan-password: -- ipv4.method: auto ipv4.dns: -- ipv4.dns-search: -- ipv4.dns-options: -- ipv4.dns-priority: 0 ipv4.addresses: -- ipv4.gateway: -- ipv4.routes: -- ipv4.route-metric: -1 ipv4.route-table: 0 (unspec) ipv4.routing-rules: -- ipv4.ignore-auto-routes: no ipv4.ignore-auto-dns: no ipv4.dhcp-client-id: -- ipv4.dhcp-iaid: -- ipv4.dhcp-timeout: 0 (default) ipv4.dhcp-send-hostname: yes ipv4.dhcp-hostname: -- ipv4.dhcp-fqdn: -- ipv4.dhcp-hostname-flags: 0x0 (none) ipv4.never-default: no ipv4.may-fail: yes ipv4.dad-timeout: -1 (default) ipv4.dhcp-vendor-class-identifier: -- ipv4.dhcp-reject-servers: -- ipv6.method: auto ipv6.dns: -- ipv6.dns-search: -- ipv6.dns-options: -- ipv6.dns-priority: 0 ipv6.addresses: -- ipv6.gateway: -- ipv6.routes: -- ipv6.route-metric: -1 ipv6.route-table: 0 (unspec) ipv6.routing-rules: -- ipv6.ignore-auto-routes: no ipv6.ignore-auto-dns: no ipv6.never-default: no ipv6.may-fail: yes ipv6.ip6-privacy: 0 (disabled) ipv6.addr-gen-mode: eui64 ...... ...... ~~~~~~ So move bug status to "VERIFIED"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Virtualization Host security update [ovirt-4.4.6]), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2239