Bug 1825666

Summary: ipv6 nfs mount point does not mount automatically at boot, remote-fs-pre.target starts too early and conflicts with network start up
Product: Red Hat Enterprise Linux 8 Reporter: Sangam <sababu>
Component: NetworkManagerAssignee: Beniamino Galvani <bgalvani>
Status: CLOSED ERRATA QA Contact: Filip Pokryvka <fpokryvk>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.1CC: acardace, atragler, bgalvani, fge, fpokryvk, jeharris, jmaxwell, lrintel, msekleta, rkhan, rmetrich, sukulkar, systemd-maint-list, thaller, till, vbenes
Target Milestone: rcKeywords: Reopened, Triaged
Target Release: 8.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: NetworkManager-1.26.0-1.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-09 19:28:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1935910    

Description Sangam 2020-04-19 19:23:06 UTC
Description of problem: Unable to mount ipv6 mount point automatically at boot, post system comes up restarting "remote-fs.target" or "mount -a" works and mountpoint gets mounted successfully.

If we introduce a delay of 2 seconds before remote-fs-pre.targets starts to mount, there is no issue observed.

If we  start remote-fs.target at a later stage after NetworkManager-wait-online has finished started up then it works as expected.


Version-Release number of selected component (if applicable):
Linux RHEL8-test2 4.18.0-147.5.1.el8_1.x86_64 #1 SMP Tue Jan 14 15:50:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[root@RHEL8-test2 ~]# rpm -qa | grep -i systemd
rpm-plugin-systemd-inhibit-4.14.2-9.el8.x86_64
pcp-pmda-systemd-4.3.0-3.el8.x86_64
systemd-239-13.el8.x86_64
systemd-container-239-13.el8.x86_64
python3-systemd-234-8.el8.x86_64
systemd-pam-239-13.el8.x86_64
systemd-udev-239-13.el8.x86_64
systemd-libs-239-13.el8.x86_64
oci-systemd-hook-0.1.15-2.git2d0b8a3.module+el8+2769+577ad176.x86_64


How reproducible: This is easily reproducible on RHEL 8 systems.

->  The same nfs mount point works without any issues with RHEL7 systems, mounts automatically at boot.
->  There are no issues with ipv4 mount points on RHEL8.


Steps to Reproduce:
1. Configure nfs server and export a FS with ipv6 ip address
2. Configure nfs client to mount it during boot time.
3. Reboot, check nfs mount point post reboot. If its not mounted then execute "mount -a" or restart "remote-fs.target" .

Actual results: ipv6 nfs mount point does not mount automatically at boot.


Expected results: ipv6 mount point should be mounted automatically.


Additional info: I will share the relevant logs to this BZ

Comment 1 Renaud Métrich 2020-04-20 07:12:46 UTC
Hi,

This is "expected": by default NetworkManager-wait-online.service doesn't wait for IPv6 IP addresses, this is described in nm-online(1) manpage:

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
By default NetworkManager waits for IPv4 dynamic addressing to complete but does not wait for the auto IPv6 dynamic addressing. To wait for IPv6 addressing to complete, either (1) change the network connection's IPv6 may-fail setting to no, and/or (2) change the IPv6 addressing method to manual or dhcp, to indicate that IPv6 connectivity is expected.
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Hence, to wait for IPv6 IP addresses, set IPV6_FAILURE_FATAL=yes in /etc/sysconfig/network-scripts/ifcfg-XXX or use corresponding nmcli property "ipv6.may-fail=no".

See also BZ 1706646 on RHEL 7 and KCS https://access.redhat.com/solutions/4879291

Renaud.

Comment 2 Sangam 2020-04-20 07:39:31 UTC
(In reply to Renaud Métrich from comment #1)
> Hi,
> 
> This is "expected": by default NetworkManager-wait-online.service doesn't
> wait for IPv6 IP addresses, this is described in nm-online(1) manpage:
> 
> -------- 8< ---------------- 8< ---------------- 8< ---------------- 8<
> --------
> By default NetworkManager waits for IPv4 dynamic addressing to complete but
> does not wait for the auto IPv6 dynamic addressing. To wait for IPv6
> addressing to complete, either (1) change the network connection's IPv6
> may-fail setting to no, and/or (2) change the IPv6 addressing method to
> manual or dhcp, to indicate that IPv6 connectivity is expected.
> -------- 8< ---------------- 8< ---------------- 8< ---------------- 8<
> --------
> 
> Hence, to wait for IPv6 IP addresses, set IPV6_FAILURE_FATAL=yes in
> /etc/sysconfig/network-scripts/ifcfg-XXX or use corresponding nmcli property
> "ipv6.may-fail=no".
> 
> See also BZ 1706646 on RHEL 7 and KCS
> https://access.redhat.com/solutions/4879291
> 
> Renaud.

Hello Renaud,

Thank you for your inputs, I was able to mount the ipv6 nfs mount point with that option set. I will close this BZ as "NOTABUG".

Comment 3 Renaud Métrich 2020-04-20 08:00:51 UTC
I'm reopening this because systemd manpages / docs must clearly explain this behaviour.
In particular, when it affects Mount Points: the admin just won't notice it will fail by default, e.g. when a FQDN is used in the /etc/fstab:

my-nfs-server:/exports    /my-export    nfs    defaults    0    0

In the example if a IPv6 IP address is returned by the DNS server for "my-nfs-server", the mount will likely fail.


Hence, something is definitely wrong with the current behaviour.

Comment 4 Michal Sekletar 2020-04-20 12:17:06 UTC
I think there is nothing to fix on systemd side. Moving to NM.

Comment 5 Beniamino Galvani 2020-04-23 15:52:20 UTC
(In reply to Renaud Métrich from comment #3)
> I'm reopening this because systemd manpages / docs must clearly explain this
> behaviour.
> In particular, when it affects Mount Points: the admin just won't notice it
> will fail by default, e.g. when a FQDN is used in the /etc/fstab:
> 
> my-nfs-server:/exports    /my-export    nfs    defaults    0    0
> 
> In the example if a IPv6 IP address is returned by the DNS server for
> "my-nfs-server", the mount will likely fail.
> 
> 
> Hence, something is definitely wrong with the current behaviour.

Hi Renaud,

can you please elaborate on what you would improve or explain better in the
NetworkManager or systemd documentation?

Comment 6 Renaud Métrich 2020-04-24 07:39:03 UTC
Hi Beniamino,

Below is where the issue is:

systemd.special(7) states:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
       network-online.target
       [...]
       What precisely this requires is left to the implementation
           of the network managing service.
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

This is already obscure, the admin must understand that on RHEL8, it's NetworkManager and achieved through exectuing NetworkManager-wait-online.service.
First I think that a patch should be made in systemd manpage to clear state NetworkManager is the network managing service.

Then on the NetworkManager side, we have the following man page referred inside NetworkManager-wait-online.service: "Documentation=man:nm-online(1)" property

nm-online(1) states:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
       By default NetworkManager waits for IPv4 dynamic addressing to complete but does not wait for
       the auto IPv6 dynamic addressing. To wait for IPv6 addressing to complete, either (1) change
       the network connection's IPv6 may-fail setting to no, and/or (2) change the IPv6 addressing
       method to manual or dhcp, to indicate that IPv6 connectivity is expected.
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

This is also obscure:
1. we don't explain how to change "network connection's IPv6 may-fail setting to no"
2. we don't explain how to "change the IPv6 addressing method to manual or dhcp"

I believe we should clearly state that:
1. is "ipv6.may-fail = no"
2. is "ipv6.method" property

Renaud.

Comment 7 Beniamino Galvani 2020-04-27 16:10:27 UTC
Hi Renaud,

I updated the nm-online documentation in the following way:

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/25583de20ba0df49d9364c580011d4c75c3be8f8

Note that point 2. (changing ipv6.method) is no longer mentioned because I think it's wrong to suggest to switch from IPv6 autoconf to manual addressing to solve a service ordering issue. The right solution is to set ipv6.may-fail=no. What do you think?

Comment 9 Renaud Métrich 2021-03-03 07:24:29 UTC
Hi Beniamino, yes, this makes perfect sense. For some reason this got out of my radar.

Comment 10 Thomas Haller 2021-03-03 09:45:29 UTC
this is only a fix to the documentation, and it was already done before 1.26.0 release (and thus is already in rhel-8.3)

But it slipped through the cracks, so it will be properly targeted for rhel-8.5...

Comment 15 errata-xmlrpc 2021-11-09 19:28:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: NetworkManager security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4361