Bug 2077585

Summary: update to nfs-utils-2.6.1-1.rc4 causes systemd-udev-settle.service not to start properly
Product: [Fedora] Fedora Reporter: Martin Wolf <mwolf>
Component: nfs-utilsAssignee: Steve Dickson <steved>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 36CC: bfields, camelot, justinkb, kparal, luk.claes, steved, tbecker
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: nfs-utils-2.6.1-2.rc4.fc36 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-02 17:28:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
cockpit output of errors upon boot
none
"unable to find device" is still present none

Description Martin Wolf 2022-04-21 16:17:53 UTC
Created attachment 1874177 [details]
cockpit output of errors upon boot

Created attachment 1874177 [details]
cockpit output of errors upon boot

After the update of nfs-utils-2.6.1-1.rc4
my system hangs for up to three minutes at boot
it seems it has a problem mounting my nfs share?

I added a picture of the error as listed in cockpit.

downgrading to nfs-utils-2.6.1-1.rc3.fc36 and libnfsidmap solves the issue.

Comment 1 Martin Wolf 2022-04-21 17:37:39 UTC
Apr 21 17:56:19 el-ryzerino audit[1467]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:udev_t:s0-s0:c0.c1023 pid=1467 comm="nfsrahead" exe="/usr/libexec/nfsrahead" sig=11 res=1
Apr 21 17:56:19 el-ryzerino nfsrahead[1463]: unable to find device 253:0
Apr 21 17:56:19 el-ryzerino nfsrahead[1464]: unable to find device 8:0
Apr 21 17:56:19 el-ryzerino nfsrahead[1466]: unable to find device 259:0
Apr 21 17:56:19 el-ryzerino nfsrahead[1465]: unable to find device 252:0
Apr 21 17:56:21 el-ryzerino nfsrahead[2053]: unable to find device 253:1
...
Apr 21 17:58:19 el-ryzerino systemd-coredump[2128]: [🡕] Process 2071 (nfsrahead) of user 0 dumped core.
                                                    
                                                    Module linux-vdso.so.1 with build-id 04b708c7c91e947d7413a24109a43e4cc798e340
                                                    Module libpcre2-8.so.0 with build-id 2827cb8b86a0a697fa6e6646690d1b01de9b6fba
                                                    Metadata for module libpcre2-8.so.0 owned by FDO found: {
                                                            "type" : "rpm",
                                                            "name" : "pcre2",
                                                            "version" : "10.39-1.fc36.1",
                                                            "architecture" : "x86_64",
                                                            "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                                                    }
                                                    
                                                    Module ld-linux-x86-64.so.2 with build-id f89cd66df7f1b9e65eacad5a07364c3168df5409
                                                    Module libselinux.so.1 with build-id 9ee47929aa5779a0fe6469346be26bfa6426a516
                                                    Metadata for module libselinux.so.1 owned by FDO found: {
                                                            "type" : "rpm",
                                                            "name" : "libselinux",
                                                            "version" : "3.3-4.fc36",
                                                            "architecture" : "x86_64",
                                                            "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                                                    }
                                                    
                                                    Module libblkid.so.1 with build-id 9e58123c73f92b29e14b080448cade11b65edde2
                                                    Metadata for module libblkid.so.1 owned by FDO found: {
                                                            "type" : "rpm",
                                                            "name" : "util-linux",
                                                            "version" : "2.38-0.2.fc36",
                                                            "architecture" : "x86_64",
                                                            "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                                                    }
                                                    
                                                    Module libc.so.6 with build-id 0532ce157b523f362aaee706381a390672f3db8a
                                                    Module libmount.so.1 with build-id 598330551ee15a053368f71ffc841cc243654899
                                                    Metadata for module libmount.so.1 owned by FDO found: {
                                                            "type" : "rpm",
                                                            "name" : "util-linux",
                                                            "version" : "2.38-0.2.fc36",
                                                            "architecture" : "x86_64",
                                                            "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                                                    }
                                                    
                                                    Module nfsrahead with build-id 5216b3d441a8d01b3b7eec51431e6a9336864cc1
                                                    Metadata for module nfsrahead owned by FDO found: {
                                                            "type" : "rpm",
                                                            "name" : "nfs-utils",
                                                            "version" : "2.6.1-1.rc4.fc36",
                                                            "architecture" : "x86_64",
                                                            "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                                                    }
                                                    
                                                    Stack trace of thread 2071:
                                                    #0  0x000055bed0a4b8ac main (nfsrahead + 0x28ac)
                                                    #1  0x00007fe21be17590 __libc_start_call_main (libc.so.6 + 0x2d590)
                                                    #2  0x00007fe21be17649 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2d649)
                                                    #3  0x000055bed0a4bbe5 _start (nfsrahead + 0x2be5)
                                                    ELF object binary architecture: AMD x86-64
Apr 21 17:58:19 el-ryzerino systemd-udevd[1349]: btrfs-2: Process '/usr/libexec/nfsrahead btrfs-2' terminated by signal SEGV.
Apr 21 17:58:19 el-ryzerino systemd-udevd[1349]: btrfs-2: Failed to wait for spawned command '/usr/libexec/nfsrahead btrfs-2': Input/output error
Apr 21 17:58:19 el-ryzerino systemd-udevd[1349]: btrfs-2: /usr/lib/udev/rules.d/99-nfs.rules:1 Failed to execute "/usr/libexec/nfsrahead btrfs-2": Input/output error
Apr 21 17:58:19 el-ryzerino systemd-coredump[2127]: [🡕] Process 1467 (nfsrahead) of user 0 dumped core.
                                                    
                                                    Module linux-vdso.so.1 with build-id 04b708c7c91e947d7413a24109a43e4cc798e340
                                                    Module libpcre2-8.so.0 with build-id 2827cb8b86a0a697fa6e6646690d1b01de9b6fba
                                                    Metadata for module libpcre2-8.so.0 owned by FDO found: {
                                                            "type" : "rpm",
                                                            "name" : "pcre2",
                                                            "version" : "10.39-1.fc36.1",
                                                            "architecture" : "x86_64",
                                                            "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                                                    }
                                                    
                                                    Module ld-linux-x86-64.so.2 with build-id f89cd66df7f1b9e65eacad5a07364c3168df5409
                                                    Module libselinux.so.1 with build-id 9ee47929aa5779a0fe6469346be26bfa6426a516
                                                    Metadata for module libselinux.so.1 owned by FDO found: {
                                                            "type" : "rpm",
                                                            "name" : "libselinux",
                                                            "version" : "3.3-4.fc36",
                                                            "architecture" : "x86_64",
                                                            "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                                                    }
                                                    
                                                    Module libblkid.so.1 with build-id 9e58123c73f92b29e14b080448cade11b65edde2
                                                    Metadata for module libblkid.so.1 owned by FDO found: {
                                                            "type" : "rpm",
                                                            "name" : "util-linux",
                                                            "version" : "2.38-0.2.fc36",
                                                            "architecture" : "x86_64",
                                                            "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                                                    }
                                                    
                                                    Module libc.so.6 with build-id 0532ce157b523f362aaee706381a390672f3db8a
                                                    Module libmount.so.1 with build-id 598330551ee15a053368f71ffc841cc243654899
                                                    Metadata for module libmount.so.1 owned by FDO found: {
                                                            "type" : "rpm",
                                                            "name" : "util-linux",
                                                            "version" : "2.38-0.2.fc36",
                                                            "architecture" : "x86_64",
                                                            "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                                                    }
                                                    
                                                    Module nfsrahead with build-id 5216b3d441a8d01b3b7eec51431e6a9336864cc1
                                                    Metadata for module nfsrahead owned by FDO found: {
                                                            "type" : "rpm",
                                                            "name" : "nfs-utils",
                                                            "version" : "2.6.1-1.rc4.fc36",
                                                            "architecture" : "x86_64",
                                                            "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                                                    }
                                                    
                                                    Stack trace of thread 1467:
                                                    #0  0x00005564060d68ac main (nfsrahead + 0x28ac)
                                                    #1  0x00007f5e3c6df590 __libc_start_call_main (libc.so.6 + 0x2d590)
                                                    #2  0x00007f5e3c6df649 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2d649)
                                                    #3  0x00005564060d6be5 _start (nfsrahead + 0x2be5)
                                                    ELF object binary architecture: AMD x86-64
Apr 21 17:58:19 el-ryzerino systemd-udevd[1288]: btrfs-1: Process '/usr/libexec/nfsrahead btrfs-1' terminated by signal SEGV.
Apr 21 17:58:19 el-ryzerino audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-update-utmp comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=>
Apr 21 17:58:19 el-ryzerino systemd-udevd[1288]: btrfs-1: Failed to wait for spawned command '/usr/libexec/nfsrahead btrfs-1': Input/output error
Apr 21 17:58:19 el-ryzerino systemd-udevd[1288]: btrfs-1: /usr/lib/udev/rules.d/99-nfs.rules:1 Failed to execute "/usr/libexec/nfsrahead btrfs-1": Input/output error
...
Apr 21 17:58:26 el-ryzerino nfsrahead[2502]: unable to find device 0:65
Apr 21 17:58:26 el-ryzerino nfsrahead[2508]: setting /backup readahead to 128
Apr 21 17:58:26 el-ryzerino nfsrahead[2503]: unable to find device 0:66
Apr 21 17:58:26 el-ryzerino nfsrahead[2504]: unable to find device 0:67
Apr 21 17:58:26 el-ryzerino nfsrahead[2506]: unable to find device 0:68

Comment 2 Martin Wolf 2022-04-21 17:38:24 UTC
/backup is my nfs4 mount

Comment 3 Paul Mulders 2022-04-22 06:47:21 UTC
this happens here too, and I don't even have any nfs mounts. it also causes gdm to show up only after a minute or so, when normally it shows up after 5 secs on booting.

Comment 4 Kamil Páral 2022-04-22 15:01:21 UTC
systemd-udev-settle.service times out during boot, that makes the boot 3 minutes longer. A pretty clean F36 Workstation VM (upgraded from F35). No NFS mounts. Verified to be caused by nfs-mount update.

Comment 5 Thiago Rafael Becker 2022-04-25 20:04:43 UTC
Patch submitted upstream: https://lore.kernel.org/linux-nfs/20220425195948.2627428-1-tbecker@redhat.com/T/#u

Comment 6 Steve Dickson 2022-04-28 14:40:26 UTC
(In reply to Thiago Rafael Becker from comment #5)
> Patch submitted upstream:
> https://lore.kernel.org/linux-nfs/20220425195948.2627428-1-tbecker@redhat.
> com/T/#u

Here is a scratch build that has the above patch

https://koji.fedoraproject.org/koji/taskinfo?taskID=86356846

Could you please verify it does fix the problem... tia!

Comment 7 Martin Wolf 2022-04-28 16:18:50 UTC
it indeed starts faster now again, also the service does not fail anymore, but I still see these "unable to find device xyz:xy"

Comment 8 Martin Wolf 2022-04-28 16:19:37 UTC
Created attachment 1875720 [details]
"unable to find device" is still present

Comment 9 Steve Dickson 2022-04-29 14:29:39 UTC
(In reply to Martin Wolf from comment #7)
> it indeed starts faster now again, also the service does not fail anymore,
> but I still see these "unable to find device xyz:xy"

It appears the logging is being a bit too verbose... 
Once they are under control... I'll do the official 
build and release. Thanks for the help!

Comment 10 Steve Dickson 2022-04-29 20:49:44 UTC
(In reply to Martin Wolf from comment #8)
> Created attachment 1875720 [details]
> "unable to find device" is still present

Here is an updated scratch build that turns down the
verbosity... Would you mind do a quick run to make
sure both problems are solved? tia!

 https://koji.fedoraproject.org/koji/taskinfo?taskID=86414374

Comment 11 Martin Wolf 2022-04-29 21:02:21 UTC
yes. that works as your predicted.

Comment 12 camelot 2022-05-12 01:21:30 UTC
Causes https://bugzilla.redhat.com/show_bug.cgi?id=2001058