Bug 1873720

The issue remains with the client on kernel-5.8.4 and after the NFS server was upgraded to kernel-5.8.4:  Unable to "see" files or directories (via ls -l) in the /home/<username> directory *except* the first "dot" directory, in my case ".aqbanking"  If I manually enter a subdirectory, /home/<username>/subdirectory, I am able to see files with ls -l

I have the same problem with kernel-5.7.17-200.fc32.x86_64 and kernel-5.8.4-200.fc32.x86_64 and kernel-5.8.5-200.fc32.x86_64 (on nfs client, nfs server still runs 5.7.12-200.fc32.x86_64).

The problem does not occur with kernel-5.7.12-200.fc32.x86_64 and kernel-5.7.16-200.fc32.x86_64.

It seams something in the patches between kernel-5.7.16-200.fc32.x86_64 and kernel-5.7.17-200.fc32.x86_64 seams the reason for this nfs problem.

(In reply to Edgar Hoch from comment #2)
> I have the same problem with kernel-5.7.17-200.fc32.x86_64 and
> kernel-5.8.4-200.fc32.x86_64 and kernel-5.8.5-200.fc32.x86_64 (on nfs
> client, nfs server still runs 5.7.12-200.fc32.x86_64).

On Fedora 32 with home directory in nfs I see only some of the files on NFS client when using kernel 5.7.17. When using kernel 5.7.16 I see all files (same as on local (exported) filesystem on nfs server).

How to test:

1. Export a directory with some files in it from a nfs server. (I have not tested if it needs to be a home directory or any directory in nfs.) Use kernel 5.7.16 or older.
2. Use a nfs client with kernel 5.7.16 or older and list the nfs directory with "ls -l".
3. Use a nfs client with kernel 5.7.17 or newer (up to 5.8.5) and list the nfs directory with "ls -l".

Current result:
In step 3 there are some files not listed compared to step 2.

Expected result:
Listing in step 2 and 3 contains the same files.

Same thing here:

NFS server (exporting folders with protocol version 4.2) is CentOS 7, untouched in a very long time and perfectly working against other CentOS NFS clients; this means that the problem is on the Fedora 32 clients - I observe this exact behaviour (ls -l does not completely list the home folder's content even if the files accessed directly seem to be all there) for those rebooted with 5.7.17 OR 5.8.4 kernels.

I have tested kernel-5.9.0-0.rc3.1.fc32.x86_64 (build from kernel-5.9.0-0.rc3.1.fc34.src.rpm from koji website). The problem still exists with this kernel.

I have tested kernel-5.8.6-200.fc32.x86_64. The problems still exists.

1.) One problem is that on a newly booted machine with autofs on and the home directory is in nfs with autofs, then the nfs autofs directory is not mountet. So ssh login asks for a password (instead of using a ssh key), and if the password is entered, then the home directory is missing and the user login process has pwd of "/".

2.) After "setenforce 0; systemctl restart autofs" on the machine of the example above, then login is possible with ssh key, the home directory is the current working directory, but there "ls" still lists only a subset of the available files in the home directory (compared to the files on the nfs server, witch still runs kernel-5.7.12-200.fc32.x86_64).

3.) Booting kernel-5.6.16-300.fc32.x86_64 without change on selinux instead of a newer kernel works normal. All files are listed on the nfs client as on the nfs server.

Since SELinux stays the same during the tests, something may have changed in kernel between 5.7.16 and 5.7.17, which changes the interpretation or impact of SELinux rules etc. Or some code used by the kernel for nfs (or other helper tools) has changed so it will now blocked by some security rule. Just some ideas where to search for a solution for the problem.

I did a binary search of all commits between v5.7.16 and v5.7.17.

The problem is commit 4476b8282f0bdbf21c8a1e5d783ee11a0edfcaf2,
Upstream commit b4487b93545214a9db8cbf32e86411677b0cca21:
nfs: Fix getxattr kernel panic and memory overflow

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=b4487b93545214a9db8cbf32e86411677b0cca21

The problem still exists with kernel 5.8.7.

I have build modified kernel 5.8.7 packages which has reverted the commit listed above. Reverting this patch solves the problem that some files are missing in the directory listing.


But there also exists an SELinux problem with 5.8.x kernels. The implementation seems changed so a new policy rule is requires.
See bug 1874338.

Created attachment 1714163 [details]
Commit b4487b93545214a9db8cbf32e86411677b0cca21 which should be reverted

This is the commit which should be reverted. Then all nfs files are listed again.

I know that reverting the commit does not solve the reason why this commit was created for. But as it is it causes problems.

Created attachment 1714164 [details]
My example patch for kernel.spec of kernel-5.8.7-200.fc32

Here is my example patch for the spec file of kernel 5.8.7-200.fc32. It reverses the commit mentioned above.

I've followed comment #8 and #9, and built a patched kernel. 

For me, this solves the problems only partially.  I still get OOPS from fs/cachefiles/rdwr.c
which are somehow related to this.  Like rhbz 1827549.

(In reply to Bert DeKnuydt from comment #10)
> For me, this solves the problems only partially.  I still get OOPS from
> fs/cachefiles/rdwr.c
> which are somehow related to this.  Like rhbz 1827549.

We don't use cachefilesd. This may be the reason why we don't get this OOPS.

I'm experiencing the same issue using kerberized nfs mounts, and traced it to the "security_label" export parameter.  

In kernel-5.7.16-200.fc32.x86_64 on the client side everything works as expected, but with 5.8.4-200.fc32.x86_64 and later I only get partial directory listings with ls, find, dir, etc. commands (though all files can still be accessed manually using their full paths). 

I found that by removing "security_label" from the server's /etc/exports file, full directory listings are restored even in the latest kernel-5.8.7-200.fc32.x86_64 kernel.

Using wireshark, I observed that the nfs "READDIR" response packet from the server has identical directory listings after issuing "ls" on a folder regardless of whether security_label is exported or not, but for some reason most files/folders appear invisible to the client unless accessing them with the full file path

I have tested kernel-5.8.8-200.fc32.x86_64 and kernel-5.9.0-0.rc4.20200911git581cb3a26baf.8.fc32.x86_64 (build from kernel-5.9.0-0.rc4.20200911git581cb3a26baf.8.fc34.src.rpm). With both I still get a partial directory listing.

I also use the "security_label" export parameter.

probably solved by https://www.spinics.net/lists/linux-nfs/msg79253.html

Created attachment 1714917 [details]
Patch for kernel.spec of kernel-5.8.9-200.fc32.src.rpm to include fix for nfs problem

I confirm that the patch from https://www.spinics.net/lists/linux-nfs/msg79253.html solves the problem for me.

I build kernel 5.8.9-200.fc32 with this patch applied, and all files of the nfs directory are listed.

Created attachment 1714918 [details]
[PATCH] nfs: Fix security label length not being reset

This file contains the patch from https://www.spinics.net/lists/linux-nfs/msg79253.html .

But I'm not on the linux-nfs mailing list, so I have extracted it from the web page. It has not the optimal format as the other patches of the package. Someone on the mailing list may apply it to the appropriate git repository, then this commit will be a better patch file.

*** Bug 1878813 has been marked as a duplicate of this bug. ***

FEDORA-2020-957351614b has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-957351614b

FEDORA-2020-9f10c3dfae has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-9f10c3dfae

FEDORA-2020-a3b3084904 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-a3b3084904

FEDORA-2020-a3b3084904 has been pushed to the Fedora 31 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-a3b3084904`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-a3b3084904

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-9f10c3dfae has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-9f10c3dfae`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-9f10c3dfae

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-957351614b has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-957351614b`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-957351614b

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-a3b3084904 has been pushed to the Fedora 31 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2020-9f10c3dfae has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2020-957351614b has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

(In reply to Bert DeKnuydt from comment #10)
> I've followed comment #8 and #9, and built a patched kernel. 
> 
> For me, this solves the problems only partially.  I still get OOPS from
> fs/cachefiles/rdwr.c
> which are somehow related to this.  Like rhbz 1827549.

In response to this, here the new kernel 5.8.11 fixes the 'missing files in ls' problem but the crash in the 'cachefilesd' daemon remains, so it is NOT completely fixed for me.

This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 32 changed to end-of-life (EOL) status on 2021-05-25. Fedora 32 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.