Bug 1900241

+++ This bug was initially created as a clone of Bug #1900239 +++

OKD promotion jobs are tracking `testing-devel` stream, which regularly gets fresh kernels.

Recently a single storage test related to NFS has started failing - see https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/origin-ci-test/logs/promote-release-openshift-okd-machine-os-content-e2e-aws-4.5/1280012242783309824

Latest test pass was on https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/origin-ci-test/logs/promote-release-openshift-okd-machine-os-content-e2e-aws-4.5/1278975437036326912 on Jul 02 using image https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/32.20200702.20.0/x86_64/meta.json. pkg diff - https://github.com/coreos/fedora-coreos-config/compare/03c1fe0d1b09db7494e240a535fe8ce3fde18ab6...190b592282cc89d2f50045033763f65c492cbc8d

It appears its most likely related to kernel 5.7.7 update - see https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.7.7

--- Additional comment from Jan Safranek on 2020-07-13 14:42:19 UTC ---

Reproduced with kernel 5.7.8-200.fc32.x86_64. Reformatted for readability.

E0713 14:30:19.540124    5301 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/nfs/15bc6521-d0be-459b-8e1b-37e307510db9-pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e podName:15bc6521-d0be-459b-8e1b-37e307510db9 nodeName
:}" failed. No retries permitted until 2020-07-13 14:32:21.54008141 +0000 UTC m=+944.066210092 (durationBeforeRetry 2m2s).

Error: "error cleaning subPath mounts for volume \"test-volume\" (UniqueName: \"kubernetes.io/nfs/15bc6521-d0be-459b-8e1b-37e307510db9-pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e\") pod \"15bc6521-d0be-459b-8e1b-37e307510db9\" (UID: \"15bc6521-d0be-459b-8e1b-37e307510db9\") :
error processing /var/lib/kubelet/pods/15bc6521-d0be-459b-8e1b-37e307510db9/volume-subpaths/pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e/test-container-subpath-dynamicpv-wjts:
error cleaning subpath mount /var/lib/kubelet/pods/15bc6521-d0be-459b-8e1b-37e307510db9/volume-subpaths/pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e/
test-container-subpath-dynamicpv-wjts/0:
  unmount failed: exit status 16
  Unmounting arguments: /var/lib/kubelet/pods/15bc6521-d0be-459b-8e1b-37e307510db9/volume-subpaths/pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e/test-container-subpath-dynamicpv-wjts/0
  Output: umount.nfs4: /var/lib/kubelet/pods/15bc6521-d0be-459b-8e1b-37e307510db9/volume-subpaths/pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e/test-container-subpath-dynamicpv-wjts/0: Stale file handle

Brief summary of the test:
1. create a pod with a NFS volume, with 2 containers:
  - The first uses subdirectory of the PV as subpath
  - The seconds uses the whole volume

2. Exec into the second container and remove the subpath directory.

3. Delete the pod.

--- Additional comment from Jan Safranek on 2020-07-17 10:41:01 UTC ---

Starting with 5.7.x, kernel does not allow users to unmount NFS mounts with "Stale file handle"

I tested with 5.7.4-200.fc32.x86_64, the first 5.7.x kernel in Fedora 32.

Steps to reproduce (basically, get "Stale file handle" error on bind-mounted nfs dir):

1. Use this dummy /etc/exports:
/var/tmp 127.0.0.1(rw,sync,all_squash,anonuid=1000)

2. Mount it to /mnt/test:
$ mkdir /mnt/test
$ mount localhost:/var/tmp /mnt/test

3. Bind-mount a subdirectory of it to /mnt/test2:
$ mkdir /mnt/test/reproduce
$ mkdir /mnt/test2
$ mount --bind /mnt/test/reproduce /mnt/test2

4. Remove the bind-mounted dir
$ rmdir /mnt/test/reproduce

5. Check that /mnt/test2 is not happy about that
$ ls /mnt/test2
ls: cannot access '/mnt/test2': Stale file handle

This is expected.

6. Try to unmount /mnt/test2
$ umount /mnt/test2
umount.nfs4: /mnt/test2: Stale file handle

This is not expected! There is no way how to unmount the directory. It's mounted forever. Even reboot gets stuck.


With kernel-core-5.6.19-300.fc32.x86_64 (the last 5.6.x in Fedora 32), step 6. succeeds.

--- Additional comment from Vadim Rutkovsky on 2020-07-23 07:37:29 UTC ---

Steve, could you have a look? Reproducible in latest 5.7.x kernel in F32

--- Additional comment from Vadim Rutkovsky on 2020-08-20 07:48:56 UTC ---

No longer happening in 5.7.15-200.fc32.x86_64

--- Additional comment from Vadim Rutkovsky on 2020-09-23 13:53:51 UTC ---

I was wrong - the test didn't pass, instead it was skipped.

The issue still occurs on 5.8.10-200.fc32.x86_64

--- Additional comment from Colin Walters on 2020-09-23 14:20:11 UTC ---

Probably the best way to get traction on this is to bisect it and report to the linux-nfs@ email list: https://linux-nfs.org/wiki/index.php/Main_Page

Speaking with a very broad brush, Fedora kernel BZs are mostly triaged to upstream bugs and that's the best way to address them.

--- Additional comment from Colin Walters on 2020-09-23 14:30:56 UTC ---

Looking at changes and code, at a vague guess this may be related to https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=779df6a5480f1307d51b66ea72352be592265cad
Specifically 
```
	if (ctx->clone_data.sb) {
		if (d_inode(fc->root)->i_fop != &nfs_dir_operations) {
			error = -ESTALE;
```

has the right appearance for this problem at least.

--- Additional comment from Vadim Rutkovsky on 2020-10-09 15:29:57 UTC ---

Still occurs on 5.8.10-200.fc32.x86_64 (same test failing in https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/promote-release-openshift-okd-machine-os-content-e2e-gcp-4.5/1314504432246853632)

--- Additional comment from Vadim Rutkovsky on 2020-11-12 15:57:22 UTC ---

kernel-5.9.8-200.fc33 from updates-testing is also affected