+++ This bug was initially created as a clone of Bug #1900239 +++ OKD promotion jobs are tracking `testing-devel` stream, which regularly gets fresh kernels. Recently a single storage test related to NFS has started failing - see https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/origin-ci-test/logs/promote-release-openshift-okd-machine-os-content-e2e-aws-4.5/1280012242783309824 Latest test pass was on https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/origin-ci-test/logs/promote-release-openshift-okd-machine-os-content-e2e-aws-4.5/1278975437036326912 on Jul 02 using image https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/32.20200702.20.0/x86_64/meta.json. pkg diff - https://github.com/coreos/fedora-coreos-config/compare/03c1fe0d1b09db7494e240a535fe8ce3fde18ab6...190b592282cc89d2f50045033763f65c492cbc8d It appears its most likely related to kernel 5.7.7 update - see https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.7.7 --- Additional comment from Jan Safranek on 2020-07-13 14:42:19 UTC --- Reproduced with kernel 5.7.8-200.fc32.x86_64. Reformatted for readability. E0713 14:30:19.540124 5301 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/nfs/15bc6521-d0be-459b-8e1b-37e307510db9-pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e podName:15bc6521-d0be-459b-8e1b-37e307510db9 nodeName :}" failed. No retries permitted until 2020-07-13 14:32:21.54008141 +0000 UTC m=+944.066210092 (durationBeforeRetry 2m2s). Error: "error cleaning subPath mounts for volume \"test-volume\" (UniqueName: \"kubernetes.io/nfs/15bc6521-d0be-459b-8e1b-37e307510db9-pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e\") pod \"15bc6521-d0be-459b-8e1b-37e307510db9\" (UID: \"15bc6521-d0be-459b-8e1b-37e307510db9\") : error processing /var/lib/kubelet/pods/15bc6521-d0be-459b-8e1b-37e307510db9/volume-subpaths/pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e/test-container-subpath-dynamicpv-wjts: error cleaning subpath mount /var/lib/kubelet/pods/15bc6521-d0be-459b-8e1b-37e307510db9/volume-subpaths/pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e/ test-container-subpath-dynamicpv-wjts/0: unmount failed: exit status 16 Unmounting arguments: /var/lib/kubelet/pods/15bc6521-d0be-459b-8e1b-37e307510db9/volume-subpaths/pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e/test-container-subpath-dynamicpv-wjts/0 Output: umount.nfs4: /var/lib/kubelet/pods/15bc6521-d0be-459b-8e1b-37e307510db9/volume-subpaths/pvc-cb938e62-792c-41a2-b7a8-ec9650c97d6e/test-container-subpath-dynamicpv-wjts/0: Stale file handle Brief summary of the test: 1. create a pod with a NFS volume, with 2 containers: - The first uses subdirectory of the PV as subpath - The seconds uses the whole volume 2. Exec into the second container and remove the subpath directory. 3. Delete the pod. --- Additional comment from Jan Safranek on 2020-07-17 10:41:01 UTC --- Starting with 5.7.x, kernel does not allow users to unmount NFS mounts with "Stale file handle" I tested with 5.7.4-200.fc32.x86_64, the first 5.7.x kernel in Fedora 32. Steps to reproduce (basically, get "Stale file handle" error on bind-mounted nfs dir): 1. Use this dummy /etc/exports: /var/tmp 127.0.0.1(rw,sync,all_squash,anonuid=1000) 2. Mount it to /mnt/test: $ mkdir /mnt/test $ mount localhost:/var/tmp /mnt/test 3. Bind-mount a subdirectory of it to /mnt/test2: $ mkdir /mnt/test/reproduce $ mkdir /mnt/test2 $ mount --bind /mnt/test/reproduce /mnt/test2 4. Remove the bind-mounted dir $ rmdir /mnt/test/reproduce 5. Check that /mnt/test2 is not happy about that $ ls /mnt/test2 ls: cannot access '/mnt/test2': Stale file handle This is expected. 6. Try to unmount /mnt/test2 $ umount /mnt/test2 umount.nfs4: /mnt/test2: Stale file handle This is not expected! There is no way how to unmount the directory. It's mounted forever. Even reboot gets stuck. With kernel-core-5.6.19-300.fc32.x86_64 (the last 5.6.x in Fedora 32), step 6. succeeds. --- Additional comment from Vadim Rutkovsky on 2020-07-23 07:37:29 UTC --- Steve, could you have a look? Reproducible in latest 5.7.x kernel in F32 --- Additional comment from Vadim Rutkovsky on 2020-08-20 07:48:56 UTC --- No longer happening in 5.7.15-200.fc32.x86_64 --- Additional comment from Vadim Rutkovsky on 2020-09-23 13:53:51 UTC --- I was wrong - the test didn't pass, instead it was skipped. The issue still occurs on 5.8.10-200.fc32.x86_64 --- Additional comment from Colin Walters on 2020-09-23 14:20:11 UTC --- Probably the best way to get traction on this is to bisect it and report to the linux-nfs@ email list: https://linux-nfs.org/wiki/index.php/Main_Page Speaking with a very broad brush, Fedora kernel BZs are mostly triaged to upstream bugs and that's the best way to address them. --- Additional comment from Colin Walters on 2020-09-23 14:30:56 UTC --- Looking at changes and code, at a vague guess this may be related to https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=779df6a5480f1307d51b66ea72352be592265cad Specifically ``` if (ctx->clone_data.sb) { if (d_inode(fc->root)->i_fop != &nfs_dir_operations) { error = -ESTALE; ``` has the right appearance for this problem at least. --- Additional comment from Vadim Rutkovsky on 2020-10-09 15:29:57 UTC --- Still occurs on 5.8.10-200.fc32.x86_64 (same test failing in https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/promote-release-openshift-okd-machine-os-content-e2e-gcp-4.5/1314504432246853632) --- Additional comment from Vadim Rutkovsky on 2020-11-12 15:57:22 UTC --- kernel-5.9.8-200.fc33 from updates-testing is also affected
As I understood on slack, this waits for 1.19.4 rebase to land and then we need to apply this: https://github.com/openshift/origin/pull/25701
Pruned cc: list.
I used cluster-bot to launch test with the PR, test case was skipped. https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-launch-aws/1353576480851890176
Verified pass.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6.17 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0424