Description of problem: Test failed during FS Regression suite Patrick checked the traceback and confirmed this is issue with kernel not releasing the dentrie for ceph-fuse to cleanup its cache. Test: description: fs/basic_functional/{begin.yaml clusters/4-remote-clients.yaml mount/fuse.yaml overrides/{debug.yaml frag_enable.yaml no_client_pidfile.yaml whitelist_health.yaml RHEL 7.4/3.10.0-693.2.2.el7.x86_64 2017-09-17T01:44:16.920 INFO:tasks.cephfs_test_runner:====================================================================== 2017-09-17T01:44:16.921 INFO:tasks.cephfs_test_runner:ERROR: test_client_pin (tasks.cephfs.test_client_limits.TestClientLimits) 2017-09-17T01:44:16.921 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2017-09-17T01:44:16.921 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2017-09-17T01:44:16.921 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/gitlab.osas.lab.eng.rdu2.redhat.com_ceph_ceph_rh-luminous/qa/tasks/cephfs/test_client_limits.py", line 97, in test_client_pin 2017-09-17T01:44:16.921 INFO:tasks.cephfs_test_runner: self._test_client_pin(True) 2017-09-17T01:44:16.921 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/gitlab.osas.lab.eng.rdu2.redhat.com_ceph_ceph_rh-luminous/qa/tasks/cephfs/test_client_limits.py", line 89, in _test_client_pin 2017-09-17T01:44:16.922 INFO:tasks.cephfs_test_runner: reject_fn=lambda x: x < int(cache_size*.8)) 2017-09-17T01:44:16.922 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/gitlab.osas.lab.eng.rdu2.redhat.com_ceph_ceph_rh-luminous/qa/tasks/ceph_test_case.py", line 125, in wait_until_equal 2017-09-17T01:44:16.922 INFO:tasks.cephfs_test_runner: elapsed, expect_val, val 2017-09-17T01:44:16.922 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out after 600 seconds waiting for 80 (currently 202) 2017-09-17T01:44:16.922 INFO:tasks.cephfs_test_runner: 2017-09-17T01:44:16.922 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2017-09-17T01:44:16.922 INFO:tasks.cephfs_test_runner:Ran 3 tests in 848.134s 2017-09-17T01:44:16.923 INFO:tasks.cephfs_test_runner: 2017-09-17T01:44:16.923 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2017-09-17T01:44:16.923 INFO:tasks.cephfs_test_runner: 2017-09-17T01:44:16.923 INFO:tasks.cephfs_test_runner:====================================================================== 2017-09-17T01:44:16.923 INFO:tasks.cephfs_test_runner:ERROR: test_client_pin (tasks.cephfs.test_client_limits.TestClientLimits) 2017-09-17T01:44:16.923 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2017-09-17T01:44:16.923 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2017-09-17T01:44:16.923 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/gitlab.osas.lab.eng.rdu2.redhat.com_ceph_ceph_rh-luminous/qa/tasks/cephfs/test_client_limits.py", line 97, in test_client_pin 2017-09-17T01:44:16.924 INFO:tasks.cephfs_test_runner: self._test_client_pin(True) 2017-09-17T01:44:16.924 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/gitlab.osas.lab.eng.rdu2.redhat.com_ceph_ceph_rh-luminous/qa/tasks/cephfs/test_client_limits.py", line 89, in _test_client_pin 2017-09-17T01:44:16.924 INFO:tasks.cephfs_test_runner: reject_fn=lambda x: x < int(cache_size*.8)) 2017-09-17T01:44:16.924 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/gitlab.osas.lab.eng.rdu2.redhat.com_ceph_ceph_rh-luminous/qa/tasks/ceph_test_case.py", line 125, in wait_until_equal 2017-09-17T01:44:16.924 INFO:tasks.cephfs_test_runner: elapsed, expect_val, val 2017-09-17T01:44:16.924 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out after 600 seconds waiting for 80 (currently 202) Full logs: http://magna002.ceph.redhat.com/vasu-2017-09-16_00:44:06-fs-luminous---basic-multi/274192/teuthology.log
Verified in 12.2.1-21.el7cp Full FS regression run on above build is at http://pulpito.ceph.redhat.com/vasu-2017-10-18_22:32:22-fs-luminous---basic-multi 2017-10-19T17:17:34.084 INFO:teuthology.orchestra.run.clara004:Running: 'sudo umount -l -f /home/ubuntu/cephtest/mnt.3' 2017-10-19T17:17:34.154 INFO:teuthology.orchestra.run.clara004.stderr:umount: /home/ubuntu/cephtest/mnt.3: mountpoint not found 2017-10-19T17:17:34.155 INFO:teuthology.orchestra.run.clara004:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.3" 2017-10-19T17:17:34.214 INFO:teuthology.orchestra.run.clara004.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2017-10-19T17:17:34.215 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2017-10-19T17:17:34.215 INFO:teuthology.orchestra.run.clara004:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.3" 2017-10-19T17:17:34.275 INFO:teuthology.orchestra.run.clara004.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2017-10-19T17:17:34.275 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2017-10-19T17:17:34.275 INFO:teuthology.orchestra.run.clara004:Running: 'rm -rf /home/ubuntu/cephtest/mnt.3' 2017-10-19T17:17:34.336 INFO:teuthology.orchestra.run.clara004:Running: 'sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf' 2017-10-19T17:17:34.341 INFO:teuthology.orchestra.run.clara013:Running: 'sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf' 2017-10-19T17:17:34.454 INFO:tasks.cephfs_test_runner:test_client_release_bug (tasks.cephfs.test_client_limits.TestClientLimits) ... ok 2017-10-19T17:17:34.454 INFO:tasks.cephfs_test_runner: 2017-10-19T17:17:34.454 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2017-10-19T17:17:34.455 INFO:tasks.cephfs_test_runner:Ran 5 tests in 477.917s
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3387