Bug 2322352

This is the copy of the Bug #2291163

I am copying this bug because:
This BZ tracks the deadlock issue discussed in the comment https://bugzilla.redhat.com/show_bug.cgi?id=2291163#c21 while the original Bug #2291163 tracks the rm -rf hang issue which could be different. But the steps to reproduce remains the same.


Description of problem:
============
Copying from the comment https://bugzilla.redhat.com/show_bug.cgi?id=2291163#c13

Scenario-

1. Create NFS cluster
2. Create 2000 exports mapped to 2000 subvolume
3. Mount the 2000 exports on 100 clients via v4.2.
4. Run fio from 100 clients in parallel on 2000 exports.
5. Perform the rm -rf operations on mount point.

Observation:

rm -rf operation is stuck indefinitely and cluster is not freeing up the space.

# ceph -s
  cluster:
    id:     4e687a60-638e-11ee-8772-b49691cee574
    health: HEALTH_ERR
            8 backfillfull osd(s)
            1 full osd(s)
            13 nearfull osd(s)
            9 pool(s) full

  services:
    mon: 1 daemons, quorum cali013 (age 2d)
    mgr: cali013.qakwdk(active, since 2d), standbys: cali016.rhribl, cali015.hvvbwh
    mds: 1/1 daemons up, 1 standby
    osd: 35 osds: 35 up (since 2d), 35 in (since 4w)
    rgw: 2 daemons active (2 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   9 pools, 1233 pgs
    objects: 6.53M objects, 25 TiB
    usage:   75 TiB used, 11 TiB / 86 TiB avail
    pgs:     1232 active+clean
             1    active+clean+scrubbing

=======================

Cluster details:
================
Credentials - root/passwd

Installer Node - 10.8.130.13 

NFS is running on 2 nodes -

# ceph nfs cluster info nfsganesha
{
  "nfsganesha": {
    "backend": [
      {
        "hostname": "cali015",
        "ip": "10.8.130.15",
        "port": 12049
      },
      {
        "hostname": "cali016",
        "ip": "10.8.130.16",
        "port": 12049
      }
    ],
    "monitor_port": 9049,
    "port": 2049,
    "virtual_ip": "10.8.130.236"
  }
}

1 of the client on which the rm operation is stuck - 10.0.209.10 (root/passwd)

Requesting to please take a look into the cluster as it is in same state and reproducing the same issue again will be time consuming.