Bug 2174761

Summary: Manila can't create new access rules for CephFS/NFS shares - driver hits a NFS-Ganesha export ID limit
Product: Red Hat OpenStack Reporter: Dhruv Shah <dhruv>
Component: openstack-manilaAssignee: Ashley R <ashrodri>
Status: ASSIGNED --- QA Contact: vhariria
Severity: medium Docs Contact:
Priority: medium    
Version: 16.2 (Train)CC: ashrodri, gfidente, gouthamr
Target Milestone: z6Keywords: Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dhruv Shah 2023-03-02 12:11:01 UTC
Description of problem:

It is not possible to create new access rules for the Manila shares for any user or project. 
The service was working fine but now we always get an error creating a new access rule for any share:
```
$ manila access-list testnfs1
+----+-------------+-----------+--------------+-------+------------+------------+------------+
| id | access_type | access_to | access_level | state | access_key | created_at | updated_at |
+----+-------------+-----------+--------------+-------+------------+------------+------------+
+----+-------------+-----------+--------------+-------+------------+------------+------------+
```

We create a new access rule:

```
$ manila access-allow testnfs1 ip 0.0.0.0
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| id           | 84d1c08d-cb3a-4578-9886-a1e6be3db13f |
| share_id     | 38ffd41c-18d3-4578-abdf-d93660996f38 |
| access_level | rw                                   |
| access_to    | 0.0.0.0                              |
| access_type  | ip                                   |
| state        | queued_to_apply                      |
| access_key   | None                                 |
| created_at   | 2023-02-27T12:24:07.000000           |
| updated_at   | None                                 |
| metadata     | {}                                   |
+--------------+--------------------------------------+
```
But its in error state and not possible to access to the new nfs filesystem:

```
# manila access-list testnfs1
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+
| id                                   | access_type | access_to | access_level | state | access_key | created_at                 | updated_at |
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+
| 84d1c08d-cb3a-4578-9886-a1e6be3db13f | ip          | 0.0.0.0   | rw           | error | None       | 2023-02-27T12:24:07.000000 | None       |
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+
```

Ganesha is running in controller11 at this moment, from ganesha logs we can see:

```
...
...
27/02/2023 13:24:07 : epoch 63fc7e79 : controller11 : ganesha.nfsd-57[dbus_heartbeat] reclaim_reset :FSAL :EVENT :start_reclaim failed: -2
27/02/2023 13:24:07 : epoch 63fc7e79 : controller11 : ganesha.nfsd-57[dbus_heartbeat] gsh_export_addexport :EXPORT :CRIT :0 export entries in /etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r added because (invalid param value) errors
27/02/2023 13:24:07 : epoch 63fc7e79 : controller11 : ganesha.nfsd-57[dbus_heartbeat] dbus_message_entrypoint :DBUS :MAJ :Method (AddExport) on (org.ganesha.nfsd.exportmgr) failed: name = (org.freedesktop.DBus.Error.InvalidFileContent), message = (0 export entries in /etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r added because (invalid param value) errors. Details:
Config File (/etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r:2): (65618) is out of range
Config File (/etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r:1): 1 errors while processing parameters for EXPORT
Config File (/etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r:1): Errors found in configuration block EXPORT
```


What is the business impact? Please also provide timeframe information.
We need to fix this issue as soon as possible because users cannot access to the new created nfs shares.

Where are you experiencing the behavior? What environment?
Using manila service, affects all tenants and projects.

When does the behavior occur? Frequency? Repeatedly? At certain times?
Always creating new access rules.


Version-Release number of selected component (if applicable):
16.2


Actual results:
new access rules are getting into an error state.

Expected results:
The new access rule should be working 

Additional info:
Added in subsequent comments

Comment 2 Goutham Pacha Ravi 2023-03-03 19:52:54 UTC
Error obtained from Ganesha logs:

 Config File (/etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r:2): (65618) is out of range