Bug 1901143 - Manila doesn't support customized ceph cluster name
Summary: Manila doesn't support customized ceph cluster name
Keywords:
Status: CLOSED DUPLICATE of bug 1903005
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-manila
Version: 16.1 (Train)
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Tom Barron
QA Contact: vhariria
mmurray
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-24 15:30 UTC by Alfredo
Modified: 2021-04-09 17:18 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-04-09 17:18:19 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Alfredo 2020-11-24 15:30:31 UTC
Description of problem:
The following jobs:
1. https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/ceph/view/rhos/job/DFG-ceph-rhos-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph-nfs-tls-ganesha/
2. https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/ceph/view/rhos/job/DFG-ceph-rhos-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv6-geneve-ceph-rgw-mds-ganesha/
 
Fail the following tests:
manila_tempest_tests.tests.api.test_rules.ShareRulesTest.test_access_rules_deleted_if_share_deleted_1_2_27[api_with_backend,positive]	7 sec	8
manila_tempest_tests.tests.api.test_rules_negative.ShareIpRulesForNFSNegativeTest.test_create_duplicate_of_ip_rule_1_1_0[api_with_backend,negative]	3.5 sec	8
manila_tempest_tests.tests.api.test_rules_negative.ShareIpRulesForNFSNegativeTest.test_create_duplicate_of_ip_rule_2_2_9[api_with_backend,negative]	94 ms	8
manila_tempest_tests.tests.api.test_rules_negative.ShareIpRulesForNFSNegativeTest.test_create_duplicate_of_ip_rule_3_2_51[api_with_backend,negative]	0.13 sec	8
manila_tempest_tests.tests.api.test_rules_negative.ShareIpRulesForNFSNegativeTest.test_create_duplicate_single_host_rules_1_10_20_30_40[api_with_backend,negative]	3.6 sec	8
manila_tempest_tests.tests.api.test_rules.ShareRulesTest.test_access_rules_deleted_if_share_deleted_2_2_28[api_with_backend,positive]	6.8 sec	8
manila_tempest_tests.tests.api.test_rules_negative.ShareIpRulesForNFSNegativeTest.test_create_duplicate_single_host_rules_2_fd8c_b029_bba6_ac54__1[api_with_backend,negative]	3.5 sec	8
manila_tempest_tests.tests.api.test_rules_negative.ShareIpRulesForNFSNegativeTest.test_create_duplicate_single_host_rules_3_fd2c_b029_bba6_df54__1_128[api_with_backend,negative]	3.6 sec	8
manila_tempest_tests.tests.api.test_rules.ShareRulesTest.test_access_rules_deleted_if_share_deleted_5_2_51[api_with_backend,positive]	6.9 sec	8
manila_tempest_tests.tests.api.test_rules_negative.ShareIpRulesForNFSNegativeTest.test_create_duplicate_single_host_rules_4_10_10_30_40_32[api_with_backend,negative]	3.6 sec	8
manila_tempest_tests.tests.api.test_rules.ShareRulesTest.test_list_access_rules_1_2_27[api_with_backend,positive]	3.5 sec	8
manila_tempest_tests.tests.api.test_rules.ShareRulesTest.test_list_access_rules_2_2_28[api_with_backend,positive]	0.1 sec	8
manila_tempest_tests.tests.api.test_rules.ShareRulesTest.test_list_access_rules_3_2_45[api_with_backend,positive]	97 ms	8
manila_tempest_tests.tests.api.test_rules.ShareRulesTest.test_list_access_rules_6_2_51[api_with_backend,positive]	0.11 sec	8
manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rule_with_cidr_1___2_37___4_[backend,positive]	3.4 sec	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rule_with_cidr_2___2_9___4_[backend,positive]	57 ms	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rule_with_cidr_3___1_0___4_[backend,positive]	93 ms	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rule_with_cidr_4___2_51___4_[backend,positive]	3.3 sec	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rule_with_cidr_5___2_38___6_[backend,positive]	3.3 sec	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rule_with_cidr_6___2_51___6_[backend,positive]	3.3 sec	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rules_with_one_ip_1___2_37___4_[backend,positive]	3.3 sec	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rules_with_one_ip_2___2_9___4_[backend,positive]	85 ms	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rules_with_one_ip_3___1_0___4_[backend,positive]	73 ms	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rules_with_one_ip_4___2_51___4_[backend,positive]	3.3 sec	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rules_with_one_ip_5___2_38___6_[backend,positive]	3.3 sec	8
manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_access_rules_with_one_ip_6___2_51___6_[backend,positive]	3.4 sec	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_ro_access_rule_1_2_27[backend,positive]	65 ms	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_ro_access_rule_2_2_28[backend,positive]	3.3 sec	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_ro_access_rule_3_2_9[backend,positive]	77 ms	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_ro_access_rule_4_1_0[backend,positive]	70 ms	8 manila_tempest_tests.tests.api.test_rules.ShareIpRulesForNFSTest.test_create_delete_ro_access_rule_5_2_51[backend,positive]	83 ms	8

These failures are happening with tests that are trying to grant access to share. Various Shares and Share Rules are in ERROR status. 

Version-Release number of selected component (if applicable):
core_puddle: RHOS-16.1-RHEL-8-20201110.n.1
puppet-manila-15.4.1-1.20200818131916.6c1e210.el8ost.noarch
python3-manilaclient-1.29.0-0.20200310223441.1b2cafb.el8ost.noarch
python3-manila-tests-tempest-1.1.0-0.20200728083439.eba8fa9.el8ost.noarch
puppet-ceph-3.1.2-1.20200916043412.105b71e.el8ost.noarch
ceph-ansible-4.0.31-1.el8cp.noarch

How reproducible:
Consistent

Steps to Reproduce:
1. Deploy any of the 2 jobs mentioned 
2.
3.

Actual results (snippets from failed tests):
---
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ddt.py", line 145, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/manila_tempest_tests/tests/api/test_rules_negative.py", line 103, in test_create_duplicate_of_ip_rule
    self.share["id"], rule["id"], "active")
  File "/usr/lib/python3.6/site-packages/manila_tempest_tests/services/share/json/shares_client.py", line 279, in wait_for_access_rule_status
    rule_id=rule_id)
manila_tempest_tests.share_exceptions.AccessRuleBuildErrorException: Share's rule with id 73b74e66-5909-4289-8c85-402469b459fc is in ERROR status
---
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ddt.py", line 145, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/manila_tempest_tests/tests/api/test_rules_negative.py", line 99, in test_create_duplicate_of_ip_rule
    self.share["id"], access_type, access_to, version=version)
  File "/usr/lib/python3.6/site-packages/manila_tempest_tests/services/share/v2/json/shares_client.py", line 803, in create_access_rule
    extra_headers=True)
  File "/usr/lib/python3.6/site-packages/manila_tempest_tests/services/share/v2/json/shares_client.py", line 70, in post
    headers=headers)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 283, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 687, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 798, in _error_checker
    raise exceptions.BadRequest(resp_body, resp=resp)
tempest.lib.exceptions.BadRequest: Bad request
Details: {'code': 400, 'message': 'Access rules cannot be added while the share or any of its replicas or migration copies has its access_rules_status set to error. Deny any rules in error state and try again.'}

Expected results:
The tests pass

Additional info:
This is what the manila.conf looks like: 
[cephfs]
ganesha_rados_store_enable=True
ganesha_rados_store_pool_name=manila_data
driver_handles_share_servers=False
share_backend_name=cephfs
share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
cephfs_conf_path=/etc/ceph/NFSTLSGanesha0.conf
cephfs_auth_id=manila
cephfs_cluster_name=NFSTLSGanesha0
cephfs_enable_snapshots=True
cephfs_ganesha_server_ip=172.17.5.101
cephfs_ganesha_server_is_remote=False
cephfs_volume_mode=0755
cephfs_protocol_helper_type=NFS

Errors from manila-share.log
2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.InvalidFileContent: 0 export entries in /etc/ganesha/export.d/share-02d680fc-682e-4641-aa1b-0e0af8949323.conf.9BUflq added because (export create, block validation) errors. Details:\nConfig File (/etc/ganesha/export.d/share-02d680fc-682e-4641-aa1b-0e0af8949323.conf.9BUflq:4): 1 validation errors in block FSAL\nConfig File (/etc/ganesha/export.d/share-02d680fc-682e-4641-aa1b-0e0af8949323.conf.9BUflq:4): Errors processing block (FSAL)\nConfig File (/etc/ganesha/export.d/share-02d680fc-682e-4641-aa1b-0e0af8949323.conf.9BUflq:1): 1 validation errors in block EXPORT\nConfig File (/etc/ganesha/export.d/share-02d680fc-682e-4641-aa1b-0e0af8949323.conf.9BUflq:1): Errors found in configuration block EXPORT\n\n'

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server Traceback (most recent call last):

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 474, in add_export

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     "string:EXPORT(Export_Id=%d)" % xid)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 414, in _dbus_send_ganesha

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     message='dbus call %s.%s' % (service, method), **kwargs)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 242, in _execute

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     cmd=e.cmd)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-02d680fc-682e-4641-aa1b-0e0af8949323.conf.9BUflq string:EXPORT(Export_Id=1028)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server Exit code: 1

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server Stdout: ''

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.InvalidFileContent: 0 export entries in /etc/ganesha/export.d/share-02d680fc-682e-4641-aa1b-0e0af8949323.conf.9BUflq added because (export create, block validation) errors. Details:\nConfig File (/etc/ganesha/export.d/share-02d680fc-682e-4641-aa1b-0e0af8949323.conf.9BUflq:4): 1 validation errors in block FSAL\nConfig File (/etc/ganesha/export.d/share-02d680fc-682e-4641-aa1b-0e0af8949323.conf.9BUflq:4): Errors processing block (FSAL)\nConfig File (/etc/ganesha/export.d/share-02d680fc-682e-4641-aa1b-0e0af8949323.conf.9BUflq:1): 1 validation errors in block EXPORT\nConfig File (/etc/ganesha/export.d/share-02d680fc-682e-4641-aa1b-0e0af8949323.conf.9BUflq:1): Errors found in configuration block EXPORT\n\n'

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server Traceback (most recent call last):

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 187, in wrapped

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/utils.py", line 568, in wrapper

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 3554, in update_access

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     share_server=share_server)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 283, in update_access_rules

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     share_server=share_server)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 322, in _update_access_rules

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     share_server)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 390, in _update_rules_through_share_driver

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     share_server=share_server

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/cephfs/driver.py", line 289, in update_access

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     share_server=share_server)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/__init__.py", line 308, in update_access

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     self.ganesha.add_export(share['name'], confdict)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 491, in add_export

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server     cmd=e.cmd)

2020-10-29 07:51:34.444 41 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.

Comment 1 Tom Barron 2020-11-24 15:40:30 UTC
Looks like Ganesha can't parse the exports configuration that is given to it.

If this issue is reproducible it would be helpful to keep the deployment so that it can be inspected.

Comment 2 Alfredo 2020-11-24 18:37:04 UTC
I'll get started on reproducing asap!

Comment 7 Tom Barron 2020-12-04 21:42:08 UTC
As Victoria has remarked, this seems to be a test environment where https://bugzilla.redhat.com/show_bug.cgi?id=1903005 reproduces.

  (overcloud) [stack@undercloud-0 ~]$ manila create --name S1 nfs 1
+---------------------------------------+--------------------------------------+
| Property                              | Value                                |
+---------------------------------------+--------------------------------------+
| id                                    | 3a8f9ca7-da0b-42cd-9741-67433f0c7828 |
| size                                  | 1                                    |
| availability_zone                     | None                                 |
| created_at                            | 2020-12-04T21:28:50.000000           |
| status                                | creating                             |
| name                                  | S1                                   |
| description                           | None                                 |
| project_id                            | a4462a07917b42b4826727553cd9f054     |
| snapshot_id                           | None                                 |
| share_network_id                      | None                                 |
| share_proto                           | NFS                                  |
| metadata                              | {}                                   |
| share_type                            | fb5f8eb4-9984-4adb-888a-100fde81124c |
| is_public                             | False                                |
| snapshot_support                      | False                                |
| task_state                            | None                                 |
| share_type_name                       | default                              |
| access_rules_status                   | active                               |
| replication_type                      | None                                 |
| has_replicas                          | False                                |
| user_id                               | 4f27bb1cbc9d4049adb39454989347b4     |
| create_share_from_snapshot_support    | False                                |
| revert_to_snapshot_support            | False                                |
| share_group_id                        | None                                 |
| source_share_group_snapshot_member_id | None                                 |
| mount_snapshot_support                | False                                |
| share_server_id                       | None                                 |
| host                                  |                                      |
+---------------------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ manila access-allow S1 ip 0.0.0.0
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| id           | a955f0c3-9ba3-4013-a772-5eb1dfec0794 |
| share_id     | 3a8f9ca7-da0b-42cd-9741-67433f0c7828 |
| access_level | rw                                   |
| access_to    | 0.0.0.0                              |
| access_type  | ip                                   |
| state        | queued_to_apply                      |
| access_key   | None                                 |
| created_at   | 2020-12-04T21:29:43.000000           |
| updated_at   | None                                 |
| metadata     | {}                                   |
+--------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ manila show S1
+---------------------------------------+------------------------------------------------------------------------------------------+
| Property                              | Value                                                                                    |
+---------------------------------------+------------------------------------------------------------------------------------------+
| id                                    | 3a8f9ca7-da0b-42cd-9741-67433f0c7828                                                     |
| size                                  | 1                                                                                        |
| availability_zone                     | nova                                                                                     |
| created_at                            | 2020-12-04T21:28:50.000000                                                               |
| status                                | available                                                                                |
| name                                  | S1                                                                                       |
| description                           | None                                                                                     |
| project_id                            | a4462a07917b42b4826727553cd9f054                                                         |
| snapshot_id                           | None                                                                                     |
| share_network_id                      | None                                                                                     |
| share_proto                           | NFS                                                                                      |
| metadata                              | {}                                                                                       |
| share_type                            | fb5f8eb4-9984-4adb-888a-100fde81124c                                                     |
| is_public                             | False                                                                                    |
| snapshot_support                      | False                                                                                    |
| task_state                            | None                                                                                     |
| share_type_name                       | default                                                                                  |
| access_rules_status                   | error                                                                                    |   <==================== access_rules_status is *error*
| replication_type                      | None                                                                                     |
| has_replicas                          | False                                                                                    |
| user_id                               | 4f27bb1cbc9d4049adb39454989347b4                                                         |
| create_share_from_snapshot_support    | False                                                                                    |
| revert_to_snapshot_support            | False                                                                                    |
| share_group_id                        | None                                                                                     |
| source_share_group_snapshot_member_id | None                                                                                     |
| mount_snapshot_support                | False                                                                                    |
| share_server_id                       | None                                                                                     |
| host                                  | hostgroup@cephfs#cephfs                                                                  |
| export_locations                      |                                                                                          |
|                                       | id = cb79e90c-4c03-4087-90f5-14c8f16ce4c8                                                |
|                                       | path = [fd00:fd00:fd00:7000::37f]:/volumes/_nogroup/4a3257fa-a73a-4b54-a534-6cbad46cdcc2 |
|                                       | preferred = False                                                                        |
|                                       | share_instance_id = 4a3257fa-a73a-4b54-a534-6cbad46cdcc2                                 |
|                                       | is_admin_only = False                                                                    |
+---------------------------------------+------------------------------------------------------------------------------------------+


(overcloud) [stack@undercloud-0 ~]$ manila access-list S1
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+
| id                                   | access_type | access_to | access_level | state | access_key | created_at                 | updated_at |
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+
| a955f0c3-9ba3-4013-a772-5eb1dfec0794 | ip          | 0.0.0.0   | rw           | error | None       | 2020-12-04T21:29:43.000000 | None       |
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+

On the Controller we see that the Ceph cluster (and therefore its config files) do not use the default name 'ceph':

[root@controller-0 ~]# ls -l /etc/ceph
total 36
-r--------. 1  167  167  151 Dec  3 23:34 RGWGanesha0.client.admin.keyring
-rw-------. 1  167  167  268 Dec  3 23:38 RGWGanesha0.client.manila.keyring
-rw-------. 1  167  167  227 Dec  3 23:38 RGWGanesha0.client.openstack.keyring
-rw-------. 1  167  167  134 Dec  3 23:38 RGWGanesha0.client.radosgw.keyring
-rw-r--r--. 1 root root 2126 Dec  3 23:33 RGWGanesha0.conf
-r--------. 1  167  167  143 Dec  3 23:34 RGWGanesha0.mgr.controller-0.keyring
-r--------. 1  167  167  143 Dec  3 23:34 RGWGanesha0.mgr.controller-1.keyring
-r--------. 1  167  167  143 Dec  3 23:34 RGWGanesha0.mgr.controller-2.keyring
-r--------. 1 root root   77 Dec  3 23:33 RGWGanesha0.mon.keyring

When manila-share responds to the 'access-allow' command and sends a dbus-command to tap ganesha on the shoulder to process the corresponding export, we see that ganesha cannot parse the export:

2020-12-04 21:29:44.799 45 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.
2020-12-04 21:29:44.799 45 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-4a3257fa-a73a-4b54-a534-6cbad46cdcc2.conf.EeoZJO string:EXPORT(Export_Id=1021)
2020-12-04 21:29:44.799 45 ERROR oslo_messaging.rpc.server Exit code: 1
2020-12-04 21:29:44.799 45 ERROR oslo_messaging.rpc.server Stdout: ''
2020-12-04 21:29:44.799 45 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.InvalidFileContent: 0 export entries in /etc/ganesha/export.d/share-4a3257fa-a73a-4b54-a534-6cbad46cdcc2.conf.EeoZJO added because (export create, block validation) errors. Details:\nConfig File (/etc/ganesha/export.d/share-4a3257fa-a73a-4b54-a534-6cbad46cdcc2.conf.EeoZJO:4): 1 validation errors in block FSAL\nConfig File (/etc/ganesha/export.d/share-4a3257fa-a73a-4b54-a534-6cbad46cdcc2.conf.EeoZJO:4): Errors processing block (FSAL)\nConfig File (/etc/ganesha/export.d/share-4a3257fa-a73a-4b54-a534-6cbad46cdcc2.conf.EeoZJO:1): 1 validation errors in block EXPORT\nConfig File (/etc/ganesha/export.d/share-4a3257fa-a73a-4b54-a534-6cbad46cdcc2.conf.EeoZJO:1): Errors found in configuration block EXPORT\n\n'

This is the same issue reported in https://bugzilla.redhat.com/show_bug.cgi?id=1903005

Comment 8 Tom Barron 2020-12-11 18:29:55 UTC
@Alfredo: While we work out a solution with the Ganesha team, I believe that as a workaround you can do 'sudo ln /etc/ceph/RGWGanesha0.conf /etc/ceph/ceph.conf' on each Controller node.

Comment 9 Tom Barron 2021-04-09 17:18:19 UTC

*** This bug has been marked as a duplicate of bug 1903005 ***


Note You need to log in before you can comment on or make changes to this bug.