Description of problem: I'm using rhceph:7-493 (Reef) orchestrated by cephadm; and using OpenStack Manila to create an NFS export with CephFS. My setup includes a 3-node ceph nfs cluster, front-ended by the Ingress service ( with "haproxy-protocol" as ingress mode ). The setup has a Virtual IP using an IPv6 address, and the client/s specified in the export have IPv6 addresses as well. I'm unsure if this problem is in the ingress service (nothing relevant in the logging) or in NFS-Ganesha (I've added verbose/FULL_DEBUG logging to this report). OpenStack users have long used IPv6 with NFS-Ganesha, however, not with the latest version of NFS-Ganesha, and not when it is setup as a "ceph nfs" cluster service. Version-Release number of selected component (if applicable): - Ceph Version rhceph:7-493 - NFS-Ganesha package versions: nfs-utils-2.5.4-27.el9.x86_64 nfs-ganesha-selinux-5.7-5.el9cp.noarch nfs-ganesha-5.7-5.el9cp.x86_64 nfs-ganesha-rgw-5.7-5.el9cp.x86_64 nfs-ganesha-ceph-5.7-5.el9cp.x86_64 nfs-ganesha-rados-grace-5.7-5.el9cp.x86_64 nfs-ganesha-rados-urls-5.7-5.el9cp.x86_64 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Create a Manila share (CephFS subvolume) 2. Create a Manila access rule on the share with IPv6 client ("ceph nfs export apply") 3. Attempt to mount share on the client Actual results: Mount fails with mount(2): Input/output error Expected results: Mount succeeds Additional info: Manila share details ===================== $ openstack share create nfs 1 --name share1 $ openstack share access create share1 ip 2620:cf:cf:aaaa::6b +--------------+--------------------------------------+ | Field | Value | +--------------+--------------------------------------+ | id | d47920bf-3a29-4a8c-9e2c-63d672e58785 | | share_id | 4159a122-bc1d-49b3-bbb4-d11ee5f66dd4 | | access_level | rw | | access_to | 2620:cf:cf:aaaa::6b | | access_type | ip | | state | queued_to_apply | | access_key | None | | created_at | 2025-03-28T23:21:03.300483 | | updated_at | None | | properties | | +--------------+--------------------------------------+ $ openstack share access list share1 +--------------------------------------+-------------+---------------------+--------------+--------+------------+----------------------------+----------------------------+ | ID | Access Type | Access To | Access Level | State | Access Key | Created At | Updated At | +--------------------------------------+-------------+---------------------+--------------+--------+------------+----------------------------+----------------------------+ | d47920bf-3a29-4a8c-9e2c-63d672e58785 | ip | 2620:cf:cf:aaaa::6b | rw | active | None | 2025-03-28T23:21:03.300483 | 2025-03-28T23:21:03.745510 | +--------------------------------------+-------------+---------------------+--------------+--------+------------+----------------------------+----------------------------+ $ openstack share export location list share1 +--------------------------------------+------------------------------------------------------------------------------------------------------------------+-----------+ | ID | Path | Preferred | +--------------------------------------+------------------------------------------------------------------------------------------------------------------+-----------+ | 851bbcef-3273-442b-b1a3-fbbcbb992f36 | [2620:cf:cf:aaaa::2]:/volumes/_nogroup/97ea173d-f9fe-41ec-abb8-72518acd5585/b025b9ac-cd4a-4c00-b47a-4faa24dacb4b | True | +--------------------------------------+------------------------------------------------------------------------------------------------------------------+-----------+ Ceph Details: ============= # ceph orch ps | grep nfs haproxy.nfs.cephfs.ceph-r1wfskb8-0.dbjetm ceph-r1wfskb8-0 *:2049,9049 running (31h) 3m ago 31h 11.5M - 2.4.22-f8e3218 ff05a790b4e9 336fca2a766d haproxy.nfs.cephfs.ceph-r1wfskb8-1.qkkbfz ceph-r1wfskb8-1 *:2049,9049 running (31h) 3m ago 31h 12.4M - 2.4.22-f8e3218 ff05a790b4e9 7a483dcb904b haproxy.nfs.cephfs.ceph-r1wfskb8-2.lnzmst ceph-r1wfskb8-2 *:2049,9049 running (31h) 3m ago 31h 13.0M - 2.4.22-f8e3218 ff05a790b4e9 f86848797a67 keepalived.nfs.cephfs.ceph-r1wfskb8-0.becivx ceph-r1wfskb8-0 running (31h) 3m ago 31h 1828k - 2.2.8 617308907a17 b88da97a3b3f keepalived.nfs.cephfs.ceph-r1wfskb8-1.tvdbbj ceph-r1wfskb8-1 running (31h) 3m ago 31h 1828k - 2.2.8 617308907a17 f9e06c8633fc keepalived.nfs.cephfs.ceph-r1wfskb8-2.eqdyqa ceph-r1wfskb8-2 running (31h) 3m ago 31h 1832k - 2.2.8 617308907a17 f3382352a78d nfs.cephfs.0.0.ceph-r1wfskb8-1.cfnejd ceph-r1wfskb8-1 *:12049 running (33m) 3m ago 31h 83.7M - 5.7 11fbfe5b88bc 81d63c7af4d1 nfs.cephfs.1.0.ceph-r1wfskb8-2.raroai ceph-r1wfskb8-2 *:12049 running (33m) 3m ago 31h 82.2M - 5.7 11fbfe5b88bc b995553191c1 nfs.cephfs.2.0.ceph-r1wfskb8-0.qedgcv ceph-r1wfskb8-0 *:12049 running (33m) 3m ago 31h 81.2M - 5.7 11fbfe5b88bc bb990f3a2bee Ceph NFS cluster: $ ceph nfs cluster info cephfs { "cephfs": { "backend": [ { "hostname": "ceph-r1wfskb8-0", "ip": "2620:cf:cf:aaaa::6a", "port": 12049 }, { "hostname": "ceph-r1wfskb8-1", "ip": "2620:cf:cf:aaaa::6b", "port": 12049 }, { "hostname": "ceph-r1wfskb8-2", "ip": "2620:cf:cf:aaaa::6c", "port": 12049 } ], "monitor_port": 9049, "port": 2049, "virtual_ip": "2620:cf:cf:aaaa::2" } } $ ceph nfs export info cephfs /volumes/_nogroup/97ea173d-f9fe-41ec-abb8-72518acd5585/b025b9ac-cd4a-4c00-b47a-4faa24dacb4b { "access_type": "RO", "clients": [ { "access_type": "rw", "addresses": [ "2620:cf:cf:aaaa::6b" ], "squash": "none" } ], "cluster_id": "cephfs", "export_id": 1, "fsal": { "cmount_path": "/", "fs_name": "cephfs", "name": "CEPH", "user_id": "nfs.cephfs.cephfs" }, "path": "/volumes/_nogroup/97ea173d-f9fe-41ec-abb8-72518acd5585/b025b9ac-cd4a-4c00-b47a-4faa24dacb4b", "protocols": [ 3, 4 ], "pseudo": "/volumes/_nogroup/97ea173d-f9fe-41ec-abb8-72518acd5585/b025b9ac-cd4a-4c00-b47a-4faa24dacb4b", "security_label": true, "squash": "none", "transports": [ "TCP" ] } From the client =============== [zuul@ceph-r1wfskb8-1 ~]$ rpcinfo -T tcp6 -s 2620:cf:cf:aaaa::2 program version(s) netid(s) service owner 100000 2,3,4 local,udp,tcp,udp6,tcp6 portmapper superuser 100003 4,3 tcp6,tcp,udp6,udp nfs superuser 100005 3,1 tcp6,tcp,udp6,udp mountd superuser [zuul@ceph-r1wfskb8-1 ~]$ sudo mount -t nfs [2620:cf:cf:aaaa::2]:/volumes/_nogroup/97ea173d-f9fe-41ec-abb8-72518acd5585/b025b9ac-cd4a-4c00-b47a-4faa24dacb4b test -vvv mount.nfs: timeout set for Fri Mar 28 19:27:06 2025 mount.nfs: trying text-based options 'vers=4.2,addr=2620:cf:cf:aaaa::2,clientaddr=2620:cf:cf:aaaa::6b' mount.nfs: mount(2): Input/output error mount.nfs: mount system call failed
Please specify the severity of this bug. Severity is defined here: https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Hi Sachin, Goutham provided access information out of band, however the env is no longer accessible today. Please let us know if further triage in needed.
(In reply to Sachin Punadikar from comment #19) > As per discussion with Goutham, with latest code, he was not able to > reproduce this issue. > Waiting for more details on this from him. Thank you Sachin! Apologies for the late response here. Yes, I'm unable to reproduce this issue with RHOSO 18 and RHCS 8. Originally, the issue was observed with RHCS 7 (container tag: 7-493), I'll try to reproduce this with RHCS 7. However, I'll close this bug and report a new one if the issue does occur with RHCS 7. Ceph Image tag: '8-492' ceph (client) version: ceph version 19.2.1-222.el9cp (f2cd71cc2f7b46709c2351134ac89ea3e9f609b6) squid (stable) Version of NFS-Ganesha: # rpm -qa | grep ganesha nfs-ganesha-selinux-6.5-18.el9cp.noarch nfs-ganesha-6.5-18.el9cp.x86_64 nfs-ganesha-ceph-6.5-18.el9cp.x86_64 nfs-ganesha-rados-grace-6.5-18.el9cp.x86_64 nfs-ganesha-rados-urls-6.5-18.el9cp.x86_64 nfs-ganesha-rgw-6.5-18.el9cp.x86_64 nfs-ganesha-utils-6.5-18.el9cp.x86_64 Version of haproxy: # rpm -qa | grep haproxy haproxy-2.4.22-4.el9.x86_64 [zuul@ceph-4b8r13sm-0 ~]$ sudo netstat -anp | grep ganesha tcp6 0 0 :::50051 :::* LISTEN 674756/ganesha.nfsd tcp6 0 0 :::9587 :::* LISTEN 674756/ganesha.nfsd tcp6 0 0 :::12049 :::* LISTEN 674756/ganesha.nfsd tcp6 0 0 :::35843 :::* LISTEN 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::48316 2620:cf:cf:aaaa::6:6800 ESTABLISHED 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::48284 2620:cf:cf:aaaa::6:6804 ESTABLISHED 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::37740 2620:cf:cf:aaaa::6:3300 ESTABLISHED 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::37720 2620:cf:cf:aaaa::6:3300 ESTABLISHED 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::38992 2620:cf:cf:aaaa::6:6808 ESTABLISHED 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::34506 2620:cf:cf:aaaa::6:6800 ESTABLISHED 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::46436 2620:cf:cf:aaaa::6:6812 ESTABLISHED 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::37706 2620:cf:cf:aaaa::6:3300 ESTABLISHED 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::34498 2620:cf:cf:aaaa::6:6800 ESTABLISHED 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::37324 2620:cf:cf:aaaa::6:6800 ESTABLISHED 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::48302 2620:cf:cf:aaaa::6:6800 ESTABLISHED 674756/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::44354 2620:cf:cf:aaaa::6:6802 ESTABLISHED 674756/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 17506330 674756/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 17506331 674756/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 17506333 674756/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 17506332 674756/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 17502972 674756/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 17502970 674756/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 17502981 674756/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 17502982 674756/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 17502971 674756/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 17502969 674756/ganesha.nfsd ceph-1 [zuul@ceph-4b8r13sm-1 ~]$ sudo netstat -anp | grep ganesha tcp6 0 0 :::9587 :::* LISTEN 691585/ganesha.nfsd tcp6 0 0 :::50051 :::* LISTEN 691585/ganesha.nfsd tcp6 0 0 :::12049 :::* LISTEN 691585/ganesha.nfsd tcp6 0 0 :::43483 :::* LISTEN 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::47312 2620:cf:cf:aaaa::6:6804 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::56904 2620:cf:cf:aaaa::6:6800 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::33774 2620:cf:cf:aaaa::6:6800 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::57410 2620:cf:cf:aaaa::6:3300 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::33764 2620:cf:cf:aaaa::6:6800 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::34792 2620:cf:cf:aaaa::6:6802 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::37048 2620:cf:cf:aaaa::6:6808 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::56912 2620:cf:cf:aaaa::6:6800 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::40768 2620:cf:cf:aaaa::6:3300 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::60898 2620:cf:cf:aaaa::6:6812 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::45504 2620:cf:cf:aaaa::6:6806 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::40756 2620:cf:cf:aaaa::6:3300 ESTABLISHED 691585/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::59378 2620:cf:cf:aaaa::6:6800 ESTABLISHED 691585/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 15663086 691585/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 15663085 691585/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 15663088 691585/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 15663087 691585/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 15662055 691585/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 15662054 691585/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 15662056 691585/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 15662057 691585/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 15662066 691585/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 15662067 691585/ganesha.nfsd ceph-2 [zuul@ceph-4b8r13sm-2 ~]$ sudo netstat -anp | grep ganesha tcp6 0 0 :::9587 :::* LISTEN 967256/ganesha.nfsd tcp6 0 0 :::33849 :::* LISTEN 967256/ganesha.nfsd tcp6 0 0 :::50051 :::* LISTEN 967256/ganesha.nfsd tcp6 0 0 :::12049 :::* LISTEN 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::51848 2620:cf:cf:aaaa::6:6800 ESTABLISHED 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::45662 2620:cf:cf:aaaa::6:6800 ESTABLISHED 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::45894 2620:cf:cf:aaaa::6:6800 ESTABLISHED 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::56328 2620:cf:cf:aaaa::6:3300 ESTABLISHED 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::45130 2620:cf:cf:aaaa::6:6808 ESTABLISHED 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::55542 2620:cf:cf:aaaa::6:6802 ESTABLISHED 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::37670 2620:cf:cf:aaaa::6:3300 ESTABLISHED 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::45880 2620:cf:cf:aaaa::6:6800 ESTABLISHED 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::33644 2620:cf:cf:aaaa::6:6806 ESTABLISHED 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::37682 2620:cf:cf:aaaa::6:3300 ESTABLISHED 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::36392 2620:cf:cf:aaaa::6:6800 ESTABLISHED 967256/ganesha.nfsd tcp6 0 0 2620:cf:cf:aaaa:::35836 2620:cf:cf:aaaa::6:6812 ESTABLISHED 967256/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 19131074 967256/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 19131073 967256/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 19131072 967256/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 19131075 967256/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 19132465 967256/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 19132463 967256/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 19132464 967256/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 19132475 967256/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 19132474 967256/ganesha.nfsd unix 3 [ ] STREAM CONNECTED 19132462 967256/ganesha.nfsd On Ceph-0: [ceph: root@ceph-4b8r13sm-0 /]# ceph orch ps | grep nfs haproxy.nfs.cephfs.ceph-4b8r13sm-0.avbymj ceph-4b8r13sm-0 *:2049,9049 running (5d) 9m ago 5d 13.7M - 2.4.22-f8e3218 9a50affc328d 49f12de5e9c7 haproxy.nfs.cephfs.ceph-4b8r13sm-1.fyoctw ceph-4b8r13sm-1 *:2049,9049 running (5d) 9m ago 5d 13.5M - 2.4.22-f8e3218 9a50affc328d 3f524765fa81 haproxy.nfs.cephfs.ceph-4b8r13sm-2.dhrwvp ceph-4b8r13sm-2 *:2049,9049 running (5d) 9m ago 5d 15.6M - 2.4.22-f8e3218 9a50affc328d 1edaac2a2572 keepalived.nfs.cephfs.ceph-4b8r13sm-0.qhyesk ceph-4b8r13sm-0 running (5d) 9m ago 5d 1559k - 2.2.8 65ab4b13c05e f404ffddd7b3 keepalived.nfs.cephfs.ceph-4b8r13sm-1.ptcdap ceph-4b8r13sm-1 running (5d) 9m ago 5d 1551k - 2.2.8 65ab4b13c05e 4d4d5cb12837 keepalived.nfs.cephfs.ceph-4b8r13sm-2.rqynkl ceph-4b8r13sm-2 running (5d) 9m ago 5d 1555k - 2.2.8 65ab4b13c05e b1e44a52e9a5 nfs.cephfs.0.0.ceph-4b8r13sm-1.ruinft ceph-4b8r13sm-1 *:12049 running (83m) 9m ago 5d 98.0M - 6.5 62a14a43dc5b fdfb12d9b712 nfs.cephfs.1.0.ceph-4b8r13sm-2.lkpgpj ceph-4b8r13sm-2 *:12049 running (10m) 9m ago 5d 73.8M - 6.5 62a14a43dc5b 1dede608d005 nfs.cephfs.2.0.ceph-4b8r13sm-0.vdxryv ceph-4b8r13sm-0 *:12049 running (13m) 9m ago 5d 74.1M - 6.5 62a14a43dc5b 8e9161072e81 [ceph: root@ceph-4b8r13sm-0 /]# ceph nfs cluster info cephfs { "cephfs": { "backend": [ { "hostname": "ceph-4b8r13sm-0", "ip": "2620:cf:cf:aaaa::6a", "port": 12049 }, { "hostname": "ceph-4b8r13sm-1", "ip": "2620:cf:cf:aaaa::6b", "port": 12049 }, { "hostname": "ceph-4b8r13sm-2", "ip": "2620:cf:cf:aaaa::6c", "port": 12049 } ], "ingress_mode": "haproxy-protocol", "monitor_port": 9049, "port": 2049, "virtual_ip": "2620:cf:cf:aaaa::2" } } [zuul@ceph-4b8r13sm-2 ~]$ rpcinfo -T tcp6 -s 2620:cf:cf:aaaa::2 program version(s) netid(s) service owner 100000 2,3,4 local,udp,tcp,udp6,tcp6 portmapper superuser 100003 4,3 tcp6,tcp nfs superuser 100005 3,1 tcp6,tcp mountd superuser [ceph: root@ceph-4b8r13sm-0 /]# ceph nfs export ls cephfs [ "/volumes/_nogroup/60b76a8e-2380-43f4-ac4a-96cf7ea06ffe/1ef19db7-d593-4a30-9cde-4a3c9c14f7a1" ] [ceph: root@ceph-4b8r13sm-0 /]# ceph nfs export info cephfs /volumes/_nogroup/60b76a8e-2380-43f4-ac4a-96cf7ea06ffe/1ef19db7-d593-4a30-9cde-4a3c9c14f7a1 { "access_type": "RW", "clients": [], "cluster_id": "cephfs", "export_id": 1, "fsal": { "cmount_path": "/", "fs_name": "cephfs", "name": "CEPH", "user_id": "nfs.cephfs.cephfs.fc95c256" }, "path": "/volumes/_nogroup/60b76a8e-2380-43f4-ac4a-96cf7ea06ffe/1ef19db7-d593-4a30-9cde-4a3c9c14f7a1", "protocols": [ 3, 4 ], "pseudo": "/volumes/_nogroup/60b76a8e-2380-43f4-ac4a-96cf7ea06ffe/1ef19db7-d593-4a30-9cde-4a3c9c14f7a1", "security_label": true, "squash": "none", "transports": [ "TCP" ] }