Description of problem: Setting of any option using volume set fails when the clients are from base RHEL repo in RHGS 3.1.1 but no issue in RHGS 3.0.4 which seems to be a regression bug. RHGS storage node: [root@node1 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.7 (Santiago) [root@node1 ~]# cat /etc/redhat-storage-release Red Hat Gluster Storage Server 3.1 Update 1 [root@node1 ~]# rpm -qa |grep glusterfs glusterfs-3.7.1-16.el6rhs.x86_64 glusterfs-cli-3.7.1-16.el6rhs.x86_64 glusterfs-libs-3.7.1-16.el6rhs.x86_64 glusterfs-client-xlators-3.7.1-16.el6rhs.x86_64 glusterfs-api-3.7.1-16.el6rhs.x86_64 glusterfs-server-3.7.1-16.el6rhs.x86_64 samba-vfs-glusterfs-4.1.17-14.el6rhs.x86_64 glusterfs-fuse-3.7.1-16.el6rhs.x86_64 glusterfs-rdma-3.7.1-16.el6rhs.x86_64 [root@node1 ~]# cat /var/lib/glusterd/glusterd.info UUID=b8ff28ad-ca68-4e0e-94ed-426dfd636355 operating-version=30703 [root@node1 ~]# gluster volume set dir-perm-test nfs.disable on volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again Client node: [root@rhel6 /]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.7 (Santiago) [root@rhel6 /]# yum list installed |grep glusterfs glusterfs.x86_64 3.6.0.54-1.el6 @rhel-x86_64-server-6 glusterfs-api.x86_64 3.6.0.54-1.el6 @rhel-x86_64-server-6 glusterfs-fuse.x86_64 3.6.0.54-1.el6 @rhel-x86_64-server-6 glusterfs-libs.x86_64 3.6.0.54-1.el6 @rhel-x86_64-server-6 glusterfs-rdma.x86_64 3.6.0.54-1.el6 @rhel-x86_64-server-6 [root@rhel6 /]# df -Th |grep -A9 dir 10.65.3.14:/dir-perm-test fuse.glusterfs 10G 3.3G 6.8G 33% /gluster-test Its not a issue in RHGS 3.0.4 as, RHGS storage node: [root@dhcp10-105 ~]# cat /etc/redhat-storage-release Red Hat Storage Server 3.0 Update 4 [root@dhcp10-105 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.6 (Santiago) [root@dhcp10-105 ~]# rpm -qa |grep glusterfs glusterfs-3.6.0.53-1.el6rhs.x86_64 glusterfs-libs-3.6.0.53-1.el6rhs.x86_64 glusterfs-rdma-3.6.0.53-1.el6rhs.x86_64 glusterfs-api-3.6.0.53-1.el6rhs.x86_64 glusterfs-geo-replication-3.6.0.53-1.el6rhs.x86_64 glusterfs-cli-3.6.0.53-1.el6rhs.x86_64 glusterfs-fuse-3.6.0.53-1.el6rhs.x86_64 glusterfs-server-3.6.0.53-1.el6rhs.x86_64 [root@dhcp10-105 ~]# gluster volume set test nfs.disable on volume set: success [root@dhcp10-105 ~]# gluster v info Volume Name: test Type: Distribute Volume ID: 474da060-ca0f-47e3-bbaa-85e6c3c70a25 Status: Started Snap Volume: no Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.65.10.105:/mnt/gluster/brick1 Options Reconfigured: nfs.disable: on performance.readdir-ahead: on snap-max-hard-limit: 256 snap-max-soft-limit: 90 auto-delete: disable Client node: [root@rhel6 /]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.7 (Santiago) [root@rhel6 /]# yum list installed |grep glusterfs glusterfs.x86_64 3.6.0.54-1.el6 @rhel-x86_64-server-6 glusterfs-api.x86_64 3.6.0.54-1.el6 @rhel-x86_64-server-6 glusterfs-fuse.x86_64 3.6.0.54-1.el6 @rhel-x86_64-server-6 glusterfs-libs.x86_64 3.6.0.54-1.el6 @rhel-x86_64-server-6 glusterfs-rdma.x86_64 3.6.0.54-1.el6 @rhel-x86_64-server-6 [root@rhel6 /]# df -Th |grep -A9 gluster 10.65.10.105:/test fuse.glusterfs 1014M 33M 982M 4% /gluster-test Version-Release number of selected component (if applicable): RHGS 3.1.1 glusterfs-3.7.1-16.el6rhs How reproducible: Always Steps to Reproduce: 1. Create a volume on RHGS 3.1.1 2. Install gluster-fuse clients from base rhel repo "rhel-x86_64-server-6" 3. Mount the volume on client 4. Set the volume option on the volume Actual results: volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again Expected results: volume set: success Additional info: https://bugzilla.redhat.com/show_bug.cgi?id=1004332 Thanks Mukul
We'll look into the case and get back asap.
Does the problem go away if the clients get installed from the rhel-6-server-rhs-client-1-rpms repo?
Hi Sayan, I had suggested the customer to subscribe the rhgs client channel as a workaround for now & waiting for the customer response. Thanks Mukul
(In reply to Mukul Malhotra from comment #3) > Hi Sayan, > > I had suggested the customer to subscribe the rhgs client channel as a > workaround for now & waiting for the customer response. This is not a work around. This is the preferred method of obtaining the clients as they are always refreshed and kept current while the RHEL ones can not be changed so easily. If the clients from the rhgs channel work then we can close this BZ as WONTFIX. Sayan > > Thanks > Mukul
Hello Sayan, Yes you are correct but we do support the fuse client from rhel repo as well which have no issues with RHGS 3.0.4 so why with RHGS 3.1.1 Can you guide me if its documented somewhere which I am not aware of as we only support fuse client from rhgs repo & not the fuse client from rhel repo ? As I remember, the fuse client from rhgs repo is recommended but not mandatory, please correct me if I am missing something. https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Accessing_Data_-_Setting_Up_Clients.html#sect-Native_Client Thanks Mukul
I think the issue here is the different op-version for RHGS vs RHEL repo. Now coming to why you are not seeing this with RHGS 3.0.4 server bits is because that there were no new options introduced in 3.0.4 which are incompatible with the connected client however the same doesn't hold true with RHGS 3.1.1. So irrespective of whether we support fuse client over rhel channel or not this still doesn't look like a bug in that case.
Mukul, As discussed over IRC, we have a sort of agreement about this not been a bug, however please update the bugzilla based on the customer feedback. Thanks, Atin
Hello Atin, >I think the issue here is the different op-version for RHGS vs RHEL repo. Now coming to why you are not seeing this with RHGS 3.0.4 server bits is because that there were no new options introduced in 3.0.4 which are incompatible with the connected client however the same doesn't hold true with RHGS 3.1.1. So irrespective of whether we support fuse client over rhel channel or not this still doesn't look like a bug in that case. The customer issue has resolved by updating packages from RHGS client channel as expected but this does not solves the actual issue as, * When the fuse client from rhel repo will be compatible with RHGS 3.1.x as if the new bits for RHGS 3.1.1 will be merged in next release of RHEL ? but it will still be an issue for further RHGS releases * I am observing this behaviour with 3.0.4 client & 3.1.1 server as, RHGS 3.1.1 : [root@node1 ~]# rpm -qa |grep glusterfs glusterfs-3.7.1-16.el6rhs.x86_64 glusterfs-cli-3.7.1-16.el6rhs.x86_64 glusterfs-libs-3.7.1-16.el6rhs.x86_64 glusterfs-client-xlators-3.7.1-16.el6rhs.x86_64 glusterfs-api-3.7.1-16.el6rhs.x86_64 glusterfs-server-3.7.1-16.el6rhs.x86_64 samba-vfs-glusterfs-4.1.17-14.el6rhs.x86_64 glusterfs-fuse-3.7.1-16.el6rhs.x86_64 glusterfs-rdma-3.7.1-16.el6rhs.x86_64 [root@node1 ~]# cat /var/lib/glusterd/glusterd.info UUID=b8ff28ad-ca68-4e0e-94ed-426dfd636355 operating-version=30703 [root@node1 ~]# gluster volume set dir-perm-test nfs.disable on volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again <<================= RHGS 3.0.4 : [root@dhcp10-105 ~]# rpm -qa |grep glusterfs glusterfs-3.6.0.53-1.el6rhs.x86_64 glusterfs-libs-3.6.0.53-1.el6rhs.x86_64 glusterfs-rdma-3.6.0.53-1.el6rhs.x86_64 glusterfs-api-3.6.0.53-1.el6rhs.x86_64 glusterfs-geo-replication-3.6.0.53-1.el6rhs.x86_64 glusterfs-cli-3.6.0.53-1.el6rhs.x86_64 glusterfs-fuse-3.6.0.53-1.el6rhs.x86_64 glusterfs-server-3.6.0.53-1.el6rhs.x86_64 [root@dhcp10-105 ~]# cat /var/lib/glusterd/glusterd.info UUID=3e2a53e6-e318-4b5a-b033-6ec9a13f3bb6 operating-version=30004 Thanks Mukul
http://review.gluster.org/#/c/5786/ is going to address it. However this is not a regression and the issue is there from day 1.
Hello Atin, >http://review.gluster.org/#/c/5786/ is going to address it. However this is not a regression and the issue is there from day 1. I wanted to understand if this is not a regression than its a known issue as the issue is there from day 1 ? Also, you mean say that 3.0.4 clients are not compatible with 3.1.1 ? Thanks Mukul
(In reply to Mukul Malhotra from comment #10) > Hello Atin, > > >http://review.gluster.org/#/c/5786/ is going to address it. However this is not a regression and the issue is there from day 1. > > I wanted to understand if this is not a regression than its a known issue as > the issue is there from day 1 ? Also, you mean say that 3.0.4 clients are > not compatible with 3.1.1 ? It should be a known issue. From the compatibility perspective, yes this is an issue, you wouldn't be able to set any option when server is running with higher op-version with clients. I'd request Kaushal to provide more in sight to this problem and provide the doc text. > > Thanks > Mukul
Kaushal, is the issue here that the op-versions do not match and therefore 'volume set' doesn't work from one node to another with a different version? Or is this specific to 3.0.4 and 3.1.1 being used together?
The intention with this restriction is to not allow enabling of newer features on a volume that the old clients cannot support. But the check being done is not a right, and will disallow even valid server side options from being set. #1004332 was opened for fixing the check, but the change hasn't been merged yet. We'll try to get the change merged ASAP.
Thanks Kaushal. We have a note about something similar in the RHGS 3.1.2 Installation Guide: Clients installed from the Red Hat Enterprise Linux 6 repository (rhel-x86_64-server-6) are compatible only with Red Hat Gluster Storage 3.0.x. Clients installed from the Red Hat Enterprise Linux 7 repository (rhel-7-server-aus-rpms) are currently compatible with Red Hat Gluster Storage 3.1.x, but Red Hat recommends updating the client using the Red Hat Gluster Storage repository to maintain compatibility in future. In general, Red Hat recommends installing the client from the Red Hat Gluster Storage repository. For details, see https://access.redhat.com/articles/145553. With that in mind, should this be documented as a known issue for 3.1.2 at all?
It looks good to me, however I'd like to get it verified by Kaushal too.
Looks good to me. -Regards, Bipin
Downstream patch is merged now, moving the state to MODIFIED now.
Verified this bug using the build "glusterfs-3.7.9-2.". Steps: ===== 1.Had two node rhgs cluster with 3.1.3 build 2. created a sampple volume and mounted on lower version client ( glusterfs-3.6.0.55-1 ) ]# rpm -qa |grep gluster glusterfs-api-3.6.0.55-1.el6.x86_64 glusterfs-libs-3.6.0.55-1.el6.x86_64 glusterfs-3.6.0.55-1.el6.x86_64 glusterfs-fuse-3.6.0.55-1.el6.x86_64 [root@~]# [root@dhcp ~]# [root@ ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_dhcp42156-lv_root ext4 18G 2.2G 15G 14% / tmpfs tmpfs 939M 0 939M 0% /dev/shm /dev/vda1 ext4 477M 39M 413M 9% /boot IP:/Dis fuse.glusterfs 10G 34M 10G 1% /mnt [root@dhcp42-156 ~]# 3. Done the volume set operation on volume, it worked [root@dhcp ~]# gluster volume set Dis performance.readdir-ahead off volume set: success [root@dhcp ~]# gluster volume set Dis performance.readdir-ahead on volume set: success [root@dhcp ~]# With all above details, moving the bug to verified state.
Changed the Doc type from "Known issue" to "Bug Fix" and removed the Doc text.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1240