Bug 1298955 - [GSS] - Setting of any option using volume set fails when the clients are 3.0.4 and server is 3.1.1
[GSS] - Setting of any option using volume set fails when the clients are 3.0...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
3.1
x86_64 Linux
high Severity high
: ---
: RHGS 3.1.3
Assigned To: Kaushal
Byreddy
: ZStream
Depends On:
Blocks: 1268895 1299183
  Show dependency treegraph
 
Reported: 2016-01-15 08:50 EST by Mukul Malhotra
Modified: 2016-09-17 12:47 EDT (History)
11 users (show)

See Also:
Fixed In Version: glusterfs-3.7.9-2
Doc Type: Bug Fix
Doc Text:
The 'gluster volume set' command has a check intended to prevent newer options from being set on a volume when older clients are connected. However, this check was not implemented correctly, and instead prevented any option from being set when old clients were connected. Attempts to set a volume option resulted in the following error message: "volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command" This check has been corrected so that it only fails when new options are being set on a volume with connections to old clients.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-06-23 01:02:39 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Mukul Malhotra 2016-01-15 08:50:01 EST
Description of problem:

Setting of any option using volume set fails when the clients are from base RHEL repo in RHGS 3.1.1 but no issue in RHGS 3.0.4 which seems to be a regression bug.

RHGS storage node:

[root@node1 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.7 (Santiago)

[root@node1 ~]# cat /etc/redhat-storage-release 
Red Hat Gluster Storage Server 3.1 Update 1

[root@node1 ~]# rpm -qa |grep glusterfs
glusterfs-3.7.1-16.el6rhs.x86_64
glusterfs-cli-3.7.1-16.el6rhs.x86_64
glusterfs-libs-3.7.1-16.el6rhs.x86_64
glusterfs-client-xlators-3.7.1-16.el6rhs.x86_64
glusterfs-api-3.7.1-16.el6rhs.x86_64
glusterfs-server-3.7.1-16.el6rhs.x86_64
samba-vfs-glusterfs-4.1.17-14.el6rhs.x86_64
glusterfs-fuse-3.7.1-16.el6rhs.x86_64
glusterfs-rdma-3.7.1-16.el6rhs.x86_64

[root@node1 ~]# cat /var/lib/glusterd/glusterd.info 
UUID=b8ff28ad-ca68-4e0e-94ed-426dfd636355
operating-version=30703

[root@node1 ~]# gluster volume set dir-perm-test nfs.disable on
volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again

Client node:

[root@rhel6 /]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.7 (Santiago)

[root@rhel6 /]# yum list installed |grep glusterfs
glusterfs.x86_64                 3.6.0.54-1.el6            @rhel-x86_64-server-6
glusterfs-api.x86_64             3.6.0.54-1.el6            @rhel-x86_64-server-6
glusterfs-fuse.x86_64            3.6.0.54-1.el6            @rhel-x86_64-server-6
glusterfs-libs.x86_64            3.6.0.54-1.el6            @rhel-x86_64-server-6
glusterfs-rdma.x86_64            3.6.0.54-1.el6            @rhel-x86_64-server-6

[root@rhel6 /]# df -Th |grep -A9 dir
10.65.3.14:/dir-perm-test
                     fuse.glusterfs   10G  3.3G  6.8G  33% /gluster-test

Its not a issue in RHGS 3.0.4 as,

RHGS storage node:

[root@dhcp10-105 ~]# cat /etc/redhat-storage-release 
Red Hat Storage Server 3.0 Update 4

[root@dhcp10-105 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.6 (Santiago)

[root@dhcp10-105 ~]# rpm -qa |grep glusterfs
glusterfs-3.6.0.53-1.el6rhs.x86_64
glusterfs-libs-3.6.0.53-1.el6rhs.x86_64
glusterfs-rdma-3.6.0.53-1.el6rhs.x86_64
glusterfs-api-3.6.0.53-1.el6rhs.x86_64
glusterfs-geo-replication-3.6.0.53-1.el6rhs.x86_64
glusterfs-cli-3.6.0.53-1.el6rhs.x86_64
glusterfs-fuse-3.6.0.53-1.el6rhs.x86_64
glusterfs-server-3.6.0.53-1.el6rhs.x86_64

[root@dhcp10-105 ~]# gluster volume set test nfs.disable on
volume set: success

[root@dhcp10-105 ~]# gluster v info
 
Volume Name: test
Type: Distribute
Volume ID: 474da060-ca0f-47e3-bbaa-85e6c3c70a25
Status: Started
Snap Volume: no
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.65.10.105:/mnt/gluster/brick1
Options Reconfigured:
nfs.disable: on
performance.readdir-ahead: on
snap-max-hard-limit: 256
snap-max-soft-limit: 90
auto-delete: disable

Client node:

[root@rhel6 /]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.7 (Santiago)

[root@rhel6 /]# yum list installed |grep glusterfs
glusterfs.x86_64                 3.6.0.54-1.el6            @rhel-x86_64-server-6
glusterfs-api.x86_64             3.6.0.54-1.el6            @rhel-x86_64-server-6
glusterfs-fuse.x86_64            3.6.0.54-1.el6            @rhel-x86_64-server-6
glusterfs-libs.x86_64            3.6.0.54-1.el6            @rhel-x86_64-server-6
glusterfs-rdma.x86_64            3.6.0.54-1.el6            @rhel-x86_64-server-6

[root@rhel6 /]# df -Th |grep -A9 gluster
10.65.10.105:/test   fuse.glusterfs 1014M   33M  982M   4% /gluster-test

Version-Release number of selected component (if applicable):

RHGS 3.1.1
glusterfs-3.7.1-16.el6rhs

How reproducible:

Always

Steps to Reproduce:

1. Create a volume on RHGS 3.1.1
2. Install gluster-fuse clients from base rhel repo "rhel-x86_64-server-6"
3. Mount the volume on client
4. Set the volume option on the volume

Actual results:

volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again

Expected results:

volume set: success

Additional info:

https://bugzilla.redhat.com/show_bug.cgi?id=1004332

Thanks
Mukul
Comment 1 Atin Mukherjee 2016-01-17 12:36:02 EST
We'll look into the case and get back asap.
Comment 2 Sayan Saha 2016-01-17 23:04:17 EST
Does the problem go away if the clients get installed from the rhel-6-server-rhs-client-1-rpms repo?
Comment 3 Mukul Malhotra 2016-01-18 02:21:03 EST
Hi Sayan,

I had suggested the customer to subscribe the rhgs client channel as a workaround for now & waiting for the customer response.

Thanks
Mukul
Comment 4 Sayan Saha 2016-01-18 09:28:50 EST
(In reply to Mukul Malhotra from comment #3)
> Hi Sayan,
> 
> I had suggested the customer to subscribe the rhgs client channel as a
> workaround for now & waiting for the customer response.

This is not a work around. This is the preferred method of obtaining the clients as they are always refreshed and kept current while the RHEL ones can not be changed so easily. If the clients from the rhgs channel work then we can close this BZ as WONTFIX.

Sayan

> 
> Thanks
> Mukul
Comment 5 Mukul Malhotra 2016-01-18 09:56:56 EST
Hello Sayan,

Yes you are correct but we do support the fuse client from rhel repo as well which have no issues with RHGS 3.0.4 so why with RHGS 3.1.1

Can you guide me if its documented somewhere which I am not aware of as we only support fuse client from rhgs repo & not the fuse client from rhel repo ?

As I remember, the fuse client from rhgs repo is recommended but not mandatory, please correct me if I am missing something.

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Accessing_Data_-_Setting_Up_Clients.html#sect-Native_Client

Thanks
Mukul
Comment 6 Atin Mukherjee 2016-01-18 23:30:37 EST
I think the issue here is the different op-version for RHGS vs RHEL repo. Now coming to why you are not seeing this with RHGS 3.0.4 server bits is because that there were no new options introduced in 3.0.4 which are incompatible with the connected client however the same doesn't hold true with RHGS 3.1.1. So irrespective of whether we support fuse client over rhel channel or not this still doesn't look like a bug in that case.
Comment 7 Atin Mukherjee 2016-01-19 22:53:51 EST
Mukul,

As discussed over IRC, we have a sort of agreement about this not been a bug, however please update the bugzilla based on the customer feedback.

Thanks,
Atin
Comment 8 Mukul Malhotra 2016-02-05 05:00:43 EST
Hello Atin,

>I think the issue here is the different op-version for RHGS vs RHEL repo. Now coming to why you are not seeing this with RHGS 3.0.4 server bits is because that there were no new options introduced in 3.0.4 which are incompatible with the connected client however the same doesn't hold true with RHGS 3.1.1. So irrespective of whether we support fuse client over rhel channel or not this still doesn't look like a bug in that case.

The customer issue has resolved by updating packages from RHGS client channel as expected but this does not solves the actual issue as,

* When the fuse client from rhel repo will be compatible with RHGS 3.1.x as if the new bits for RHGS 3.1.1 will be merged in next release of RHEL ? but it will still be an issue for further RHGS releases

* I am observing this behaviour with 3.0.4 client & 3.1.1 server as,

 RHGS 3.1.1 :

[root@node1 ~]# rpm -qa |grep glusterfs
glusterfs-3.7.1-16.el6rhs.x86_64
glusterfs-cli-3.7.1-16.el6rhs.x86_64
glusterfs-libs-3.7.1-16.el6rhs.x86_64
glusterfs-client-xlators-3.7.1-16.el6rhs.x86_64
glusterfs-api-3.7.1-16.el6rhs.x86_64
glusterfs-server-3.7.1-16.el6rhs.x86_64
samba-vfs-glusterfs-4.1.17-14.el6rhs.x86_64
glusterfs-fuse-3.7.1-16.el6rhs.x86_64
glusterfs-rdma-3.7.1-16.el6rhs.x86_64

[root@node1 ~]# cat /var/lib/glusterd/glusterd.info 
UUID=b8ff28ad-ca68-4e0e-94ed-426dfd636355
operating-version=30703

[root@node1 ~]# gluster volume set dir-perm-test nfs.disable on
volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again    <<=================

RHGS 3.0.4 :

[root@dhcp10-105 ~]# rpm -qa |grep glusterfs
glusterfs-3.6.0.53-1.el6rhs.x86_64
glusterfs-libs-3.6.0.53-1.el6rhs.x86_64
glusterfs-rdma-3.6.0.53-1.el6rhs.x86_64
glusterfs-api-3.6.0.53-1.el6rhs.x86_64
glusterfs-geo-replication-3.6.0.53-1.el6rhs.x86_64
glusterfs-cli-3.6.0.53-1.el6rhs.x86_64
glusterfs-fuse-3.6.0.53-1.el6rhs.x86_64
glusterfs-server-3.6.0.53-1.el6rhs.x86_64

[root@dhcp10-105 ~]# cat /var/lib/glusterd/glusterd.info 
UUID=3e2a53e6-e318-4b5a-b033-6ec9a13f3bb6
operating-version=30004

Thanks
Mukul
Comment 9 Atin Mukherjee 2016-02-05 07:11:42 EST
http://review.gluster.org/#/c/5786/ is going to address it. However this is not a regression and the issue is there from day 1.
Comment 10 Mukul Malhotra 2016-02-08 04:25:19 EST
Hello Atin,

>http://review.gluster.org/#/c/5786/ is going to address it. However this is not a regression and the issue is there from day 1.

I wanted to understand if this is not a regression than its a known issue as the issue is there from day 1 ?  Also, you mean say that 3.0.4 clients are not compatible with 3.1.1  ?

Thanks
Mukul
Comment 11 Atin Mukherjee 2016-02-08 06:00:43 EST
(In reply to Mukul Malhotra from comment #10)
> Hello Atin,
> 
> >http://review.gluster.org/#/c/5786/ is going to address it. However this is not a regression and the issue is there from day 1.
> 
> I wanted to understand if this is not a regression than its a known issue as
> the issue is there from day 1 ?  Also, you mean say that 3.0.4 clients are
> not compatible with 3.1.1  ?
It should be a known issue. From the compatibility perspective, yes this is an issue, you wouldn't be able to set any option when server is running with higher op-version with clients. I'd request Kaushal to provide more in sight to this problem and provide the doc text.


> 
> Thanks
> Mukul
Comment 12 Laura Bailey 2016-02-08 20:00:44 EST
Kaushal, is the issue here that the op-versions do not match and therefore 'volume set' doesn't work from one node to another with a different version? Or is this specific to 3.0.4 and 3.1.1 being used together?
Comment 13 Kaushal 2016-02-10 01:03:17 EST
The intention with this restriction is to not allow enabling of newer features on a volume that the old clients cannot support. But the check being done is not a right, and will disallow even valid server side options from being set. #1004332 was opened for fixing the check, but the change hasn't been merged yet. We'll try to get the change merged ASAP.
Comment 15 Laura Bailey 2016-02-11 21:28:55 EST
Thanks Kaushal.

We have a note about something similar in the RHGS 3.1.2 Installation Guide:

Clients installed from the Red Hat Enterprise Linux 6 repository (rhel-x86_64-server-6) are compatible only with Red Hat Gluster Storage 3.0.x. Clients installed from the Red Hat Enterprise Linux 7 repository (rhel-7-server-aus-rpms) are currently compatible with Red Hat Gluster Storage 3.1.x, but Red Hat recommends updating the client using the Red Hat Gluster Storage repository to maintain compatibility in future. In general, Red Hat recommends installing the client from the Red Hat Gluster Storage repository. For details, see https://access.redhat.com/articles/145553.


With that in mind, should this be documented as a known issue for 3.1.2 at all?
Comment 17 Atin Mukherjee 2016-02-15 06:49:58 EST
It looks good to me, however I'd like to get it verified by Kaushal too.
Comment 19 Bipin Kunal 2016-02-16 02:13:40 EST
Looks good to me.

-Regards,
Bipin
Comment 22 Atin Mukherjee 2016-03-28 01:48:16 EDT
Downstream patch is merged now, moving the state to MODIFIED now.
Comment 24 Byreddy 2016-04-26 02:48:37 EDT
Verified this bug using the build "glusterfs-3.7.9-2.".

Steps:
=====
1.Had two node rhgs cluster with 3.1.3 build
2. created a sampple volume and mounted on lower version client ( glusterfs-3.6.0.55-1 )

]# rpm -qa |grep gluster
glusterfs-api-3.6.0.55-1.el6.x86_64
glusterfs-libs-3.6.0.55-1.el6.x86_64
glusterfs-3.6.0.55-1.el6.x86_64
glusterfs-fuse-3.6.0.55-1.el6.x86_64
[root@~]# 
[root@dhcp ~]# 
[root@ ~]# df -Th
Filesystem           Type            Size  Used Avail Use% Mounted on
/dev/mapper/vg_dhcp42156-lv_root
                     ext4             18G  2.2G   15G  14% /
tmpfs                tmpfs           939M     0  939M   0% /dev/shm
/dev/vda1            ext4            477M   39M  413M   9% /boot
IP:/Dis    fuse.glusterfs   10G   34M   10G   1% /mnt
[root@dhcp42-156 ~]# 



3. Done the volume set operation on volume, it worked 
[root@dhcp ~]# gluster volume set Dis performance.readdir-ahead off
volume set: success
[root@dhcp ~]# gluster volume set Dis performance.readdir-ahead on
volume set: success
[root@dhcp ~]# 

With all above  details, moving the bug to verified state.
Comment 25 Byreddy 2016-04-26 03:09:41 EDT
Changed the Doc type from  "Known issue"  to "Bug Fix" and removed the Doc text.
Comment 29 errata-xmlrpc 2016-06-23 01:02:39 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240

Note You need to log in before you can comment on or make changes to this bug.