Bug 1543068
| Summary: | [CIOT] : Gluster CLI says "io-threads : enabled" on existing volumes post upgrade. | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Ambarish <asoman> | |
| Component: | glusterd | Assignee: | Ravishankar N <ravishankar> | |
| Status: | CLOSED ERRATA | QA Contact: | Bala Konda Reddy M <bmekala> | |
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | rhgs-3.4 | CC: | amukherj, rcyriac, rhinduja, rhs-bugs, sasundar, storage-qa-internal, vbellur | |
| Target Milestone: | --- | Keywords: | Regression | |
| Target Release: | RHGS 3.4.0 | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.12.2-5 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1545056 (view as bug list) | Environment: | ||
| Last Closed: | 2018-09-04 06:42:04 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1503137, 1545056, 1552404 | |||
|
Description
Ambarish
2018-02-07 16:45:29 UTC
Reproduced on a single brick volume as well. I was not able to re-create this on my dev VMs with the latest rhgs-3.4.0 source install (git HEAD has 3 extra patches on top of 3.12.2-3, namely: * 4f5197f58 - (HEAD -> rhgs-3.4.0, origin/rhgs-3.4.0) cluster/dht: Add migration checks to dht_(f)xattrop (18 hours ago) <N Balachandran> * ef0809430 - rpc: Showing some unusual timer error logs during brick stop (18 hours ago) <Mohit Agrawal> * 14a8e4783 - md-cache: Add additional samba and macOS specific EAs to mdcache (19 hours ago) <Günther Deschner>) ---------------------------------------------------------------------- #gluster v create testvol replica 3 $VM1:/bricks/brick1 $VM2:/bricks/brick1 $VM1:/bricks/brick2 $VM2:/bricks/brick2 $VM1:/bricks/brick3 $VM2:/bricks/brick3 force gluster v info Volume Name: testvol Type: Distributed-Replicate Volume ID: c41ea54f-f900-455b-84b6-4abe8934dae0 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 10.70.43.122:/bricks/brick1 Brick2: 10.70.42.54:/bricks/brick1 Brick3: 10.70.43.122:/bricks/brick2 Brick4: 10.70.42.54:/bricks/brick2 Brick5: 10.70.43.122:/bricks/brick3 Brick6: 10.70.42.54:/bricks/brick3 Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off # gluster v get testvol client-io-threads Option Value ------ ----- performance.client-io-threads off ---------------------------------------------------------------------- Ambarish, could you see if you are able to reproduce this on a single node setup with fresh install bits (rhel 7.x + rhgs-3.4)? That can help us identify if this is related to upgrading only. Cannot reproduce the reported issue on a fresh install. gluster v create testvol replica 2 gqas013:/bricks1/A1 gqas016:/bricks1/A1 Volume Name: testvol Type: Replicate Volume ID: 5d2b436e-2fd1-4201-8119-7f8d1eb25928 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: gqas013:/bricks1/A1 Brick2: gqas016:/bricks1/A1 Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off STEPS TO REPRODUCE : 1. Create rep 3 volume on 3.3.1 async. [root@gqas013 /]# gluster v get drogon client-io-threads Option Value ------ ----- performance.client-io-threads off [root@gqas013 /]# No mounts necessary. 2. Stop glusterd , volume. 3. Uprade glusterfs to 3.4 bits, in an offline way. 4. Restart glusterd,start force volume 5. Check io-threads on old volume. It becomes "on" for some reason post upgrade. [root@gqas013 ~]# gluster v get drogon client-io-threads Option Value ------ ----- performance.client-io-threads on 6. Post this create a new rep3 volume. Observe that CIOT is "on" there as well. [root@gqas013 ~]# gluster v create rep3 gqas013:/bricks1/aa gqas006:/bricks/Aa gqas003:/bricks1/Aaa volume create: rep3: success: please start the volume to access data [root@gqas013 ~]# gluster v start rep3 volume start: rep3: success [root@gqas013 ~]# [root@gqas013 ~]# gluster v get rep3 client-io-threads Option Value ------ ----- performance.client-io-threads on I think this is a problem with cluster-op-version before and after upgrade still being at 31101, while a fresh install of rhgs-3.4.0 would have it at 31301? Ambarish, could you confirm that var/lib/glusterd/glusterd.info still at 31101 post upgrade? @Atin, wondering if we should bump up the cluster op-version to 31301 post upgrade? If the cluster.op-version hasn’t been bumped up as part of the upgrade process then please close this bug as that step is a mandate. Thanks Atin for https://bugzilla.redhat.com/show_bug.cgi?id=1543068#c14. I have also repro'd it for old volumes. Have a replicate volume , upgrade gluster bits ,bump up op version and check CIOT via CLI. Freshly created volumes have IOT as "off" post upgrade,which is expected. I can confirm that the xlator is not loaded in the volfile in eoither case. Giving the bug an appropriate summary. Upstream patch: https://review.gluster.org/#/c/19567/1 Build: 3.12.2-7 Upgraded from 3.8.4-54.4 to 3.12.2-7 Before upgrade: The client-io-threads options is turned off gluster v get 2cross3_98 client-io-threads Option Value ------ ----- performance.client-io-threads off After upgrade: The option is turned off as expected, earlier it was showing on. Hence marking it as verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607 |