Reproduced on a single brick volume as well.

I was not able to re-create this on my dev VMs with the latest rhgs-3.4.0 source install (git HEAD has 3 extra patches on top of 3.12.2-3, namely:
* 4f5197f58 - (HEAD -> rhgs-3.4.0, origin/rhgs-3.4.0) cluster/dht: Add migration checks to dht_(f)xattrop (18 hours ago) <N Balachandran>
* ef0809430 - rpc: Showing some unusual timer error logs during brick stop (18 hours ago) <Mohit Agrawal>
* 14a8e4783 - md-cache: Add additional samba and macOS specific EAs to mdcache (19 hours ago) <Günther Deschner>)
----------------------------------------------------------------------
#gluster v create testvol replica 3 $VM1:/bricks/brick1 $VM2:/bricks/brick1 $VM1:/bricks/brick2 $VM2:/bricks/brick2 $VM1:/bricks/brick3 $VM2:/bricks/brick3 force

gluster v info
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: c41ea54f-f900-455b-84b6-4abe8934dae0
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.43.122:/bricks/brick1
Brick2: 10.70.42.54:/bricks/brick1
Brick3: 10.70.43.122:/bricks/brick2
Brick4: 10.70.42.54:/bricks/brick2
Brick5: 10.70.43.122:/bricks/brick3
Brick6: 10.70.42.54:/bricks/brick3
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

# gluster  v get testvol client-io-threads
Option                                  Value                                   
------                                  -----                                   
performance.client-io-threads           off    
----------------------------------------------------------------------
Ambarish, could you see if you are able to reproduce this on a single node setup with fresh install bits (rhel 7.x + rhgs-3.4)? That can help us identify if this is related to upgrading only.

Cannot reproduce the reported issue on a fresh install.

gluster v create testvol replica 2 gqas013:/bricks1/A1 gqas016:/bricks1/A1

Volume Name: testvol
Type: Replicate
Volume ID: 5d2b436e-2fd1-4201-8119-7f8d1eb25928
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gqas013:/bricks1/A1
Brick2: gqas016:/bricks1/A1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

STEPS TO  REPRODUCE :

1. Create rep 3 volume on 3.3.1 async.

[root@gqas013 /]# gluster v get drogon client-io-threads
Option                                  Value                                   
------                                  -----                                   
performance.client-io-threads           off                                     
[root@gqas013 /]# 

No mounts necessary.

2. Stop glusterd , volume.

3. Uprade glusterfs to 3.4 bits, in an offline way.

4. Restart glusterd,start force volume

5. Check io-threads on old volume. It becomes "on" for some reason post upgrade.

[root@gqas013 ~]# gluster v get drogon client-io-threads
Option                                  Value                                   
------                                  -----                                   
performance.client-io-threads           on                                      


6. Post this create a new rep3 volume. Observe that CIOT is "on" there as well.

[root@gqas013 ~]# gluster v create rep3 gqas013:/bricks1/aa gqas006:/bricks/Aa gqas003:/bricks1/Aaa 
volume create: rep3: success: please start the volume to access data

[root@gqas013 ~]# gluster v start rep3
volume start: rep3: success
[root@gqas013 ~]# 

[root@gqas013 ~]# gluster v get rep3 client-io-threads
Option                                  Value                                   
------                                  -----                                   
performance.client-io-threads           on

I think this is a problem with cluster-op-version before and after upgrade still being at 31101, while a fresh install of rhgs-3.4.0 would have it at 31301?

Ambarish, could you confirm that var/lib/glusterd/glusterd.info still at 31101 post upgrade?

@Atin, wondering if we should bump up the cluster op-version to 31301 post upgrade?

If the cluster.op-version hasn’t been bumped up as part of the upgrade process then please close this bug as that step is a mandate.

Thanks Atin for https://bugzilla.redhat.com/show_bug.cgi?id=1543068#c14.

I have also repro'd it for old volumes. Have a replicate volume , upgrade gluster bits ,bump up op version and check CIOT via CLI.

Freshly created volumes have IOT as "off" post upgrade,which is expected.

I can confirm that the xlator is not loaded in the volfile in eoither case.

Giving the bug an appropriate summary.

Upstream patch: https://review.gluster.org/#/c/19567/1

Build: 3.12.2-7

Upgraded from 3.8.4-54.4 to 3.12.2-7
Before upgrade: The client-io-threads options is turned off
gluster v get 2cross3_98 client-io-threads
Option                                  Value                                   
------                                  -----                                   
performance.client-io-threads           off    

After upgrade: The option is turned off as expected, earlier it was showing on.

Hence marking it as verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607