Bug 1283861

Description of problem:
======================
Even in test mode(which is the default), the watermark hi and low levels are seen by default in the volfile. This is neither required nor recommeneded.
In test mode, watermark should be shown up only after user dedicatedly enables by setting some values.

NOTE: Not marking as blocker or high severe/priority as they are not being in effect and hindering the desired functionality

EG:In the below volume, where it is in test mode, it can be seen that the vol file has watermarks
[root@zod ~]# gluster v info distrep
 
Volume Name: distrep
Type: Tier
Volume ID: fb6cf57c-6969-42b4-97cf-8ddaa0f5fc22
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: yarrow:/rhs/brick7/distrep_hot
Brick2: zod:/rhs/brick7/distrep_hot
Brick3: yarrow:/rhs/brick6/distrep_hot
Brick4: zod:/rhs/brick6/distrep_hot
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick5: zod:/rhs/brick1/distrep
Brick6: yarrow:/rhs/brick1/distrep
Brick7: zod:/rhs/brick2/distrep
Brick8: yarrow:/rhs/brick2/distrep
Options Reconfigured:
features.ctr-enabled: on
performance.readdir-ahead: on
[root@zod ~]# gluster v get distrep|grep water
Usage: volume get <VOLNAME> <key|all>
[root@zod ~]# gluster v get distrep all |grep water
cluster.watermark-hi                    90                                      
cluster.watermark-low                   75                                      
[root@zod ~]# gluster v get distrep all |grep mode
cluster.read-hash-mode                  1                                       
storage.batch-fsync-mode                reverse-fsync                           
cluster.tier-mode                       test                                  




VOLFILE:
[root@zod distrep]# pwd
/var/lib/glusterd/vols/distrep
[root@zod distrep]# cat distrep.tcp-fuse.vol
volume distrep-client-7
    type protocol/client
    option send-gids true
    option transport-type tcp
    option remote-subvolume /rhs/brick7/distrep_hot
    option remote-host yarrow
    option ping-timeout 42
end-volume

volume distrep-client-6
    type protocol/client
    option send-gids true
    option transport-type tcp
    option remote-subvolume /rhs/brick7/distrep_hot
    option remote-host zod
    option ping-timeout 42
end-volume

volume distrep-client-5
    type protocol/client
    option send-gids true
    option transport-type tcp
    option remote-subvolume /rhs/brick6/distrep_hot
    option remote-host yarrow
    option ping-timeout 42
end-volume

volume distrep-client-4
    type protocol/client
    option send-gids true
    option transport-type tcp
    option remote-subvolume /rhs/brick6/distrep_hot
    option remote-host zod
    option ping-timeout 42
end-volume

volume distrep-client-0
    type protocol/client
    option send-gids true
    option transport-type tcp
    option remote-subvolume /rhs/brick1/distrep
    option remote-host zod
    option ping-timeout 42
end-volume

volume distrep-client-1
    type protocol/client
    option send-gids true
    option transport-type tcp
    option remote-subvolume /rhs/brick1/distrep
    option remote-host yarrow
    option ping-timeout 42
end-volume

volume distrep-client-2
    type protocol/client
    option send-gids true
    option transport-type tcp
    option remote-subvolume /rhs/brick2/distrep
    option remote-host zod
    option ping-timeout 42
end-volume

volume distrep-client-3
    type protocol/client
    option send-gids true
    option transport-type tcp
    option remote-subvolume /rhs/brick2/distrep
    option remote-host yarrow
    option ping-timeout 42
end-volume

volume distrep-replicate-0
    type cluster/replicate
    subvolumes distrep-client-0 distrep-client-1
end-volume

volume distrep-replicate-1
    type cluster/replicate
    subvolumes distrep-client-2 distrep-client-3
end-volume

volume distrep-cold-dht
    type cluster/distribute
    subvolumes distrep-replicate-0 distrep-replicate-1
end-volume

volume distrep-replicate-2
    type cluster/replicate
    subvolumes distrep-client-7 distrep-client-6
end-volume

volume distrep-replicate-3
    type cluster/replicate
    subvolumes distrep-client-5 distrep-client-4
end-volume

volume distrep-hot-dht
    type cluster/distribute
    subvolumes distrep-replicate-2 distrep-replicate-3
end-volume

volume distrep-tier-dht
    type cluster/tier
    option tier-max-files 5000
    option tier-max-mb 1000
    option tier-mode test
    option watermark-low 75
    option watermark-hi 90
    option tier-demote-frequency 120
    option tier-promote-frequency 120
    option read-freq-threshold 0
    option write-freq-threshold 0
    option xattr-name trusted.tier.tier-dht
    option rule distrep-hot-dht
    subvolumes distrep-cold-dht distrep-hot-dht
end-volume

volume distrep-write-behind
    type performance/write-behind
    subvolumes distrep-tier-dht
end-volume

volume distrep-read-ahead
    type performance/read-ahead
    subvolumes distrep-write-behind
end-volume

volume distrep-readdir-ahead
    type performance/readdir-ahead
    subvolumes distrep-read-ahead
end-volume

volume distrep-io-cache
    type performance/io-cache
    subvolumes distrep-readdir-ahead
end-volume

volume distrep-quick-read
    type performance/quick-read
    subvolumes distrep-io-cache
end-volume

volume distrep-open-behind
    type performance/open-behind
    subvolumes distrep-quick-read
end-volume

volume distrep-md-cache
    type performance/md-cache
    subvolumes distrep-open-behind
end-volume

volume distrep
    type debug/io-stats
    option count-fop-hits off
    option latency-measurement off
    subvolumes distrep-md-cache
end-volume



Version-Release number of selected component (if applicable):
========================================
[root@zod distrep]# rpm -qa|grep gluste
rglusterfs-libs-3.7.5-6.el7rhgs.x86_64
glusterfs-fuse-3.7.5-6.el7rhgs.x86_64
glusterfs-3.7.5-6.el7rhgs.x86_64
glusterfs-server-3.7.5-6.el7rhgs.x86_64
glusterfs-client-xlators-3.7.5-6.el7rhgs.x86_64
glusterfs-cli-3.7.5-6.el7rhgs.x86_64
glusterfs-api-3.7.5-6.el7rhgs.x86_64
glusterfs-debuginfo-3.7.5-6.el7rhgs.x86_64
[root@zod distrep]# r



How reproducible:
===============
always

Steps to Reproduce:
==================
1.create a tiered volume
2.now as the volume is in test mode by default or change it to test mode
3. now see the volfile
4. it shows watermark limits which is not required
3.