Bug 1477091

Summary: [Gluster-block]: Block-create with same name on another volume results in backend deletion of the original block volume.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Sweta Anandpara <sanandpa>
Component: gluster-blockAssignee: Prasanna Kumar Kalever <prasanna.kalever>
Status: CLOSED ERRATA QA Contact: Sweta Anandpara <sanandpa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: cns-3.9CC: amukherj, kramdoss, pkarampu, prasanna.kalever, rcyriac, rhs-bugs, storage-qa-internal
Target Milestone: ---   
Target Release: CNS 3.10   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: gluster-block-0.2.1-14.1.el7rhgs Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1545104 (view as bug list) Environment:
Last Closed: 2018-09-12 09:25:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1545104, 1568861    

Description Sweta Anandpara 2017-08-01 08:30:23 UTC 
Description of problem:
=======================
While verifying bug 1456231, I was required to set "GB_GLFS_LRU_COUNT" to 'n' and create blocks on n+1 volumes and time it, to verify if the caching works as expected. 

Multiple times I observed that the block-create command used to fail, with no particular reason mentioned in the logs. A partially-created block used to exist in the volume. I would then execute a block delete command to delete the non-functional block and that used to intermittently fail as well, on some or the other remote node. 

The same command block-create/delete when given again used to succeed, getting the setup back to healthy state. I have been unable to figure out much from the log files, as there is nothing mentioned other than the failure message. In the CLI logs pasted below, I have marked [A] to [E], where the commands failed and succeeded, with me having changed nothing on the setup/config side in-between. 

Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.8.4-35 and gluster-block-0.2.1-6


How reproducible:
=================
Saw it >5 times


Additional info:
=================

[root@dhcp47-121 ~]# time gluster-block create vol1/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:7584d7b9-5fb3-465f-81c1-ca47521a1d19
USERNAME: 7584d7b9-5fb3-465f-81c1-ca47521a1d19
PASSWORD: 6228a2ad-1be2-42c2-af8b-0605b9352c3f
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m15.554s
user	0m0.001s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol1/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:4a6a677a-1f67-4f9d-96fd-894e563e6268
USERNAME: 4a6a677a-1f67-4f9d-96fd-894e563e6268
PASSWORD: 0b4e671f-fc94-4504-b020-0851920683d4
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m15.181s
user	0m0.003s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol2/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M               >> [A] fail
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.121 10.70.47.113 10.70.47.114  
RESULT: FAIL

real	0m11.294s
user	0m0.003s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol2/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M                >> [B] fail
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.114 10.70.47.113 10.70.47.121  
RESULT: FAIL

real	0m7.541s
user	0m0.003s
sys	0m0.011s
[root@dhcp47-121 ~]# time gluster-block create vol3/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:184dc69d-3fd2-4ff4-9a69-62e1c1d9ae78
USERNAME: 184dc69d-3fd2-4ff4-9a69-62e1c1d9ae78
PASSWORD: f3567801-6b64-4466-88ee-bf8cf7b7fbc6
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m20.550s
user	0m0.001s
sys	0m0.012s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol3/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:73e24dff-72f6-4b1c-b619-5dfd9900ba8c
USERNAME: 73e24dff-72f6-4b1c-b619-5dfd9900ba8c
PASSWORD: f30b464d-be9a-4284-80ea-9ff7ff441414
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m16.461s
user	0m0.002s
sys	0m0.025s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol4/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M                     >> [C] fail
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.113 10.70.47.114 10.70.47.121  
RESULT: FAIL

real	0m10.641s
user	0m0.004s
sys	0m0.016s
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-blockd.log 
[root@dhcp47-121 ~]# vim /var/log/audit/audit.log
[root@dhcp47-121 ~]# time gluster-block create vol4/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
BLOCK with name: 'bk1' already EXIST

RESULT:FAIL

real	0m0.030s
user	0m0.000s
sys	0m0.009s
[root@dhcp47-121 ~]# gluster-block list vol4/bk1
Volume vol4/bk1 does not exist
[root@dhcp47-121 ~]# gluster-block list vol4
bk1
[root@dhcp47-121 ~]# gluster-block info vol4/bk1
NAME: bk1
VOLUME: vol4
GBID: 143ec4c2-bb7a-4d50-878e-4c1f743c95b4
SIZE: 1048576
HA: 3
PASSWORD: d03ccc95-708a-4a59-a3cc-5162d4d678a0
BLOCK CONFIG NODE(S):
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# cat /mnt/testvol/block-meta/
bk1_ha1       bk2_ha2       bk3_ha3       bk4_ha3_auth  meta.lock     
[root@dhcp47-121 ~]# cat /mnt/testvol/block-
block-meta/  block-store/ 
[root@dhcp47-121 ~]# cat /mnt/testvol/block-store/
34b427b6-7622-420f-ac52-a08f2cbb2ef5  bc597f3d-82bc-4908-9401-3c931a9fab67  
ac45465f-8104-48e6-8cf1-f4855a6d16cb  f31df0ad-2422-4241-9236-fe83c0e90cb3  
[root@dhcp47-121 ~]# gluster-block list vol1
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol2
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol3
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol4
bk1
[root@dhcp47-121 ~]# mkdir /mnt/vol1
[root@dhcp47-121 ~]# mkdir /mnt/vol2
[root@dhcp47-121 ~]# mkdir /mnt/vol3
[root@dhcp47-121 ~]# mkdir /mnt/vol4
[root@dhcp47-121 ~]# mkdir /mnt/vol5
[root@dhcp47-121 ~]# mkdir /mnt/vol6
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol1 /mnt/vol1
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol2 /mnt/vol2
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol3 /mnt/vol3
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol4 /mnt/vol4
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol5 /mnt/vol5
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol6 /mnt/vol6
[root@dhcp47-121 ~]# cat /mnt/vol1/block-meta/
bk1        bk2        meta.lock  
[root@dhcp47-121 ~]# cat /mnt/vol1/block-meta/bk*
VOLUME: vol1
GBID: 7584d7b9-5fb3-465f-81c1-ca47521a1d19
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: 6228a2ad-1be2-42c2-af8b-0605b9352c3f
10.70.47.121: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.114: CONFIGINPROGRESS
10.70.47.114: CONFIGSUCCESS
10.70.47.114: AUTHENFORCED
10.70.47.113: CONFIGSUCCESS
10.70.47.113: AUTHENFORCED
10.70.47.121: CONFIGSUCCESS
10.70.47.121: AUTHENFORCED
VOLUME: vol1
GBID: 4a6a677a-1f67-4f9d-96fd-894e563e6268
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: 0b4e671f-fc94-4504-b020-0851920683d4
10.70.47.121: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.114: CONFIGINPROGRESS
10.70.47.114: CONFIGSUCCESS
10.70.47.114: AUTHENFORCED
10.70.47.113: CONFIGSUCCESS
10.70.47.113: AUTHENFORCED
10.70.47.121: CONFIGSUCCESS
10.70.47.121: AUTHENFORCED
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# ls /mnt/vol2/block-meta/
bk1  bk2  meta.lock
[root@dhcp47-121 ~]# cat /mnt/vol2/block-meta/bk*
VOLUME: vol2
GBID: eabc08bd-1c5d-4595-9b5b-e24133180d92
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: 573f21dc-1a4a-4c2f-8f34-a6c9f51d931d
10.70.47.121: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.114: CONFIGINPROGRESS
10.70.47.113: CONFIGFAIL
10.70.47.114: CONFIGFAIL
10.70.47.121: CONFIGFAIL
10.70.47.113: CLEANUPINPROGRESS
10.70.47.121: CLEANUPINPROGRESS
10.70.47.114: CLEANUPINPROGRESS
10.70.47.114: CLEANUPFAIL
10.70.47.113: CLEANUPFAIL
10.70.47.121: CLEANUPFAIL
VOLUME: vol2
GBID: 68e26751-c379-4c75-9fa5-baff77efb7f9
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: bcc19bb9-e179-4dbf-ad0a-5a0e4a6f6713
10.70.47.114: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.121: CONFIGINPROGRESS
10.70.47.114: CONFIGFAIL
10.70.47.113: CONFIGFAIL
10.70.47.121: CONFIGFAIL
10.70.47.113: CLEANUPINPROGRESS
10.70.47.121: CLEANUPINPROGRESS
10.70.47.114: CLEANUPINPROGRESS
10.70.47.114: CLEANUPFAIL
10.70.47.113: CLEANUPFAIL
10.70.47.121: CLEANUPFAIL
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# vim /var/lo
local/ lock/  log/   
[root@dhcp47-121 ~]# vim /var/log/gluster-block/
gluster-block-cli.log          gluster-block-configshell.log  gluster-blockd.log             gluster-block-gfapi.log        tcmu-runner-glfs.log           tcmu-runner.log                
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-block-cli.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-block-configshell.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-block-gfapi.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/tcmu-runner.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/tcmu-runner-glfs.log 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# gluster v info vol2
 
Volume Name: vol2
Type: Replicate
Volume ID: c9c63135-7f40-4923-b014-e06a320c844f
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.47.121:/bricks/brick12/vol2_0
Brick2: 10.70.47.113:/bricks/brick12/vol2_1
Brick3: 10.70.47.114:/bricks/brick12/vol2_2
Options Reconfigured:
server.allow-insecure: on
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: enable
performance.readdir-ahead: off
performance.open-behind: off
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
nfs.disable: on
cluster.brick-multiplex: disable
cluster.enable-shared-storage: disable
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol2/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
BLOCK with name: 'bk1' already EXIST

RESULT:FAIL

real	0m0.030s
user	0m0.001s
sys	0m0.011s
[root@dhcp47-121 ~]# gluster-block delete vol2/bk1
SUCCESSFUL ON:   10.70.47.121 10.70.47.113 10.70.47.114
RESULT: SUCCESS
[root@dhcp47-121 ~]# gluster-block delete vol2/bk2                                     >> [D] fail
FAILED ON:   10.70.47.114 10.70.47.113 10.70.47.121
SUCCESSFUL ON: None
RESULT: FAIL
[root@dhcp47-121 ~]# gluster-block list vol2                         
bk2
[root@dhcp47-121 ~]# gluster-block delete vol2/bk2                                    >> [D] success
SUCCESSFUL ON:   10.70.47.114 10.70.47.113 10.70.47.121
RESULT: SUCCESS
[root@dhcp47-121 ~]# gluster-block list vol2
*Nil*
[root@dhcp47-121 ~]# time gluster-block create vol2/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M             >> [A] success
IQN: iqn.2016-12.org.gluster-block:81e7ff54-3559-4cb1-b8a6-6afb8b758719
USERNAME: 81e7ff54-3559-4cb1-b8a6-6afb8b758719
PASSWORD: 5fc15341-afb5-462e-9ffc-fae5207b0c71
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m13.653s
user	0m0.001s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol2/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M               >> [B] success
IQN: iqn.2016-12.org.gluster-block:2fb4c04a-043d-4aee-979f-9ab1c9ca6226
USERNAME: 2fb4c04a-043d-4aee-979f-9ab1c9ca6226
PASSWORD: 393aa7b2-4bfc-4c12-a750-3a2ad81a88ac
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m14.674s
user	0m0.006s
sys	0m0.013s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# gluster-block list vol3
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol4
bk1
[root@dhcp47-121 ~]# gluster-block delete vol4/bk1                                    >> [E] fail
FAILED ON:   10.70.47.113 10.70.47.114 10.70.47.121
SUCCESSFUL ON: None
RESULT: FAIL
[root@dhcp47-121 ~]# gluster-block delete vol4/bk1                                    >> [E] success
SUCCESSFUL ON:   10.70.47.113 10.70.47.114 10.70.47.121
RESULT: SUCCESS
[root@dhcp47-121 ~]# gluster-block list vol4
*Nil*
[root@dhcp47-121 ~]# time gluster-block create vol4/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M                  >> [C] success
IQN: iqn.2016-12.org.gluster-block:a0e01668-b754-48a5-b2b9-3cbedc8160ad
USERNAME: a0e01668-b754-48a5-b2b9-3cbedc8160ad
PASSWORD: 85a46002-3194-4b97-a197-3d1f77792dfa
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m18.967s
user	0m0.005s
sys	0m0.010s
[root@dhcp47-121 ~]# time gluster-block create vol4/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.113 10.70.47.121 10.70.47.114  
RESULT: FAIL

real	0m7.590s
user	0m0.002s
sys	0m0.018s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol5/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.113 10.70.47.114 10.70.47.121  
RESULT: FAIL

real	0m11.757s
user	0m0.005s
sys	0m0.015s
[root@dhcp47-121 ~]#
Comment 2 Sweta Anandpara 2017-08-01 08:41:42 UTC 
I have run gluster-block creates and deletes in a loop, while verifying another bug 1452936 (with gluster-block-0.2.1-6 and glusterfs-3.8.4-33) , and I had not hit this then. 

Atin, is there a chance of any patches between 3.8.4-33 and 3.8.4-35 to have impacted gluster-block behaviour?
Comment 3 Sweta Anandpara 2017-08-01 08:44:50 UTC 
(In reply to Sweta Anandpara from comment #2)
> I have run gluster-block creates and deletes in a loop, while verifying
> another bug 1452936 (with gluster-block-0.2.1-6 and glusterfs-3.8.4-33) ,
> and I had not hit this then. 
> 
> Atin, is there a chance of any patches between 3.8.4-33 and 3.8.4-35 to have
> impacted gluster-block behaviour?

If not, then it must be something to do with the steps that I am following for verifying bz 1456231.
Comment 4 Sweta Anandpara 2017-08-01 09:04:58 UTC 
Apologies. Correction in the bug number mentioned in Description and Comment3 - bz 1464421
Comment 13 Sweta Anandpara 2017-08-03 10:58:51 UTC 
Resetting the blocker flag as this bug will not be hit in CNS environment.
Comment 21 errata-xmlrpc 2018-09-12 09:25:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2691