Bug 1477091 - [Gluster-block]: Block-create with same name on another volume results in backend deletion of the original block volume. [NEEDINFO]
[Gluster-block]: Block-create with same name on another volume results in bac...
Status: POST
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: gluster-block (Show other bugs)
cns-3.9
Unspecified Unspecified
unspecified Severity medium
: ---
: CNS 3.10
Assigned To: Prasanna Kumar Kalever
Sweta Anandpara
:
Depends On:
Blocks: 1568861 1545104
  Show dependency treegraph
 
Reported: 2017-08-01 04:30 EDT by Sweta Anandpara
Modified: 2018-04-26 13:44 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1545104 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
sanandpa: needinfo? (prasanna.kalever)
sanandpa: needinfo? (pkarampu)


Attachments (Terms of Use)

  None (edit)
Description Sweta Anandpara 2017-08-01 04:30:23 EDT
Description of problem:
=======================
While verifying bug 1456231, I was required to set "GB_GLFS_LRU_COUNT" to 'n' and create blocks on n+1 volumes and time it, to verify if the caching works as expected. 

Multiple times I observed that the block-create command used to fail, with no particular reason mentioned in the logs. A partially-created block used to exist in the volume. I would then execute a block delete command to delete the non-functional block and that used to intermittently fail as well, on some or the other remote node. 

The same command block-create/delete when given again used to succeed, getting the setup back to healthy state. I have been unable to figure out much from the log files, as there is nothing mentioned other than the failure message. In the CLI logs pasted below, I have marked [A] to [E], where the commands failed and succeeded, with me having changed nothing on the setup/config side in-between. 

Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.8.4-35 and gluster-block-0.2.1-6


How reproducible:
=================
Saw it >5 times


Additional info:
=================

[root@dhcp47-121 ~]# time gluster-block create vol1/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:7584d7b9-5fb3-465f-81c1-ca47521a1d19
USERNAME: 7584d7b9-5fb3-465f-81c1-ca47521a1d19
PASSWORD: 6228a2ad-1be2-42c2-af8b-0605b9352c3f
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m15.554s
user	0m0.001s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol1/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:4a6a677a-1f67-4f9d-96fd-894e563e6268
USERNAME: 4a6a677a-1f67-4f9d-96fd-894e563e6268
PASSWORD: 0b4e671f-fc94-4504-b020-0851920683d4
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m15.181s
user	0m0.003s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol2/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M               >> [A] fail
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.121 10.70.47.113 10.70.47.114  
RESULT: FAIL

real	0m11.294s
user	0m0.003s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol2/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M                >> [B] fail
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.114 10.70.47.113 10.70.47.121  
RESULT: FAIL

real	0m7.541s
user	0m0.003s
sys	0m0.011s
[root@dhcp47-121 ~]# time gluster-block create vol3/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:184dc69d-3fd2-4ff4-9a69-62e1c1d9ae78
USERNAME: 184dc69d-3fd2-4ff4-9a69-62e1c1d9ae78
PASSWORD: f3567801-6b64-4466-88ee-bf8cf7b7fbc6
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m20.550s
user	0m0.001s
sys	0m0.012s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol3/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:73e24dff-72f6-4b1c-b619-5dfd9900ba8c
USERNAME: 73e24dff-72f6-4b1c-b619-5dfd9900ba8c
PASSWORD: f30b464d-be9a-4284-80ea-9ff7ff441414
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m16.461s
user	0m0.002s
sys	0m0.025s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol4/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M                     >> [C] fail
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.113 10.70.47.114 10.70.47.121  
RESULT: FAIL

real	0m10.641s
user	0m0.004s
sys	0m0.016s
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-blockd.log 
[root@dhcp47-121 ~]# vim /var/log/audit/audit.log
[root@dhcp47-121 ~]# time gluster-block create vol4/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
BLOCK with name: 'bk1' already EXIST

RESULT:FAIL

real	0m0.030s
user	0m0.000s
sys	0m0.009s
[root@dhcp47-121 ~]# gluster-block list vol4/bk1
Volume vol4/bk1 does not exist
[root@dhcp47-121 ~]# gluster-block list vol4
bk1
[root@dhcp47-121 ~]# gluster-block info vol4/bk1
NAME: bk1
VOLUME: vol4
GBID: 143ec4c2-bb7a-4d50-878e-4c1f743c95b4
SIZE: 1048576
HA: 3
PASSWORD: d03ccc95-708a-4a59-a3cc-5162d4d678a0
BLOCK CONFIG NODE(S):
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# cat /mnt/testvol/block-meta/
bk1_ha1       bk2_ha2       bk3_ha3       bk4_ha3_auth  meta.lock     
[root@dhcp47-121 ~]# cat /mnt/testvol/block-
block-meta/  block-store/ 
[root@dhcp47-121 ~]# cat /mnt/testvol/block-store/
34b427b6-7622-420f-ac52-a08f2cbb2ef5  bc597f3d-82bc-4908-9401-3c931a9fab67  
ac45465f-8104-48e6-8cf1-f4855a6d16cb  f31df0ad-2422-4241-9236-fe83c0e90cb3  
[root@dhcp47-121 ~]# gluster-block list vol1
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol2
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol3
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol4
bk1
[root@dhcp47-121 ~]# mkdir /mnt/vol1
[root@dhcp47-121 ~]# mkdir /mnt/vol2
[root@dhcp47-121 ~]# mkdir /mnt/vol3
[root@dhcp47-121 ~]# mkdir /mnt/vol4
[root@dhcp47-121 ~]# mkdir /mnt/vol5
[root@dhcp47-121 ~]# mkdir /mnt/vol6
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol1 /mnt/vol1
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol2 /mnt/vol2
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol3 /mnt/vol3
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol4 /mnt/vol4
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol5 /mnt/vol5
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol6 /mnt/vol6
[root@dhcp47-121 ~]# cat /mnt/vol1/block-meta/
bk1        bk2        meta.lock  
[root@dhcp47-121 ~]# cat /mnt/vol1/block-meta/bk*
VOLUME: vol1
GBID: 7584d7b9-5fb3-465f-81c1-ca47521a1d19
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: 6228a2ad-1be2-42c2-af8b-0605b9352c3f
10.70.47.121: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.114: CONFIGINPROGRESS
10.70.47.114: CONFIGSUCCESS
10.70.47.114: AUTHENFORCED
10.70.47.113: CONFIGSUCCESS
10.70.47.113: AUTHENFORCED
10.70.47.121: CONFIGSUCCESS
10.70.47.121: AUTHENFORCED
VOLUME: vol1
GBID: 4a6a677a-1f67-4f9d-96fd-894e563e6268
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: 0b4e671f-fc94-4504-b020-0851920683d4
10.70.47.121: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.114: CONFIGINPROGRESS
10.70.47.114: CONFIGSUCCESS
10.70.47.114: AUTHENFORCED
10.70.47.113: CONFIGSUCCESS
10.70.47.113: AUTHENFORCED
10.70.47.121: CONFIGSUCCESS
10.70.47.121: AUTHENFORCED
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# ls /mnt/vol2/block-meta/
bk1  bk2  meta.lock
[root@dhcp47-121 ~]# cat /mnt/vol2/block-meta/bk*
VOLUME: vol2
GBID: eabc08bd-1c5d-4595-9b5b-e24133180d92
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: 573f21dc-1a4a-4c2f-8f34-a6c9f51d931d
10.70.47.121: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.114: CONFIGINPROGRESS
10.70.47.113: CONFIGFAIL
10.70.47.114: CONFIGFAIL
10.70.47.121: CONFIGFAIL
10.70.47.113: CLEANUPINPROGRESS
10.70.47.121: CLEANUPINPROGRESS
10.70.47.114: CLEANUPINPROGRESS
10.70.47.114: CLEANUPFAIL
10.70.47.113: CLEANUPFAIL
10.70.47.121: CLEANUPFAIL
VOLUME: vol2
GBID: 68e26751-c379-4c75-9fa5-baff77efb7f9
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: bcc19bb9-e179-4dbf-ad0a-5a0e4a6f6713
10.70.47.114: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.121: CONFIGINPROGRESS
10.70.47.114: CONFIGFAIL
10.70.47.113: CONFIGFAIL
10.70.47.121: CONFIGFAIL
10.70.47.113: CLEANUPINPROGRESS
10.70.47.121: CLEANUPINPROGRESS
10.70.47.114: CLEANUPINPROGRESS
10.70.47.114: CLEANUPFAIL
10.70.47.113: CLEANUPFAIL
10.70.47.121: CLEANUPFAIL
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# vim /var/lo
local/ lock/  log/   
[root@dhcp47-121 ~]# vim /var/log/gluster-block/
gluster-block-cli.log          gluster-block-configshell.log  gluster-blockd.log             gluster-block-gfapi.log        tcmu-runner-glfs.log           tcmu-runner.log                
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-block-cli.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-block-configshell.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-block-gfapi.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/tcmu-runner.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/tcmu-runner-glfs.log 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# gluster v info vol2
 
Volume Name: vol2
Type: Replicate
Volume ID: c9c63135-7f40-4923-b014-e06a320c844f
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.47.121:/bricks/brick12/vol2_0
Brick2: 10.70.47.113:/bricks/brick12/vol2_1
Brick3: 10.70.47.114:/bricks/brick12/vol2_2
Options Reconfigured:
server.allow-insecure: on
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: enable
performance.readdir-ahead: off
performance.open-behind: off
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
nfs.disable: on
cluster.brick-multiplex: disable
cluster.enable-shared-storage: disable
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol2/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
BLOCK with name: 'bk1' already EXIST

RESULT:FAIL

real	0m0.030s
user	0m0.001s
sys	0m0.011s
[root@dhcp47-121 ~]# gluster-block delete vol2/bk1
SUCCESSFUL ON:   10.70.47.121 10.70.47.113 10.70.47.114
RESULT: SUCCESS
[root@dhcp47-121 ~]# gluster-block delete vol2/bk2                                     >> [D] fail
FAILED ON:   10.70.47.114 10.70.47.113 10.70.47.121
SUCCESSFUL ON: None
RESULT: FAIL
[root@dhcp47-121 ~]# gluster-block list vol2                         
bk2
[root@dhcp47-121 ~]# gluster-block delete vol2/bk2                                    >> [D] success
SUCCESSFUL ON:   10.70.47.114 10.70.47.113 10.70.47.121
RESULT: SUCCESS
[root@dhcp47-121 ~]# gluster-block list vol2
*Nil*
[root@dhcp47-121 ~]# time gluster-block create vol2/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M             >> [A] success
IQN: iqn.2016-12.org.gluster-block:81e7ff54-3559-4cb1-b8a6-6afb8b758719
USERNAME: 81e7ff54-3559-4cb1-b8a6-6afb8b758719
PASSWORD: 5fc15341-afb5-462e-9ffc-fae5207b0c71
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m13.653s
user	0m0.001s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol2/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M               >> [B] success
IQN: iqn.2016-12.org.gluster-block:2fb4c04a-043d-4aee-979f-9ab1c9ca6226
USERNAME: 2fb4c04a-043d-4aee-979f-9ab1c9ca6226
PASSWORD: 393aa7b2-4bfc-4c12-a750-3a2ad81a88ac
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m14.674s
user	0m0.006s
sys	0m0.013s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# gluster-block list vol3
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol4
bk1
[root@dhcp47-121 ~]# gluster-block delete vol4/bk1                                    >> [E] fail
FAILED ON:   10.70.47.113 10.70.47.114 10.70.47.121
SUCCESSFUL ON: None
RESULT: FAIL
[root@dhcp47-121 ~]# gluster-block delete vol4/bk1                                    >> [E] success
SUCCESSFUL ON:   10.70.47.113 10.70.47.114 10.70.47.121
RESULT: SUCCESS
[root@dhcp47-121 ~]# gluster-block list vol4
*Nil*
[root@dhcp47-121 ~]# time gluster-block create vol4/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M                  >> [C] success
IQN: iqn.2016-12.org.gluster-block:a0e01668-b754-48a5-b2b9-3cbedc8160ad
USERNAME: a0e01668-b754-48a5-b2b9-3cbedc8160ad
PASSWORD: 85a46002-3194-4b97-a197-3d1f77792dfa
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m18.967s
user	0m0.005s
sys	0m0.010s
[root@dhcp47-121 ~]# time gluster-block create vol4/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.113 10.70.47.121 10.70.47.114  
RESULT: FAIL

real	0m7.590s
user	0m0.002s
sys	0m0.018s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol5/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.113 10.70.47.114 10.70.47.121  
RESULT: FAIL

real	0m11.757s
user	0m0.005s
sys	0m0.015s
[root@dhcp47-121 ~]#
Comment 2 Sweta Anandpara 2017-08-01 04:41:42 EDT
I have run gluster-block creates and deletes in a loop, while verifying another bug 1452936 (with gluster-block-0.2.1-6 and glusterfs-3.8.4-33) , and I had not hit this then. 

Atin, is there a chance of any patches between 3.8.4-33 and 3.8.4-35 to have impacted gluster-block behaviour?
Comment 3 Sweta Anandpara 2017-08-01 04:44:50 EDT
(In reply to Sweta Anandpara from comment #2)
> I have run gluster-block creates and deletes in a loop, while verifying
> another bug 1452936 (with gluster-block-0.2.1-6 and glusterfs-3.8.4-33) ,
> and I had not hit this then. 
> 
> Atin, is there a chance of any patches between 3.8.4-33 and 3.8.4-35 to have
> impacted gluster-block behaviour?

If not, then it must be something to do with the steps that I am following for verifying bz 1456231.
Comment 4 Sweta Anandpara 2017-08-01 05:04:58 EDT
Apologies. Correction in the bug number mentioned in Description and Comment3 - bz 1464421
Comment 13 Sweta Anandpara 2017-08-03 06:58:51 EDT
Resetting the blocker flag as this bug will not be hit in CNS environment.

Note You need to log in before you can comment on or make changes to this bug.