1477091 – [Gluster-block]: Block-create with same name on another volume results in backend deletion of the original block volume.

Bug 1477091 - [Gluster-block]: Block-create with same name on another volume results in backend deletion of the original block volume.

Summary: [Gluster-block]: Block-create with same name on another volume results in bac...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	gluster-block
Sub Component:
Version:	cns-3.9
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	CNS 3.10
Assignee:	Prasanna Kumar Kalever
QA Contact:	Sweta Anandpara
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1545104 1568861
TreeView+	depends on / blocked

Reported:	2017-08-01 08:30 UTC by Sweta Anandpara
Modified:	2018-09-12 09:26 UTC (History)
CC List:	7 users (show)
Fixed In Version:	gluster-block-0.2.1-14.1.el7rhgs
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1545104 (view as bug list)
Environment:
Last Closed:	2018-09-12 09:25:17 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHEA-2018:2691	0	None	None	None	2018-09-12 09:26:31 UTC

Description Sweta Anandpara 2017-08-01 08:30:23 UTC

Description of problem:
=======================
While verifying bug 1456231, I was required to set "GB_GLFS_LRU_COUNT" to 'n' and create blocks on n+1 volumes and time it, to verify if the caching works as expected. 

Multiple times I observed that the block-create command used to fail, with no particular reason mentioned in the logs. A partially-created block used to exist in the volume. I would then execute a block delete command to delete the non-functional block and that used to intermittently fail as well, on some or the other remote node. 

The same command block-create/delete when given again used to succeed, getting the setup back to healthy state. I have been unable to figure out much from the log files, as there is nothing mentioned other than the failure message. In the CLI logs pasted below, I have marked [A] to [E], where the commands failed and succeeded, with me having changed nothing on the setup/config side in-between. 

Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.8.4-35 and gluster-block-0.2.1-6


How reproducible:
=================
Saw it >5 times


Additional info:
=================

[root@dhcp47-121 ~]# time gluster-block create vol1/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:7584d7b9-5fb3-465f-81c1-ca47521a1d19
USERNAME: 7584d7b9-5fb3-465f-81c1-ca47521a1d19
PASSWORD: 6228a2ad-1be2-42c2-af8b-0605b9352c3f
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m15.554s
user	0m0.001s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol1/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:4a6a677a-1f67-4f9d-96fd-894e563e6268
USERNAME: 4a6a677a-1f67-4f9d-96fd-894e563e6268
PASSWORD: 0b4e671f-fc94-4504-b020-0851920683d4
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m15.181s
user	0m0.003s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol2/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M               >> [A] fail
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.121 10.70.47.113 10.70.47.114  
RESULT: FAIL

real	0m11.294s
user	0m0.003s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol2/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M                >> [B] fail
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.114 10.70.47.113 10.70.47.121  
RESULT: FAIL

real	0m7.541s
user	0m0.003s
sys	0m0.011s
[root@dhcp47-121 ~]# time gluster-block create vol3/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:184dc69d-3fd2-4ff4-9a69-62e1c1d9ae78
USERNAME: 184dc69d-3fd2-4ff4-9a69-62e1c1d9ae78
PASSWORD: f3567801-6b64-4466-88ee-bf8cf7b7fbc6
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m20.550s
user	0m0.001s
sys	0m0.012s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol3/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: iqn.2016-12.org.gluster-block:73e24dff-72f6-4b1c-b619-5dfd9900ba8c
USERNAME: 73e24dff-72f6-4b1c-b619-5dfd9900ba8c
PASSWORD: f30b464d-be9a-4284-80ea-9ff7ff441414
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m16.461s
user	0m0.002s
sys	0m0.025s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol4/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M                     >> [C] fail
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.113 10.70.47.114 10.70.47.121  
RESULT: FAIL

real	0m10.641s
user	0m0.004s
sys	0m0.016s
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-blockd.log 
[root@dhcp47-121 ~]# vim /var/log/audit/audit.log
[root@dhcp47-121 ~]# time gluster-block create vol4/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
BLOCK with name: 'bk1' already EXIST

RESULT:FAIL

real	0m0.030s
user	0m0.000s
sys	0m0.009s
[root@dhcp47-121 ~]# gluster-block list vol4/bk1
Volume vol4/bk1 does not exist
[root@dhcp47-121 ~]# gluster-block list vol4
bk1
[root@dhcp47-121 ~]# gluster-block info vol4/bk1
NAME: bk1
VOLUME: vol4
GBID: 143ec4c2-bb7a-4d50-878e-4c1f743c95b4
SIZE: 1048576
HA: 3
PASSWORD: d03ccc95-708a-4a59-a3cc-5162d4d678a0
BLOCK CONFIG NODE(S):
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# cat /mnt/testvol/block-meta/
bk1_ha1       bk2_ha2       bk3_ha3       bk4_ha3_auth  meta.lock     
[root@dhcp47-121 ~]# cat /mnt/testvol/block-
block-meta/  block-store/ 
[root@dhcp47-121 ~]# cat /mnt/testvol/block-store/
34b427b6-7622-420f-ac52-a08f2cbb2ef5  bc597f3d-82bc-4908-9401-3c931a9fab67  
ac45465f-8104-48e6-8cf1-f4855a6d16cb  f31df0ad-2422-4241-9236-fe83c0e90cb3  
[root@dhcp47-121 ~]# gluster-block list vol1
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol2
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol3
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol4
bk1
[root@dhcp47-121 ~]# mkdir /mnt/vol1
[root@dhcp47-121 ~]# mkdir /mnt/vol2
[root@dhcp47-121 ~]# mkdir /mnt/vol3
[root@dhcp47-121 ~]# mkdir /mnt/vol4
[root@dhcp47-121 ~]# mkdir /mnt/vol5
[root@dhcp47-121 ~]# mkdir /mnt/vol6
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol1 /mnt/vol1
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol2 /mnt/vol2
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol3 /mnt/vol3
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol4 /mnt/vol4
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol5 /mnt/vol5
[root@dhcp47-121 ~]# mount -t glusterfs 10.70.47.121:vol6 /mnt/vol6
[root@dhcp47-121 ~]# cat /mnt/vol1/block-meta/
bk1        bk2        meta.lock  
[root@dhcp47-121 ~]# cat /mnt/vol1/block-meta/bk*
VOLUME: vol1
GBID: 7584d7b9-5fb3-465f-81c1-ca47521a1d19
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: 6228a2ad-1be2-42c2-af8b-0605b9352c3f
10.70.47.121: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.114: CONFIGINPROGRESS
10.70.47.114: CONFIGSUCCESS
10.70.47.114: AUTHENFORCED
10.70.47.113: CONFIGSUCCESS
10.70.47.113: AUTHENFORCED
10.70.47.121: CONFIGSUCCESS
10.70.47.121: AUTHENFORCED
VOLUME: vol1
GBID: 4a6a677a-1f67-4f9d-96fd-894e563e6268
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: 0b4e671f-fc94-4504-b020-0851920683d4
10.70.47.121: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.114: CONFIGINPROGRESS
10.70.47.114: CONFIGSUCCESS
10.70.47.114: AUTHENFORCED
10.70.47.113: CONFIGSUCCESS
10.70.47.113: AUTHENFORCED
10.70.47.121: CONFIGSUCCESS
10.70.47.121: AUTHENFORCED
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# ls /mnt/vol2/block-meta/
bk1  bk2  meta.lock
[root@dhcp47-121 ~]# cat /mnt/vol2/block-meta/bk*
VOLUME: vol2
GBID: eabc08bd-1c5d-4595-9b5b-e24133180d92
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: 573f21dc-1a4a-4c2f-8f34-a6c9f51d931d
10.70.47.121: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.114: CONFIGINPROGRESS
10.70.47.113: CONFIGFAIL
10.70.47.114: CONFIGFAIL
10.70.47.121: CONFIGFAIL
10.70.47.113: CLEANUPINPROGRESS
10.70.47.121: CLEANUPINPROGRESS
10.70.47.114: CLEANUPINPROGRESS
10.70.47.114: CLEANUPFAIL
10.70.47.113: CLEANUPFAIL
10.70.47.121: CLEANUPFAIL
VOLUME: vol2
GBID: 68e26751-c379-4c75-9fa5-baff77efb7f9
SIZE: 1048576
HA: 3
ENTRYCREATE: INPROGRESS
ENTRYCREATE: SUCCESS
PASSWORD: bcc19bb9-e179-4dbf-ad0a-5a0e4a6f6713
10.70.47.114: CONFIGINPROGRESS
10.70.47.113: CONFIGINPROGRESS
10.70.47.121: CONFIGINPROGRESS
10.70.47.114: CONFIGFAIL
10.70.47.113: CONFIGFAIL
10.70.47.121: CONFIGFAIL
10.70.47.113: CLEANUPINPROGRESS
10.70.47.121: CLEANUPINPROGRESS
10.70.47.114: CLEANUPINPROGRESS
10.70.47.114: CLEANUPFAIL
10.70.47.113: CLEANUPFAIL
10.70.47.121: CLEANUPFAIL
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# vim /var/lo
local/ lock/  log/   
[root@dhcp47-121 ~]# vim /var/log/gluster-block/
gluster-block-cli.log          gluster-block-configshell.log  gluster-blockd.log             gluster-block-gfapi.log        tcmu-runner-glfs.log           tcmu-runner.log                
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-block-cli.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-block-configshell.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/gluster-block-gfapi.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/tcmu-runner.log 
[root@dhcp47-121 ~]# vim /var/log/gluster-block/tcmu-runner-glfs.log 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# gluster v info vol2
 
Volume Name: vol2
Type: Replicate
Volume ID: c9c63135-7f40-4923-b014-e06a320c844f
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.47.121:/bricks/brick12/vol2_0
Brick2: 10.70.47.113:/bricks/brick12/vol2_1
Brick3: 10.70.47.114:/bricks/brick12/vol2_2
Options Reconfigured:
server.allow-insecure: on
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: enable
performance.readdir-ahead: off
performance.open-behind: off
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
nfs.disable: on
cluster.brick-multiplex: disable
cluster.enable-shared-storage: disable
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol2/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
BLOCK with name: 'bk1' already EXIST

RESULT:FAIL

real	0m0.030s
user	0m0.001s
sys	0m0.011s
[root@dhcp47-121 ~]# gluster-block delete vol2/bk1
SUCCESSFUL ON:   10.70.47.121 10.70.47.113 10.70.47.114
RESULT: SUCCESS
[root@dhcp47-121 ~]# gluster-block delete vol2/bk2                                     >> [D] fail
FAILED ON:   10.70.47.114 10.70.47.113 10.70.47.121
SUCCESSFUL ON: None
RESULT: FAIL
[root@dhcp47-121 ~]# gluster-block list vol2                         
bk2
[root@dhcp47-121 ~]# gluster-block delete vol2/bk2                                    >> [D] success
SUCCESSFUL ON:   10.70.47.114 10.70.47.113 10.70.47.121
RESULT: SUCCESS
[root@dhcp47-121 ~]# gluster-block list vol2
*Nil*
[root@dhcp47-121 ~]# time gluster-block create vol2/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M             >> [A] success
IQN: iqn.2016-12.org.gluster-block:81e7ff54-3559-4cb1-b8a6-6afb8b758719
USERNAME: 81e7ff54-3559-4cb1-b8a6-6afb8b758719
PASSWORD: 5fc15341-afb5-462e-9ffc-fae5207b0c71
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m13.653s
user	0m0.001s
sys	0m0.016s
[root@dhcp47-121 ~]# time gluster-block create vol2/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M               >> [B] success
IQN: iqn.2016-12.org.gluster-block:2fb4c04a-043d-4aee-979f-9ab1c9ca6226
USERNAME: 2fb4c04a-043d-4aee-979f-9ab1c9ca6226
PASSWORD: 393aa7b2-4bfc-4c12-a750-3a2ad81a88ac
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m14.674s
user	0m0.006s
sys	0m0.013s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# gluster-block list vol3
bk1
bk2
[root@dhcp47-121 ~]# gluster-block list vol4
bk1
[root@dhcp47-121 ~]# gluster-block delete vol4/bk1                                    >> [E] fail
FAILED ON:   10.70.47.113 10.70.47.114 10.70.47.121
SUCCESSFUL ON: None
RESULT: FAIL
[root@dhcp47-121 ~]# gluster-block delete vol4/bk1                                    >> [E] success
SUCCESSFUL ON:   10.70.47.113 10.70.47.114 10.70.47.121
RESULT: SUCCESS
[root@dhcp47-121 ~]# gluster-block list vol4
*Nil*
[root@dhcp47-121 ~]# time gluster-block create vol4/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M                  >> [C] success
IQN: iqn.2016-12.org.gluster-block:a0e01668-b754-48a5-b2b9-3cbedc8160ad
USERNAME: a0e01668-b754-48a5-b2b9-3cbedc8160ad
PASSWORD: 85a46002-3194-4b97-a197-3d1f77792dfa
PORTAL(S):  10.70.47.121:3260 10.70.47.113:3260 10.70.47.114:3260
RESULT: SUCCESS

real	0m18.967s
user	0m0.005s
sys	0m0.010s
[root@dhcp47-121 ~]# time gluster-block create vol4/bk2 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.113 10.70.47.121 10.70.47.114  
RESULT: FAIL

real	0m7.590s
user	0m0.002s
sys	0m0.018s
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# 
[root@dhcp47-121 ~]# time gluster-block create vol5/bk1 ha 3 auth enable 10.70.47.121,10.70.47.113,10.70.47.114 1M
IQN: -
(null)PORTAL(S): -
ROLLBACK ON: 10.70.47.113 10.70.47.114 10.70.47.121  
RESULT: FAIL

real	0m11.757s
user	0m0.005s
sys	0m0.015s
[root@dhcp47-121 ~]#

Comment 2 Sweta Anandpara 2017-08-01 08:41:42 UTC

I have run gluster-block creates and deletes in a loop, while verifying another bug 1452936 (with gluster-block-0.2.1-6 and glusterfs-3.8.4-33) , and I had not hit this then. 

Atin, is there a chance of any patches between 3.8.4-33 and 3.8.4-35 to have impacted gluster-block behaviour?

Comment 3 Sweta Anandpara 2017-08-01 08:44:50 UTC

(In reply to Sweta Anandpara from comment #2)
> I have run gluster-block creates and deletes in a loop, while verifying
> another bug 1452936 (with gluster-block-0.2.1-6 and glusterfs-3.8.4-33) ,
> and I had not hit this then. 
> 
> Atin, is there a chance of any patches between 3.8.4-33 and 3.8.4-35 to have
> impacted gluster-block behaviour?

If not, then it must be something to do with the steps that I am following for verifying bz 1456231.

Comment 4 Sweta Anandpara 2017-08-01 09:04:58 UTC

Apologies. Correction in the bug number mentioned in Description and Comment3 - bz 1464421

Comment 13 Sweta Anandpara 2017-08-03 10:58:51 UTC

Resetting the blocker flag as this bug will not be hit in CNS environment.

Comment 21 errata-xmlrpc 2018-09-12 09:25:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2691

Note You need to log in before you can comment on or make changes to this bug.