Bug 1214249

Summary:	I/O (fresh writes) failure of failure of hot tier
Product:	[Community] GlusterFS	Reporter:	Anoop <annair>
Component:	tiering	Assignee:	bugs <bugs>
Status:	CLOSED NOTABUG	QA Contact:	bugs <bugs>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	mainline	CC:	bugs, dlambrig, nchilaka
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-05-14 06:48:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1260923

Description Anoop 2015-04-22 09:57:22 UTC

Description of problem:

Seeing I/O error eventaul I/O timeout while failing hot tier 

Version-Release number of selected component (if applicable):
glusterfs-server-3.7dev-0.994.git0d36d4f.el6.x86_64

Volume Name: vol1
Type: Tier
Volume ID: b77d4050-7fdc-45ff-a084-f85eec2470fc
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.35.67:/rhs/brick2
Brick2: 10.70.35.56:/rhs/brick2
Brick3: 10.70.35.56:/rhs/brick1
Brick4: 10.70.35.67:/rhs/brick1



Steps to Reproduce:
1.Create a tiered volume 
2. Start I/O, 100% writes to the volume
3. While the I/O is in progress, fail the bricks in hot tier

Actual results:
bzip2: I/O or other error, bailing out.  Possible reason follows.
bzip2: Transport endpoint is not connected
	Input file = (stdin), output file = (stdout)
tar: linux-2.6.31.1/arch/ia64/kernel/gate.lds.S: Cannot open: No such file or directory
linux-2.6.31.1/arch/ia64/kernel/head.S
tar: linux-2.6.31.1/arch/ia64/kernel/head.S: Cannot open: No such file or directory
linux-2.6.31.1/arch/ia64/kernel/ia64_ksyms.c
tar: linux-2.6.31.1/arch/ia64/kernel/ia64_ksyms.c: Cannot open: No such file or directory
linux-2.6.31.1/arch/ia64/kernel/init_task.c
tar: linux-2.6.31.1/arch/ia64/kernel/init_task.c: Cannot open: No such file or directory
linux-2.6.31.1/arch/ia64/kernel/iosapic.c
tar: linux-2.6.31.1/arch/ia64/kernel/iosapic.c: Cannot open: No such file or directory
tar: Child returned status 1


Expected results:
Even if the hot tier is not present, the fresh writes should be routed to the cold tier. 

*Looks like all new write are default routed to the hot tier.

Comment 1 Dan Lambright 2015-04-22 17:25:27 UTC

This problem is basically the same as 1212822. We do not keep copies of files on both tiers and it is expected behavior for the file not to be found if all the bricks in the hot tier are taken down. This is why you would want redundancy on the hot tier. We can discuss further to iron out any details in the understanding, so I'll put this ON_QA for now.

Comment 2 Anoop 2015-05-13 12:30:27 UTC

Reproduced this ont the BETA2 build too, hence moving it to ASSIGNED.

Comment 3 Dan Lambright 2015-05-14 06:48:42 UTC

Even if it is a fresh write we cannot create a new file. We cannot know if the file already exists on the hot tier. If it turned out the file did already exist, and we created a second file with the same name on the cold, this would overwrite the old file and be dataloss. This is not a bug.