Bug 1401390

Summary:	[Disperse] Linux tarball untar fails with error "Device or resource busy"
Product:	Red Hat Gluster Storage	Reporter:	Prasad Desala <tdesala>
Component:	disperse	Assignee:	Sunil Kumar Acharya <sheggodu>
Status:	CLOSED WORKSFORME	QA Contact:	Nag Pavan Chilakam <nchilaka>
Severity:	high	Docs Contact:
Priority:	medium
Version:	rhgs-3.2	CC:	aspandey, nbalacha, pkarampu, rhs-bugs, storage-qa-internal, tdesala
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-12-06 14:33:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Prasad Desala 2016-12-05 06:31:51 UTC

Description of problem:
=======================
When started untarring linux kernel package from two different client machines,  one of the client untar is failing with the below "Device or resource busy" error messages,

tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc/max1027-adc.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc/mcp320x.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc/mcp3422.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc/nuvoton-nau7802.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc/qcom,spmi-iadc.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc/qcom,spmi-vadc.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc/rockchip-saradc.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc/ti-adc128s052.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc/twl4030-madc.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc/vf610-adc.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc/xilinx-xadc.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc: Cannot utime: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc: Cannot change ownership to uid 0, gid 0: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/adc: Cannot change mode to rwxrwxr-x: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/dac: Cannot mkdir: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/dac/ad7303.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/dac/max5821.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/dac/ti,dac7512.txt: Cannot open: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/frequency: Cannot mkdir: Device or resource busy
tar: linux-4.4.1/Documentation/devicetree/bindings/iio/frequency/adf4350.txt: Cannot open: Device or resource busy

Version-Release number of selected component (if applicable):
3.8.4-6.el7rhgs.x86_64

Steps to Reproduce:
===================
1) Create a distributed-disperse volume and start it.
2) FUSE mount the volume on two clients.
3) Create a directory on the mount point lets say it as dir. Now, copy linux kernel package to both mount point and dir. 
4) From two different clients, simultaneously start untarring the kernel package and also start doing some lookups.

At some point, we can see the "Device or resource busy" errors.

Actual results:
===============
Linux tarball untar fails with error "Device or resource busy"

Expected results:
=================
There should not be any errors.

Comment 3 Nag Pavan Chilakam 2016-12-06 10:53:18 UTC

I have seen errors like below
1024 bytes (1.0 kB) copied, 0.036641 s, 27.9 kB/s
dd: closing input file `/dev/urandom': Bad file descriptor

Comment 4 Nag Pavan Chilakam 2016-12-06 12:54:35 UTC

For now this bug would be deferred to 3.2.0_beyond.
However If we hit it again, we would bring in this.

Comment 6 Sunil Kumar Acharya 2017-02-01 15:33:43 UTC

Tried to recreate the issue by following steps mentioned below.

1. Created a distributed disperse volume (4+2) using 4 servers.Options were  configured as shown in comment 2.
2. FUSE Mounted the volume on 2 clients.
3. Copied linux kernel package to mount point and a directory below it.
4. Initiated untar of package copied to mount point from client1 and 
   package copied to the directory below from client2 , simultaneously.
5. Initiated lookups on both the clients.
6. When the untar in progress added bricks to volume.

I wasn't able to recreate the issue on both upstream and downstream v3.8.4-6.

Comment 7 Pranith Kumar K 2017-02-10 06:24:11 UTC

Ashish, Sunil,
    Only posix xlatorgives EBUSY in unlink. Check posix_unlink() code where this happens. I think DHT uses this code. May be you should understand it from them and proceed forward.

Let me know what you find. For now leaving a needinfo on you Ashish

Comment 10 Sunil Kumar Acharya 2017-09-11 11:40:10 UTC

Prasad,

Please try to recreate the issue. If it is not seen again we will close this bug as suggested by Nithya.

Comment 11 Sunil Kumar Acharya 2017-12-06 14:33:37 UTC

Closing this case as we didn't hear anything from the reporter. Please re-open the case if the issue is seen again.