Bug 1367830

Summary: Regression tests leave a slave20.cloud.gluster.org: folder owned by root.
Product: [Community] GlusterFS Reporter: Nigel Babu <nigelb>
Component: testsAssignee: bugs <bugs>
Status: CLOSED WONTFIX QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: mainlineCC: bugs, rtalur
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
URL: https://build.gluster.org/job/centos6-regression/19/console
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-08 03:37:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nigel Babu 2016-08-17 15:04:42 UTC
Here's the output from https://build.gluster.org/job/centos6-regression/19/console:

20:14:44 Retriggered by user nigelb for Gerrit: http://review.gluster.org/15170
20:14:44 Building remotely on slave22.cloud.gluster.org (smoke_tests rackspace_regression_2gb glusterfs-devrpms) in workspace /home/jenkins/root/workspace/centos6-regression
20:14:44 Wiping out workspace first.
20:14:46 java.io.IOException: remote file operation failed: /home/jenkins/root/workspace/centos6-regression at hudson.remoting.Channel@c883098:slave22.cloud.gluster.org: java.nio.file.AccessDeniedException: /home/jenkins/root/workspace/centos6-regression/slave22.cloud.gluster.org:/d
20:14:46 	at hudson.FilePath.act(FilePath.java:986)
20:14:46 	at hudson.FilePath.act(FilePath.java:968)
20:14:46 	at hudson.FilePath.deleteContents(FilePath.java:1183)
20:14:46 	at hudson.plugins.git.extensions.impl.WipeWorkspace.beforeCheckout(WipeWorkspace.java:28)
20:14:46 	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1094)
20:14:46 	at hudson.scm.SCM.checkout(SCM.java:485)
20:14:46 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
20:14:46 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
20:14:46 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
20:14:46 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
20:14:46 	at hudson.model.Run.execute(Run.java:1738)
20:14:46 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
20:14:46 	at hudson.model.ResourceController.execute(ResourceController.java:98)
20:14:46 	at hudson.model.Executor.run(Executor.java:410)
20:14:46 Caused by: java.nio.file.AccessDeniedException: /home/jenkins/root/workspace/centos6-regression/slave22.cloud.gluster.org:/d
20:14:46 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
20:14:46 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
20:14:46 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
20:14:46 	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
20:14:46 	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
20:14:46 	at java.nio.file.Files.delete(Files.java:1079)
20:14:46 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
20:14:46 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
20:14:46 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
20:14:46 	at java.lang.reflect.Method.invoke(Method.java:606)
20:14:46 	at hudson.Util.deleteFile(Util.java:255)
20:14:46 	at hudson.FilePath.deleteRecursive(FilePath.java:1203)
20:14:46 	at hudson.FilePath.deleteContentsRecursive(FilePath.java:1212)
20:14:46 	at hudson.FilePath.deleteRecursive(FilePath.java:1194)
20:14:46 	at hudson.FilePath.deleteContentsRecursive(FilePath.java:1212)
20:14:46 	at hudson.FilePath.access$1100(FilePath.java:190)
20:14:46 	at hudson.FilePath$15.invoke(FilePath.java:1186)
20:14:46 	at hudson.FilePath$15.invoke(FilePath.java:1183)
20:14:46 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719)
20:14:46 	at hudson.remoting.UserRequest.perform(UserRequest.java:120)
20:14:46 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
20:14:46 	at hudson.remoting.Request$2.run(Request.java:332)
20:14:46 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
20:14:46 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
20:14:46 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
20:14:46 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
20:14:46 	at java.lang.Thread.run(Thread.java:745)
20:14:46 	at ......remote call to slave22.cloud.gluster.org(Native Method)
20:14:46 	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
20:14:46 	at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
20:14:46 	at hudson.remoting.Channel.call(Channel.java:781)
20:14:46 	at hudson.FilePath.act(FilePath.java:979)
20:14:46 	... 13 more
20:14:46 Finished: FAILURE


Also seen in the following:

https://build.gluster.org/job/centos6-regression/16/
https://build.gluster.org/job/centos6-regression/15/
https://build.gluster.org/job/centos6-regression/2/
https://build.gluster.org/job/centos6-regression/8/

Comment 1 Nigel Babu 2016-08-17 15:06:26 UTC
The folder is owned by root, so when Jenkins tries to wipe-workspace, it runs into trouble. I'm keeping the option turned on for tonight so I know what's going on. Ideally, I'd like to leave it on.

Justin said he's seen it a few times last year and was never able to find a root cause. So it's not something new, it's a latent bug in our test system that we used to work around.

Comment 2 Nigel Babu 2016-08-17 15:17:24 UTC
Raghavendra, do you have suggestions on where to start looking to debug this?

Comment 3 Nigel Babu 2016-08-17 15:17:59 UTC
I've seen this happen after a successful run as well, so this is not a one-off case off bad post-job clean up.

Comment 4 Nigel Babu 2018-10-08 03:37:07 UTC
This machine no longer exists and this doesn't happen with the new nodes.