Here's the output from https://build.gluster.org/job/centos6-regression/19/console: 20:14:44 Retriggered by user nigelb for Gerrit: http://review.gluster.org/15170 20:14:44 Building remotely on slave22.cloud.gluster.org (smoke_tests rackspace_regression_2gb glusterfs-devrpms) in workspace /home/jenkins/root/workspace/centos6-regression 20:14:44 Wiping out workspace first. 20:14:46 java.io.IOException: remote file operation failed: /home/jenkins/root/workspace/centos6-regression at hudson.remoting.Channel@c883098:slave22.cloud.gluster.org: java.nio.file.AccessDeniedException: /home/jenkins/root/workspace/centos6-regression/slave22.cloud.gluster.org:/d 20:14:46 at hudson.FilePath.act(FilePath.java:986) 20:14:46 at hudson.FilePath.act(FilePath.java:968) 20:14:46 at hudson.FilePath.deleteContents(FilePath.java:1183) 20:14:46 at hudson.plugins.git.extensions.impl.WipeWorkspace.beforeCheckout(WipeWorkspace.java:28) 20:14:46 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1094) 20:14:46 at hudson.scm.SCM.checkout(SCM.java:485) 20:14:46 at hudson.model.AbstractProject.checkout(AbstractProject.java:1269) 20:14:46 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607) 20:14:46 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) 20:14:46 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529) 20:14:46 at hudson.model.Run.execute(Run.java:1738) 20:14:46 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 20:14:46 at hudson.model.ResourceController.execute(ResourceController.java:98) 20:14:46 at hudson.model.Executor.run(Executor.java:410) 20:14:46 Caused by: java.nio.file.AccessDeniedException: /home/jenkins/root/workspace/centos6-regression/slave22.cloud.gluster.org:/d 20:14:46 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) 20:14:46 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) 20:14:46 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) 20:14:46 at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) 20:14:46 at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) 20:14:46 at java.nio.file.Files.delete(Files.java:1079) 20:14:46 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 20:14:46 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 20:14:46 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 20:14:46 at java.lang.reflect.Method.invoke(Method.java:606) 20:14:46 at hudson.Util.deleteFile(Util.java:255) 20:14:46 at hudson.FilePath.deleteRecursive(FilePath.java:1203) 20:14:46 at hudson.FilePath.deleteContentsRecursive(FilePath.java:1212) 20:14:46 at hudson.FilePath.deleteRecursive(FilePath.java:1194) 20:14:46 at hudson.FilePath.deleteContentsRecursive(FilePath.java:1212) 20:14:46 at hudson.FilePath.access$1100(FilePath.java:190) 20:14:46 at hudson.FilePath$15.invoke(FilePath.java:1186) 20:14:46 at hudson.FilePath$15.invoke(FilePath.java:1183) 20:14:46 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719) 20:14:46 at hudson.remoting.UserRequest.perform(UserRequest.java:120) 20:14:46 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 20:14:46 at hudson.remoting.Request$2.run(Request.java:332) 20:14:46 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) 20:14:46 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 20:14:46 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 20:14:46 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 20:14:46 at java.lang.Thread.run(Thread.java:745) 20:14:46 at ......remote call to slave22.cloud.gluster.org(Native Method) 20:14:46 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) 20:14:46 at hudson.remoting.UserResponse.retrieve(UserRequest.java:220) 20:14:46 at hudson.remoting.Channel.call(Channel.java:781) 20:14:46 at hudson.FilePath.act(FilePath.java:979) 20:14:46 ... 13 more 20:14:46 Finished: FAILURE Also seen in the following: https://build.gluster.org/job/centos6-regression/16/ https://build.gluster.org/job/centos6-regression/15/ https://build.gluster.org/job/centos6-regression/2/ https://build.gluster.org/job/centos6-regression/8/
The folder is owned by root, so when Jenkins tries to wipe-workspace, it runs into trouble. I'm keeping the option turned on for tonight so I know what's going on. Ideally, I'd like to leave it on. Justin said he's seen it a few times last year and was never able to find a root cause. So it's not something new, it's a latent bug in our test system that we used to work around.
Raghavendra, do you have suggestions on where to start looking to debug this?
I've seen this happen after a successful run as well, so this is not a one-off case off bad post-job clean up.
This machine no longer exists and this doesn't happen with the new nodes.