1128251 – In order to use a standard gear as a jenkins slave, ~/.jenkins must exist and be writable

Bug 1128251 - In order to use a standard gear as a jenkins slave, ~/.jenkins must exist and be writable

Summary: In order to use a standard gear as a jenkins slave, ~/.jenkins must exist and...

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	OpenShift Online
Classification:	Red Hat
Component:	Image
Sub Component:
Version:	1.x
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	---
Assignee:	Ben Parees
QA Contact:	libra bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2014-08-08 17:16 UTC by Andy Grimm
Modified:	2018-12-06 17:42 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2014-08-15 16:01:18 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Andy Grimm 2014-08-08 17:16:10 UTC

Description of problem:

I have no idea whether we consider this a valid use case, but it worked until last month.

After the update to jenkins 1.554.2, users who configured openshift gears as build slaves for their jenkins server are seeing errors because ~/.jenkins/cache is not writable in slave gears.

See:  https://issues.jenkins-ci.org/browse/JENKINS-18578

This is not specifically a jenkins or jenkins client cartridge issue.  I don't think there's any reason a build slave would need to have the jenkins-client cart.

Version-Release number of selected component (if applicable):

?

How reproducible:

uncertain

Steps to Reproduce:

The user who reported this has done a bit of customization, so it will take me some time to enumerate the steps.  Essentially, they are using the jenkins ssh Agent plugin, and they've added several gears as slaves (build executors).  Then they have jobs set up to execute on these gears.

Actual results:

Building remotely on <SLAVE> in workspace /var/lib/openshift/<UUID>/app-root/data/jenkins/workspace/<WORKSPACE>
[ssh-agent] Using credentials <UUID>
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Java/JNR ssh-agent
[ssh-agent] FATAL: Could not find a suitable ssh-agent provider
[ssh-agent] Diagnostic report
[ssh-agent] * Java/JNR ssh-agent
java.io.IOException: Remote call on <SLAVE> failed
	at hudson.remoting.Channel.call(Channel.java:731)
	at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgentFactory.start(JNRRemoteAgentFactory.java:61)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper$SSHAgentEnvironment.<init>(SSHAgentBuildWrapper.java:203)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.createSSHAgentEnvironment(SSHAgentBuildWrapper.java:119)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.preCheckout(SSHAgentBuildWrapper.java:89)
	at jenkins.scm.SCMCheckoutStrategy.preCheckout(SCMCheckoutStrategy.java:78)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:579)
	at hudson.model.Run.execute(Run.java:1684)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:231)
Caused by: java.lang.ExceptionInInitializerError
	at jnr.ffi.FFIProvider.getSystemProvider(FFIProvider.java:33)
	at jnr.ffi.Runtime$SingletonHolder.<clinit>(Runtime.java:37)
	at jnr.ffi.Runtime.getSystemRuntime(Runtime.java:32)
	at jnr.unixsocket.SockAddrUnix.<init>(SockAddrUnix.java:34)
	at jnr.unixsocket.SockAddrUnix$DefaultSockAddrUnix.<init>(SockAddrUnix.java:128)
	at jnr.unixsocket.SockAddrUnix.create(SockAddrUnix.java:99)
	at jnr.unixsocket.UnixSocketAddress.<init>(UnixSocketAddress.java:32)
	at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer.start(AgentServer.java:66)
	at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgent.<init>(JNRRemoteAgent.java:64)
	at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgentStarter.call(JNRRemoteAgentStarter.java:54)
	at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgentStarter.call(JNRRemoteAgentStarter.java:35)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Could not load FFI provider jnr.ffi.provider.jffi.Provider
	at jnr.ffi.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:78)
	at jnr.ffi.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:64)
	... 19 more
Caused by: java.lang.UnsatisfiedLinkError: could not get native definition for type: POINTER
	at com.kenai.jffi.Type$InvalidLookupTable.find(Type.java:284)
	at com.kenai.jffi.Type$BuiltinTypeInfo.find(Type.java:319)
	at com.kenai.jffi.Type$Builtin.size(Type.java:216)
	at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:153)
	at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:55)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:39)
	at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:51)
	at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:47)
	at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:31)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at java.lang.Class.newInstance(Class.java:374)
	at jnr.ffi.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:75)
	... 20 more
Caused by: java.lang.UnsatisfiedLinkError
	at com.kenai.jffi.Foreign.newLoadError(Foreign.java:72)
	at com.kenai.jffi.Foreign.access$300(Foreign.java:42)
	at com.kenai.jffi.Foreign$InValidInstanceHolder.getForeign(Foreign.java:98)
	at com.kenai.jffi.Foreign.getInstance(Foreign.java:103)
	at com.kenai.jffi.Type$BuiltinTypeInfo.<clinit>(Type.java:311)
	... 34 more
Caused by: java.lang.UnsatisfiedLinkError
	at com.kenai.jffi.Init.newLoadError(Init.java:132)
	at com.kenai.jffi.Init.load(Init.java:89)
	at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:49)
	at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:45)
	... 36 more
Caused by: java.lang.Error: Unable to load resource jni/i386-Linux/libjffi-1.2.so
	at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:380)
	at java.lang.ClassLoader.getResource(ClassLoader.java:1147)
	at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:227)
	at com.kenai.jffi.internal.StubLoader.getResourceAsStream(StubLoader.java:364)
	at com.kenai.jffi.internal.StubLoader.getStubLibraryStream(StubLoader.java:334)
	at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:292)
	at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:235)
	at com.kenai.jffi.internal.StubLoader.<clinit>(StubLoader.java:395)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:270)
	at com.kenai.jffi.Init.load(Init.java:68)
	... 38 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Failed to write to /var/lib/openshift/<UUID>/.jenkins/cache/jars/45/E7EB5D4FC9AB0215E23A1C1977CAB0.jar
	at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:75)
	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
	at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:372)
	... 48 more
Caused by: java.io.IOException: Failed to write to /var/lib/openshift/<UUID>/.jenkins/cache/jars/45/E7EB5D4FC9AB0215E23A1C1977CAB0.jar
	at hudson.remoting.FileSystemJarCache.retrieve(FileSystemJarCache.java:85)
	at hudson.remoting.JarCacheSupport$1.run(JarCacheSupport.java:62)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104)
	... 1 more
Caused by: java.io.IOException: No such file or directory
	at java.io.UnixFileSystem.createFileExclusively(Native Method)
	at java.io.File.createNewFile(File.java:1006)
	at java.io.File.createTempFile(File.java:1989)
	at hudson.remoting.FileSystemJarCache.retrieve(FileSystemJarCache.java:61)
	... 5 more


Expected results:

Builds should succeed.

Comment 1 Ben Parees 2014-08-08 17:29:18 UTC

This doesn't make much sense...   existing slaves should still be running the old jenkins code (which doesn't need that dir) and new slaves would be launched w/ the new args in which we set the jar cache path to $OPENSHIFT_DATA_DIR/.jenkins/cache/jars


If they are doing something manual to configure/launch jenkins slaves, then they'll nee dto pass the jar cache path parameter themselves:

java -jar <jenkinsslave.jar> -jar-cache <path>

Is that sufficient here?

Comment 2 Andy Grimm 2014-08-08 17:50:21 UTC

I'm not sure why it didn't make sense, but you led me to the right answer, which was in the comments of the issue I linked to:

'In the slave configuration, under the Advanced method of Launch Option,
Put the following in the Suffix Start Slave Command " -jar-cache <path to jar cache directory>" (no quotes)'

That's what they need for slaves they've configured themselves.  I'm still waiting to verify that this works, but I expect that it will.  If so, I'll close the issue.

Comment 3 Ben Parees 2014-08-08 17:52:11 UTC

Sorry, what didn't make sense to me was why a normal slave (ie one launched by openshift) would be hitting this.  If it's a custom configured slave, yeah, it makes sense.

Comment 4 Andy Grimm 2014-08-15 16:01:18 UTC

The customer has not responded.  I'm going to assume this can be closed.

Comment 5 Dmitry Miroshnichenko 2015-09-08 13:50:59 UTC

Hi guys! Seems to be I have the same problem. I have a gear named netztest that creates new gear netztestbldr. Build starts with the log below. What should I do to get it work!? Really tired with jenkins. Thought there would be all out of the box. Should I manage nodes or I can leave all by default?  


Started by user Jenkins System Builder
Building remotely on netztestbldr (netztest-build) in workspace /var/lib/openshift/55eee08346dcfd4e48000001/app-root/runtime/repo
Checkout:repo / /var/lib/openshift/55eee08346dcfd4e48000001/app-root/runtime/repo - hudson.remoting.Channel@52fc9ac8:netztestbldr
Using strategy: Default
Checkout:repo / /var/lib/openshift/55eee08346dcfd4e48000001/app-root/runtime/repo - hudson.remoting.LocalChannel@14e6ca4e
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository origin
Fetching upstream changes from ssh://55d66b0d46dcfd658c0000b2.os.netzrezepte.com/~/git/netztest.git
Seen branch in repository origin/HEAD
Seen branch in repository origin/master
Commencing build of Revision d2a9fafcda6f9c7b446280bc7e108817d1d734b0 (origin/HEAD, origin/master)
Checking out Revision d2a9fafcda6f9c7b446280bc7e108817d1d734b0 (origin/HEAD, origin/master)
java.io.IOException: remote file operation failed: /var/lib/openshift/55eee08346dcfd4e48000001/app-root/runtime/repo at hudson.remoting.Channel@52fc9ac8:netztestbldr: java.io.IOException: Remote call on netztestbldr failed
	at hudson.FilePath.act(FilePath.java:987)
	at hudson.FilePath.act(FilePath.java:969)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1178)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1282)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532)
	at hudson.model.Run.execute(Run.java:1741)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:381)
Caused by: java.io.IOException: Remote call on netztestbldr failed
	at hudson.remoting.Channel.call(Channel.java:761)
	at hudson.FilePath.act(FilePath.java:980)
	... 10 more
Caused by: java.lang.Error: Unable to load resource javax/servlet/LocalStrings.properties
	at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:381)
	at java.lang.ClassLoader.getResource(ClassLoader.java:1147)
	at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:227)
	at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2604)
	at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2589)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2588)
	at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400)
	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
	at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296)
	at java.util.ResourceBundle.getBundle(ResourceBundle.java:721)
	at javax.servlet.GenericServlet.<clinit>(GenericServlet.java:95)
	at hudson.model.Node$Mode.<clinit>(Node.java:525)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at java.lang.Class.getEnumConstantsShared(Class.java:3138)
	at java.lang.Class.getEnumConstants(Class.java:3116)
	at jenkins.model.Jenkins.<clinit>(Jenkins.java:4246)
	at hudson.ExtensionList.lookup(ExtensionList.java:400)
	at hudson.FilePath.act(FilePath.java:977)
	at hudson.FilePath.act(FilePath.java:969)
	at hudson.FilePath.write(FilePath.java:1883)
	at hudson.plugins.git.GitSCM.computeChangeLog(GitSCM.java:1262)
	at hudson.plugins.git.GitSCM.access$700(GitSCM.java:79)
	at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1233)
	at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1178)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2691)
	at hudson.remoting.UserRequest.perform(UserRequest.java:121)
	at hudson.remoting.UserRequest.perform(UserRequest.java:49)
	at hudson.remoting.Request$2.run(Request.java:325)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
	at ......remote call to netztestbldr(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1361)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
	at hudson.remoting.Channel.call(Channel.java:753)
	... 11 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Failed to write to /var/lib/openshift/55eee08346dcfd4e48000001/.jenkins/cache/jars/FD/3FEA6034F56042672E45A7A2D99DC7.jar
	at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:75)
	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
	at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:373)
	at java.lang.ClassLoader.getResource(ClassLoader.java:1147)
	at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:227)
	at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2604)
	at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2589)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2588)
	at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400)
	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
	at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296)
	at java.util.ResourceBundle.getBundle(ResourceBundle.java:721)
	at javax.servlet.GenericServlet.<clinit>(GenericServlet.java:95)
	at hudson.model.Node$Mode.<clinit>(Node.java:525)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at java.lang.Class.getEnumConstantsShared(Class.java:3138)
	at java.lang.Class.getEnumConstants(Class.java:3116)
	at jenkins.model.Jenkins.<clinit>(Jenkins.java:4246)
	at hudson.ExtensionList.lookup(ExtensionList.java:400)
	at hudson.FilePath.act(FilePath.java:977)
	at hudson.FilePath.act(FilePath.java:969)
	at hudson.FilePath.write(FilePath.java:1883)
	at hudson.plugins.git.GitSCM.computeChangeLog(GitSCM.java:1262)
	at hudson.plugins.git.GitSCM.access$700(GitSCM.java:79)
	at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1233)
	at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1178)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2691)
	at hudson.remoting.UserRequest.perform(UserRequest.java:121)
	at hudson.remoting.UserRequest.perform(UserRequest.java:49)
	at hudson.remoting.Request$2.run(Request.java:325)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to write to /var/lib/openshift/55eee08346dcfd4e48000001/.jenkins/cache/jars/FD/3FEA6034F56042672E45A7A2D99DC7.jar
	at hudson.remoting.FileSystemJarCache.retrieve(FileSystemJarCache.java:85)
	at hudson.remoting.JarCacheSupport$1.run(JarCacheSupport.java:62)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
	... 1 more
Caused by: java.io.IOException: No such file or directory
	at java.io.UnixFileSystem.createFileExclusively(Native Method)
	at java.io.File.createTempFile(File.java:2001)
	at hudson.remoting.FileSystemJarCache.retrieve(FileSystemJarCache.java:61)
	... 5 more
Archiving artifacts
Finished: FAILURE

Comment 6 Ben Parees 2015-09-08 14:12:52 UTC

Dmitry can you open a new bug and include full steps to reproduce?

Was your jenkins server created recently?

Comment 7 Dmitry Miroshnichenko 2015-09-08 14:49:17 UTC

Thanks Ben for fast response!

created bug: https://bugzilla.redhat.com/show_bug.cgi?id=1261094

Yep it's a fresh jenkins server. Have openshift m4

Note You need to log in before you can comment on or make changes to this bug.