Bug 798914

Summary: ovirt-engine-core : GSSAPI Authentication errors when using Rest API
Product: [Retired] oVirt Reporter: Oded Ramraz <oramraz>
Component: ovirt-engine-coreAssignee: Roy Golan <rgolan>
Status: CLOSED NOTABUG QA Contact:
Severity: high Docs Contact:
Priority: urgent    
Version: unspecifiedCC: acathrow, iheim, ykaul
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-03-07 11:23:25 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
engine logs
none
kerberos debug messages none

Description Oded Ramraz 2012-03-01 09:16:13 UTC 
Description of problem:

Sometimes when running scripts using Rest API , few operations fails due to authentication errors ( see Additional Information )
We are using GSSAPI authentication in our environment. 

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

2012-02-29 22:12:00,160 ERROR [org.ovirt.engine.core.bll.adbroker.GSSAPIDirContextAuthenticationStrategy] (http--0.0.0.0-8080-2) Error from Kerberos: java.lang.NullPointerException
	at java.io.ByteArrayInputStream.<init>(ByteArrayInputStream.java:106)
	at sun.security.util.DerValue.<init>(DerValue.java:294)
	at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:54)
	at sun.security.krb5.KrbAsReq.getReply(KrbAsReq.java:469)
	at sun.security.krb5.Credentials.sendASRequest(Credentials.java:399)
	at sun.security.krb5.Credentials.acquireTGT(Credentials.java:348)
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:679)
	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:559)
	at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:784)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:698)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:696)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:695)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:594)
	at org.ovirt.engine.core.bll.adbroker.GSSAPIDirContextAuthenticationStrategy.authenticateToKDC(GSSAPIDirContextAuthenticationStrategy.java:127)
	at org.ovirt.engine.core.bll.adbroker.GSSAPIDirContextAuthenticationStrategy.explicitAuth(GSSAPIDirContextAuthenticationStrategy.java:119)
	at org.ovirt.engine.core.bll.adbroker.GSSAPIDirContextAuthenticationStrategy.authenticate(GSSAPIDirContextAuthenticationStrategy.java:111)
	at org.ovirt.engine.core.bll.adbroker.GSSAPILdapTemplateWrapper.useAuthenticationStrategy(GSSAPILdapTemplateWrapper.java:85)
	at org.ovirt.engine.core.bll.adbroker.PrepareLdapConnectionTask.call(PrepareLdapConnectionTask.java:56)
	at org.ovirt.engine.core.bll.adbroker.DirectorySearcher.find(DirectorySearcher.java:98)
	at org.ovirt.engine.core.bll.adbroker.DirectorySearcher.FindOne(DirectorySearcher.java:39)
	at org.ovirt.engine.core.bll.adbroker.LdapAuthenticateUserCommand.executeQuery(LdapAuthenticateUserCommand.java:44)
	at org.ovirt.engine.core.bll.adbroker.LdapBrokerCommandBase.Execute(LdapBrokerCommandBase.java:68)
	at org.ovirt.engine.core.bll.adbroker.LdapBrokerBase.RunAdAction(LdapBrokerBase.java:18)
	at org.ovirt.engine.core.bll.LoginUserCommand.AuthenticateUser(LoginUserCommand.java:36)
	at org.ovirt.engine.core.bll.LoginBaseCommand.isUserCanBeAuthenticated(LoginBaseCommand.java:183)
	at org.ovirt.engine.core.bll.LoginAdminUserCommand.canDoAction(LoginAdminUserCommand.java:18)
	at org.ovirt.engine.core.bll.CommandBase.InternalCanDoAction(CommandBase.java:391)
	at org.ovirt.engine.core.bll.CommandBase.ExecuteAction(CommandBase.java:202)
	at org.ovirt.engine.core.bll.Backend.Login(Backend.java:470)
	at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374)
	at org.ovirt.engine.core.utils.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:11)
	at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:221)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:367)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:205)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:44)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:55)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)
	at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view10.Login(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
	at $Proxy34.Login(Unknown Source)
	at org.ovirt.engine.api.restapi.security.auth.LoginValidator.validate(LoginValidator.java:66)
	at org.ovirt.engine.api.common.security.auth.Challenger.preProcess(Challenger.java:85)
	at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:245)
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:220)
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:209)
	at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:519)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:496)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:151)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
	at java.lang.Thread.run(Thread.java:679)

2012-02-29 22:12:00,162 ERROR [org.ovirt.engine.core.bll.adbroker.DirectorySearcher] (http--0.0.0.0-8080-2) Failed ldap search server ldap://qa1.qa.lab.tlv.redhat.com:389 due to org.ovirt.engine.core.bll.adbroker.EngineDirectoryServiceException. We should not try the next server: org.ovirt.engine.core.bll.adbroker.EngineDirectoryServiceException
	at org.ovirt.engine.core.bll.adbroker.GSSAPIDirContextAuthenticationStrategy.authenticateToKDC(GSSAPIDirContextAuthenticationStrategy.java:150) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.adbroker.GSSAPIDirContextAuthenticationStrategy.explicitAuth(GSSAPIDirContextAuthenticationStrategy.java:119) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.adbroker.GSSAPIDirContextAuthenticationStrategy.authenticate(GSSAPIDirContextAuthenticationStrategy.java:111) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.adbroker.GSSAPILdapTemplateWrapper.useAuthenticationStrategy(GSSAPILdapTemplateWrapper.java:85) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.adbroker.PrepareLdapConnectionTask.call(PrepareLdapConnectionTask.java:56) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.adbroker.DirectorySearcher.find(DirectorySearcher.java:98) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.adbroker.DirectorySearcher.FindOne(DirectorySearcher.java:39) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.adbroker.LdapAuthenticateUserCommand.executeQuery(LdapAuthenticateUserCommand.java:44) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.adbroker.LdapBrokerCommandBase.Execute(LdapBrokerCommandBase.java:68) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.adbroker.LdapBrokerBase.RunAdAction(LdapBrokerBase.java:18) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.LoginUserCommand.AuthenticateUser(LoginUserCommand.java:36) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.LoginBaseCommand.isUserCanBeAuthenticated(LoginBaseCommand.java:183) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.LoginAdminUserCommand.canDoAction(LoginAdminUserCommand.java:18) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.InternalCanDoAction(CommandBase.java:391) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.ExecuteAction(CommandBase.java:202) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.Backend.Login(Backend.java:470) [engine-bll.jar:]
	at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) [:1.6.0_24]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [:1.6.0_24]
	at java.lang.reflect.Method.invoke(Method.java:616) [:1.6.0_24]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.0.Beta1b.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:]
	at org.ovirt.engine.core.utils.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:11) [engine-utils.jar:]
	at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) [:1.6.0_24]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [:1.6.0_24]
	at java.lang.reflect.Method.invoke(Method.java:616) [:1.6.0_24]
	at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.1.0.Beta1b.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.0.Beta1b.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.0.Beta1b.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.0.Beta1b.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:221) [jboss-as-ejb3-7.1.0.Beta1b.jar:]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:367) [jboss-as-ejb3-7.1.0.Beta1b.jar:]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:205) [jboss-as-ejb3-7.1.0.Beta1b.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.0.Beta1b.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:44) [jboss-as-ee-7.1.0.Beta1b.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:55) [jboss-as-ejb3-7.1.0.Beta1b.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.0.Beta1b.jar:]
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.0.Beta1b.jar:]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:]
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.0.Beta1b.jar:]
	at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view10.Login(Unknown Source) [engine-common.jar:]
	at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) [:1.6.0_24]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [:1.6.0_24]
	at java.lang.reflect.Method.invoke(Method.java:616) [:1.6.0_24]
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) [spring-aop-2.5.6.SEC02.jar:]
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198) [spring-aop-2.5.6.SEC02.jar:]
	at $Proxy34.Login(Unknown Source)	at org.ovirt.engine.api.restapi.security.auth.LoginValidator.validate(LoginValidator.java:66)
	at org.ovirt.engine.api.common.security.auth.Challenger.preProcess(Challenger.java:85)
	at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:245) [resteasy-jaxrs-2.2.3.GA.jar:]
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:220) [resteasy-jaxrs-2.2.3.GA.jar:]
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:209) [resteasy-jaxrs-2.2.3.GA.jar:]
	at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:519) [resteasy-jaxrs-2.2.3.GA.jar:]
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:496) [resteasy-jaxrs-2.2.3.GA.jar:]
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.2.3.GA.jar:]
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.2.3.GA.jar:]
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.2.3.GA.jar:]
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.2.3.GA.jar:]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:151)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
	at java.lang.Thread.run(Thread.java:679) [:1.6.0_24]
Comment 1 Oded Ramraz 2012-03-01 09:16:47 UTC 
Created attachment 566785 [details]
engine logs
Comment 2 lpeer 2012-03-05 11:12:10 UTC 
According to Oved this is a hard one to catch.
Do we have a way to reproduce this in a consistent way?
Comment 3 Oded Ramraz 2012-03-05 12:41:14 UTC 
We don't have a consistent way to reproduce it , but it happen from time to time. 
rgolan recommend me to open this issue in order to investigate it / add it to release notes.
Comment 4 Roy Golan 2012-03-07 11:17:44 UTC 
Created attachment 568259 [details]
kerberos debug messages
Comment 5 Roy Golan 2012-03-07 11:23:25 UTC 
I have debugged jboss with -Dsun.security.krb5.debug=true
Took like half an our of automation cycle for that to occur but we can't rely on that as a reporducer.

The debug messages clearly shows the krb is trying to read 0 bytes from the reply:

2012-03-06 17:28:26,873 INFO  [stdout] (http--0.0.0.0-8080-2) >>> KrbKdcReq send: kdc=qa1.qa.lab.tlv.redhat.com TCP:88, timeout=30000, number of retries =3, #bytes=1368
2012-03-06 17:28:26,874 INFO  [stdout] (http--0.0.0.0-8080-2) >>>DEBUG: TCPClient could not read length field
2012-03-06 17:28:26,874 INFO  [stdout] (http--0.0.0.0-8080-2) >>> KrbKdcReq send: #bytes read=0
2012-03-06 17:28:26,874 INFO  [stdout] (http--0.0.0.0-8080-2) >>> KrbKdcReq send: #bytes read=0

Following this the NPE occurs due to null input buffer.

qa1.qa eventlog have many errors on kerberos, master browser(what ever that means) , near the ocurance of the error.

I cant find the reason why the kerberos service have an empty reply or did it 
close the connection but anyhow our behaviour is to try the next server if exist which is good.

log is attached.