From Jeff: I had to log back in (for reasons unknown) and the login keeps failing. AFAICT, the sequence is like this: (1) Click the "Sign-in with GitHub" link (2) Get redirected to GitHub, possibly log in there (3) GitHub seems happy (4) Get redirected back to r.g.o/oauth?code=FOO&state=BAR%3D%2C%2Flogin (5) "Server Error"
I see this exception in the logs 2016-07-28 14:28:20,022] [HTTP-38039] ERROR com.google.gerrit.pgm.http.jetty.HiddenErrorHandler : Error in GET /oauth?code=<snip>&state=<snip>%3D%2C%2Flogin javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1916) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1874) at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1857) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1378) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at com.googlesource.gerrit.plugins.github.oauth.OAuthProtocol.getAccessToken(OAuthProtocol.java:372) at com.googlesource.gerrit.plugins.github.oauth.OAuthProtocol.loginPhase2(OAuthProtocol.java:360) at com.googlesource.gerrit.plugins.github.oauth.GitHubLogin.login(GitHubLogin.java:100) at com.googlesource.gerrit.plugins.github.oauth.OAuthWebFilter.login(OAuthWebFilter.java:123) at com.googlesource.gerrit.plugins.github.oauth.OAuthWebFilter.doFilter(OAuthWebFilter.java:89) at com.googlesource.gerrit.plugins.github.oauth.OAuthFilter.doFilter(OAuthFilter.java:86) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:90) at sun.security.validator.Validator.getInstance(Validator.java:179) at sun.security.ssl.X509TrustManagerImpl.getValidator(X509TrustManagerImpl.java:314) at sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(X509TrustManagerImpl.java:173) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:186) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1454) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:213) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:913) at sun.security.ssl.Handshaker.process_record(Handshaker.java:849) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1035) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1344) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1371) ... 36 more Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200) at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120) at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104) at sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:88) ... 48 more
Root cause of this bug is the latest java upgrade. I tested this hypothesis on staging where it worked fine. Post-java upgrade, it started showing me the same error.
[root@dev ~]# service yum-cron status Nightly yum update is enabled. Ah, we do nightly updates. This is not a good idea without testing on a staging environment first.
After a Gerrit restart, everything seems to be working okay. Gerrit did not seem to like Java being upgraded from underneath it, basically.
Yup, we have automated upgrade since the start. I wonder how we can detect that java was upgraded and restart gerrit. We could get automated reboot every week to handle that, I guess (yes, that's unclean, but that's the only robust automation I can think of)
COnfirm that I am now able to log in again.
Thanks Jeff!