Red Hat Bugzilla – Bug 1106393
Managed server shutdown unexpectedly when timeout during connection request to HC
Last modified: 2018-03-06 15:37:52 EST
Sometimes managed server shutdown unexpectedly when managed server tries to connect to the HC and got lengthly Full GC. java.io.IOException: JBAS012175: Channel closed at org.jboss.as.server.mgmt.domain.HostControllerConnection.getChannel(HostControllerConnection.java:101) at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:117) at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:100) at org.jboss.as.server.mgmt.domain.HostControllerConnection.reConnect(HostControllerConnection.java:171) at org.jboss.as.server.mgmt.domain.HostControllerClient.reconnect(HostControllerClient.java:98) at org.jboss.as.server.DomainServerMain.main(DomainServerMain.java:138) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.modules.Module.run(Module.java:292) at org.jboss.modules.Main.main(Main.java:455) The current DomainServerMain exits on the connetion error. The scenario is: 1. server tries to connect to the HC 2. server Full GC 3. HC is waiting a request from server, read timeout, closes the sock 4. server resumed from GC, failed to send the req to the HC and exit
I have a customer hit this issue again, I extract some log as [1], this may helpful for resolving the issue. The log [1] contain the following info: 1. Server almost exhausted, from 15:04:15 to 15:11:30, more than 7 minutes no log output, this may caused by gc, OS level issue, VM hypervisor issues(customer use VMware virtual platform) 2. Server no shut down log output after 15:14:12, but the Server exit time should be 15:18:12, we can find evidence from PC log 3. HC hit Read timed out at 15:10:01, at the same time Server keep stuck as step 1 4. PC monitor Server exit, receive server exit at 15:18:12, this hints Server exit at 15:18:12 [1] https://github.com/kylinsoong/wildfly-samples/blob/master/domain/bug-1106393-log.md
Which JVM version are you using? Apparently using JDK 7 was helping with this issues, we are going to fix this though in a future release.
Merged upstream PR: https://github.com/wildfly/wildfly-core/pull/150 Upstream commit: https://github.com/wildfly/wildfly-core/commit/70e2286fa6e737df2c4daa5b7f2330a8bd6d43fb
Adding a new case 01183081 to the list of impacted customers. Thanks
Server process now waits until the host-controller is available again and reconnects. Verified on EAP 6.4.0.DR3
Good morning, we analyzed similar problem on our EAP 6 installation and we found a correlation bewteen high swap usage and HC-DC Disconnection. Full GC on a low memory machine cause this problems. Solution, upgrade ram of machine.