Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: I'm not able to access the following wildfly application I have running via ssh https://openshift.redhat.com/app/console/application/5355fbd9500446d0bb0002f6-myiot Version-Release number of selected component (if applicable): How reproducible: Currently always Steps to Reproduce: 1. git clone ssh://5355fbd9500446d0bb0002f6.com/~/git/myiot.git/ 2. ssh -v -l 5355fbd9500446d0bb0002f6 myiot-jbossdev.rhcloud.com 3. Both fail as shown below. Actual results: [tmp 545]$ git clone ssh://5355fbd9500446d0bb0002f6.com/~/git/myiot.git/ Cloning into 'myiot'... Write failed: Broken pipe fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. [tmp 546]$ ssh -v -l 5355fbd9500446d0bb0002f6 myiot-jbossdev.rhcloud.com OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011 debug1: Reading configuration data /Users/starksm/.ssh/config debug1: /Users/starksm/.ssh/config line 20: Applying options for *-jbossdev.rhcloud.com debug1: Reading configuration data /etc/ssh_config debug1: /etc/ssh_config line 20: Applying options for * debug1: /etc/ssh_config line 102: Applying options for * debug1: Connecting to myiot-jbossdev.rhcloud.com [54.196.79.225] port 22. debug1: Connection established. debug1: identity file /Users/starksm/.ssh/openshift_rsa type 1 debug1: identity file /Users/starksm/.ssh/openshift_rsa-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.2 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH_5* debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5 none debug1: kex: client->server aes128-ctr hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: Server host key: RSA cf:ee:77:cb:0e:fc:02:d7:72:7e:ae:80:c0:90:88:a7 debug1: Host 'myiot-jbossdev.rhcloud.com' is known and matches the RSA host key. debug1: Found key in /Users/starksm/.ssh/known_hosts:17 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: Roaming not allowed by server debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Next authentication method: publickey debug1: Offering RSA public key: /Users/starksm/.ssh/openshift_rsa debug1: Remote: Forced command: /usr/bin/oo-trap-user debug1: Remote: X11 forwarding disabled. debug1: Server accepts key: pkalg ssh-rsa blen 279 debug1: Remote: Forced command: /usr/bin/oo-trap-user debug1: Remote: X11 forwarding disabled. debug1: Authentication succeeded (publickey). Authenticated to myiot-jbossdev.rhcloud.com ([54.196.79.225]:22). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions debug1: Entering interactive session. Write failed: Broken pipe [tmp 547]$ Expected results: Both the clone and ssh should succeed. Additional info:
I assume you have ran rhc setup -l <login> to make sure your keys are ok. Beyond that, do you have any .ssh/config tweaks?
Yes, and I had already cloned the repository and logged in last night. It was only today that I was no longer able to interact with the code repository or ssh into the host. I restarted the application and was able to access the host via ssh, but now when I do that I'm getting into some different type of shell that is showing resource problems: [tmp 557]$ ssh 5355fbd9500446d0bb0002f6.com rhcsh> rhcsh> w bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: Resource temporarily unavailable rhcsh>
I can't even exit the shell due to resource issues. rhcsh> exit exit bash: export: `=/usr/lib/openshift/cartridge_sdk/bash/sdk': not a valid identifier bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: Resource temporarily unavailable rhcsh> exit exit bash: export: `=/usr/lib/openshift/cartridge_sdk/ruby/sdk.rb': not a valid identifier bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: Resource temporarily unavailable rhcsh>
You gear is exceeding the Online limit of 250 nprocs. The current JBoss cartridges use $(ulimit -u) to determine the number of procs available and auto-tune. Here is a pointer to the JBoss EAP cartridge: https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-jbosseap/versions/shared/bin/standalone.conf
So what does that mean for a user of the wildfly cartridge, that it can't run in a small sized gear?
does your application spawn its own threads? What else can you tell us about your application?
Created attachment 915890 [details] Comment (This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).
Created attachment 888727 [details] server.log thread dump
The server.log thread dump shows there are 160 threads running after the app is warmed up. Most of these threads are completely unused, so if the thread pool sizes are being based on the gear size, this logic needs to be scaled down because it should not take more than 60 threads for a fully functional EE7 app in a small gear, and this app is using only a fraction of the EE7 capabilities. If I dump out all process threads, I'm only seeing about 200, so I ran the part of the app that makes several requests to another server and I see this causes the thread count to exceed the 250 limit: [myiot-jbossdev.rhcloud.com 5357345550044661c7000246]\> ps -em | wc 196 784 5317 [myiot-jbossdev.rhcloud.com 5357345550044661c7000246]\> ps -em | wc 198 792 5371 [myiot-jbossdev.rhcloud.com 5357345550044661c7000246]\> ps -em | wc 198 792 5371 [myiot-jbossdev.rhcloud.com 5357345550044661c7000246]\> ps -em | wc 198 792 5371 [myiot-jbossdev.rhcloud.com 5357345550044661c7000246]\> ps -em | wc 198 792 5371 [myiot-jbossdev.rhcloud.com 5357345550044661c7000246]\> ps -em | wc 216 864 5857 [myiot-jbossdev.rhcloud.com 5357345550044661c7000246]\> ps -em | wc 233 932 6316 [myiot-jbossdev.rhcloud.com 5357345550044661c7000246]\> ps -em | wc 243 972 6586 [myiot-jbossdev.rhcloud.com 5357345550044661c7000246]\> ps -em | wc 251 1004 6802 [myiot-jbossdev.rhcloud.com 5357345550044661c7000246]\> ps -em | wc bash: fork: retry: Resource temporarily unavailable The reality is that the thread pools sizes are simply too big for the given ulimit resource constraint.
Thanks for the detailed analysis, we'll take a look at scaling the threadpools down.
the wildfly message thread pool size has been reduced. https://github.com/openshift-cartridges/openshift-wildfly-cartridge/commit/09c0a7e2a9b09278c2bbf41f042d5f0efced8d82
Create a new jboss-wildfly-8 application. After the app is warmed up, the thread stays at a relatively stable and low level.And could ssh, git clone application. [wildfly-test.dev.rhcloud.com 546af6160cffa563ea00000d]\> ps -em | wc 124 496 3249 [wildfly-test.dev.rhcloud.com 546af6160cffa563ea00000d]\> ps -em | wc 124 496 3249 [wildfly-test.dev.rhcloud.com 546af6160cffa563ea00000d]\> ps -em | wc 124 496 3249 [wildfly-test.dev.rhcloud.com 546af6160cffa563ea00000d]\> ps -em | wc 124 496 3249 [wildfly-test.dev.rhcloud.com 546af6160cffa563ea00000d]\> ps -em | wc 124 496 3249 [wildfly-test.dev.rhcloud.com 546af6160cffa563ea00000d]\> ulimit -u 250 Mark this bug as verified status.Thanks~
(In reply to XiuJuan Wang from comment #12) The log from wildfly/logs/server.log "VM Thread" os_prio=0 tid=0xf765c000 nid=0x337b runnable "VM Periodic Task Thread" os_prio=0 tid=0xf7685400 nid=0x3382 waiting on condition JNI global references: 137 Heap def new generation total 18112K, used 13550K [0xd8200000, 0xd95a0000, 0xdd750000) eden space 16128K, 83% used [0xd8200000, 0xd8f32f58, 0xd91c0000) from space 1984K, 1% used [0xd91c0000, 0xd91c8c48, 0xd93b0000) to space 1984K, 0% used [0xd93b0000, 0xd93b0000, 0xd95a0000) tenured generation total 39616K, used 37782K [0xdd750000, 0xdfe00000, 0xe8200000) the space 39616K, 95% used [0xdd750000, 0xdfc35b90, 0xdfc35c00, 0xdfe00000) Metaspace used 37490K, capacity 39627K, committed 39660K, reserved 40240K