Bug 1009237

Test this bug on puddle 2.1/2014-03-21.4, this download cartridge could be created on the node, but failed to be started due to timeout waiting for http listening port

[root@broker ~]# rhc app create aerog1 https://cartreflect-claytondev.rhcloud.com/reflect?github=aerogear/openshift-origin-cartridge-aerogear-push mysql-5.1 
The cartridge 'https://cartreflect-claytondev.rhcloud.com/reflect?github=aerogear/openshift-origin-cartridge-aerogear-push' will be downloaded and installed
Password: ******

Application Options
-------------------
Domain:     123
Cartridges: https://cartreflect-claytondev.rhcloud.com/reflect?github=aerogear/openshift-origin-cartridge-aerogear-push, mysql-5.1
Gear Size:  medium
Scaling:    no

Creating application 'aerog1' ...
Starting aerogear-push cart
Timed out waiting for http listening port
Failed to execute: 'control start' for /var/lib/openshift/532fcf71cfb77fe100000079/aerogear-push


Error log in /var/log/openshift/node/platform.log
...
March 24 02:26:44 INFO 532fcf71cfb77fe100000079 start against 'aerogear-push'

March 24 02:28:21 INFO Shell command '/sbin/runuser -s /bin/sh 532fcf71cfb77fe100000079 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c1,c441' /bin/sh -c \"set -e; /var/lib/openshift/532fcf71cfb77fe100000079/aerogear-push/bin/control start \""' ran. rc=1 out=Starting aerogear-push cart
Timed out waiting for http listening port

March 24 02:28:21 INFO Shell command 'quota --always-resolve -w 532fcf71cfb77fe100000079' ran. rc=0 out=Disk quotas for user 532fcf71cfb77fe100000079 (uid 1463):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/mapper/vg_rhel65-lv_root   59364       0 2097152             304       0   80000

March 24 02:28:21 INFO openshift-agent: request end: action=cartridge_do, requestid=7d16c6a4db6e5f3fb77a87fdd23ae4d3, senderid=broker.ose-201403214.com.cn, statuscode=1, data={:time=>nil, :output=>"CLIENT_ERROR: Failed to execute: 'control start' for /var/lib/openshift/532fcf71cfb77fe100000079/aerogear-push\nCLIENT_MESSAGE: Starting aerogear-push cart\nCLIENT_MESSAGE: Timed out waiting for http listening port\n\n\n", :exitcode=>157, :addtl_params=>nil}

March 24 02:28:22 INFO openshift-agent: request start: action=cartridge_do requestid=2a276386b0ff56c18005305566a4c782, senderid=broker.ose-201403214.com.cn, data={:cartridge=>"openshift-origin-node", :action=>"app-destroy", :args=>{"--with-app-uuid"=>"532fcf71cfb77fe100000079", "--with-app-name"=>"aerog1", "--with-expose-ports"=>false, "--with-container-uuid"=>"532fcf71cfb77fe100000079", "--with-container-name"=>"aerog1", "--with-namespace"=>"123", "--with-uid"=>1463, "--with-request-id"=>"ec6f9bc6d841ff12dd49659f6dfecfe8", "--cart-name"=>"openshift-origin-node"}, :process_results=>true}
...

*** Bug 1083662 has been marked as a duplicate of this bug. ***

The error in Comment 4 always could be reproduced on puddle 2.2/2014-11-24.3.

Tried to increase the timeout when checking if the server http port is up in openshift-origin-cartridge-aerogear-push/bin/control as:

 function ishttpup() {

    let count=0
    while [ ${count} -lt 48 ]
    do
        if echo 2> /dev/null > "/dev/tcp/${OPENSHIFT_AEROGEAR_PUSH_IP}/${OPENSHIFT_AEROGEAR_PUSH_HTTP_PORT}"; then
            echo "Found ${OPENSHIFT_AEROGEAR_PUSH_IP}:${OPENSHIFT_AEROGEAR_PUSH_HTTP_PORT} listening port"
            return 0
        fi
        let count=${count}+1

        sleep 6
...

And increase the timeout setting for broker ProxyTimeout and mcollective as 600s. Create the app again, failed with error "An error occurred while communicating with the server."

In /var/log/openshift/node/platform.log, it still said time out waiting for http listening port:
... 
November 25 17:46:10 INFO 54744fc282611dc5eb000001 start against 'aerogear-push'

November 25 17:50:58 INFO Shell command '/sbin/runuser -s /bin/sh 54744fc282611dc5eb000001 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c3,c643' /bin/sh -c \"set -e; /var/lib/openshift/54744fc282611dc5eb000001/aerogear-push/bin/control start \""' ran. rc=1 out=Starting aerogear-push cart
Timed out waiting for http listening port
..


After enable archiving gears on gear destroy during app-create, got the tar.bz2 file of the gear. 
In ./.tmp/54744fc282611dc5eb000001/aerogear-push.log, there's an error:

Error: Unable to access jarfile /var/lib/openshift/54744fc282611dc5eb000001/aerogear-push/jboss-modules.jar


The target jar file was not existed, since we don't have /etc/alternatives/jbossas-7/ directory.
[root@node1 54744fc282611dc5eb000001]# ll aerogear-push/jboss-modules.jar 
lrwxrwxrwx. 1 3706 3706 45 Nov 25 17:54 aerogear-push/jboss-modules.jar -> /etc/alternatives/jbossas-7/jboss-modules.jar
[root@node1 54744fc282611dc5eb000001]# ll /etc/alternatives/jbossas-7/jboss-modules.jar
ls: cannot access /etc/alternatives/jbossas-7/jboss-modules.jar: No such file or directory

OpenShift Enterprise v2 has officially reached EoL.  This product is no longer supported and bugs will be closed.

Please look into the replacement enterprise-grade container option, OpenShift Container Platform v3.  https://www.openshift.com/container-platform/

More information can be found here: https://access.redhat.com/support/policy/updates/openshift/