Bug 968849 - JBoss AS 7.0.13 command start restart Unable to complete the requested
JBoss AS 7.0.13 command start restart Unable to complete the requested
Status: CLOSED UPSTREAM
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Linux
unspecified Severity medium
: ---
: ---
Assigned To: Paul Morie
libra bugs
: UpcomingRelease
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-30 02:20 EDT by Ihor
Modified: 2015-05-14 20:17 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-06 09:51:11 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs (8.79 KB, application/x-gzip)
2013-05-31 14:13 EDT, Ihor
no flags Details
slogs (8.39 KB, application/x-gzip)
2013-06-04 08:57 EDT, Ihor
no flags Details

  None (edit)
Description Ihor 2013-05-30 02:20:44 EDT
Description of problem:
I maybe in last two days have next situation: all command
rhc app start -a rs
rhc app restart -a rs
return to console only:
Unable to complete the requested operation due to: Node execution failure
(invalid exit code from node).  If the problem persists please contact Red Hat
support..
--------------------------------------------------------------------------
rhc app tidy -a rs After force-stop is work. after start or restart also return this:
ihor@ihor-M50Vc:~$ rhc app tidy -a rs
Password: ******
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists
please contact Red Hat support..
Reference ID: 1de16735d0ef273c59f4506f7c376167
ihor@ihor-M50Vc:~$ 

this in Jenkins I see:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
........................
Stopping services
WARNING: This ssh terminal was started without a tty.
          It is highly recommended to login with: ssh -t
Application is already stopped
httpd (no pid file) not running
Done
.....................................
Type "help" for more info.

Starting services
WARNING: This ssh terminal was started without a tty.
          It is highly recommended to login with: ssh -t
MySQL already running
Application is explicitly stopped!  Use 'rhc app start -a rs' to start back up.
Done
+ /usr/libexec/openshift/cartridges/jenkins-1.4/info/bin/git_ssh_wrapper.sh 51a5906be0b8cdca930003b4@rs-emetemunoy.rhcloud.com post_deploy.sh
Running .openshift/action_hooks/post_deploy
Archiving artifacts
Finished: SUCCESS

=============================================================================

also:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
........................................
Stopping services
WARNING: This ssh terminal was started without a tty.
          It is highly recommended to login with: ssh -t
Application is already stopped
Waiting for stop to finish
Done
............................................
Type "help" for more info.

Starting services
WARNING: This ssh terminal was started without a tty.
          It is highly recommended to login with: ssh -t
MySQL already running
Timed out waiting for http listening port
Failed to start jbossas-7
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE

Timed out waiting for http listening port <<<<<<<----------------

Version-Release number of selected component (if applicable):
ihor@ihor-M50Vc:~$ sudo gem update rhc
[sudo] password for ihor: 
Updating installed gems
Nothing to update

How reproducible:
rhc app start -a rs
rhc app restart -a rs

Steps to Reproduce:
1. rhc app start -a rs
2. rhc app restart -a rs
3. rhc app tidy -a rs

Actual results:
ihor@ihor-M50Vc:~$ rhc app start -a rs
Password: ******
Unable to complete the requested operation due to: Node execution failure
(invalid exit code from node).  If the problem persists please contact Red Hat
support..
Reference ID: dab055723780cb9acaa4a103b831c2e1
ihor@ihor-M50Vc:~$ rhc app stop -a rs
Password: ******

RESULT:
rs stopped
ihor@ihor-M50Vc:~$ rhc app start -a rs
Password: ******
Unable to complete the requested operation due to: Node execution failure
(invalid exit code from node).  If the problem persists please contact Red Hat
support..
Reference ID: fe0c2d1c81a701f17c10e4b809ddaa26
ihor@ihor-M50Vc:~$ rhc app force-stop rs
Password: ******

RESULT:
rs force stopped
ihor@ihor-M50Vc:~$ rhc cartridge start mysql-5.1 -a rs
Password: ******

RESULT:
mysql-5.1 started
ihor@ihor-M50Vc:~$ rhc app start -a rs
Password: ******
Unable to complete the requested operation due to: Node execution failure
(invalid exit code from node).  If the problem persists please contact Red Hat
support..
Reference ID: 1e31c3bb0818592a85f603f045a3f386
ihor@ihor-M50Vc:~$ rhc app restart -a rs
Password: ******
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists
please contact Red Hat support..
Reference ID: fed4832448a636a01920395810064f9a
ihor@ihor-M50Vc:~$ rhc app tidy -a rs
Password: ******
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists
please contact Red Hat support..
Reference ID: 1de16735d0ef273c59f4506f7c376167
ihor@ihor-M50Vc:~$ 

-----------------------------------------------------------------------------

ihor@ihor-M50Vc:~$ rhc app restart -a rs -d
DEBUG: Authenticating with RHC::Auth::Basic
DEBUG: Connecting to https://openshift.redhat.com/broker/rest/api
DEBUG: Getting all domains
DEBUG: Request GET https://openshift.redhat.com/broker/rest/api
DEBUG:    code 200  717 ms
Password: ******
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains
DEBUG:    code 200  967 ms
DEBUG: Getting all domains
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains/emetemunoy/applications/rs
DEBUG:    code 200  922 ms
DEBUG: Restarting application rs
DEBUG: Request POST https://openshift.redhat.com/broker/rest/domains/emetemunoy/applications/rs/events
DEBUG:    code 500 54201 ms
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists
please contact Red Hat support..
Reference ID: b3581f6b1c5586631999c02a2612a34a

ihor@ihor-M50Vc:~$ rhc app start -a rs -d
DEBUG: Authenticating with RHC::Auth::Basic
DEBUG: Connecting to https://openshift.redhat.com/broker/rest/api
DEBUG: Getting all domains
DEBUG: Request GET https://openshift.redhat.com/broker/rest/api
DEBUG:    code 200  826 ms
Password: ******
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains
DEBUG:    code 200  947 ms
DEBUG: Getting all domains
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains/emetemunoy/applications/rs
DEBUG:    code 200  880 ms
DEBUG: Starting application rs
DEBUG: Request POST https://openshift.redhat.com/broker/rest/domains/emetemunoy/applications/rs/events
DEBUG:    code 500 60319 ms
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists
please contact Red Hat support..
Reference ID: 005512b8126fa30e2833742636acaf8f

ihor@ihor-M50Vc:~$ rhc app tidy -a rs -d
DEBUG: Authenticating with RHC::Auth::Basic
DEBUG: Connecting to https://openshift.redhat.com/broker/rest/api
DEBUG: Getting all domains
DEBUG: Request GET https://openshift.redhat.com/broker/rest/api
DEBUG:    code 200  816 ms
Password: ******
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains
DEBUG:    code 200  963 ms
DEBUG: Getting all domains
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains/emetemunoy/applications/rs
DEBUG:    code 200  964 ms
DEBUG: Starting application rs
DEBUG: Request POST https://openshift.redhat.com/broker/rest/domains/emetemunoy/applications/rs/events
DEBUG:    code 500 98056 ms
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists
please contact Red Hat support..
Reference ID: b9c7d2b159094c38da7871a17a3ff44b

Expected results:

Additional info:
about ping I live in Ukraine Europe.
looks like:
https://bugzilla.redhat.com/show_bug.cgi?id=966731

my Application maybe 4 days or more in downtime, from one bug to another one =))
you need Java Tester=)))? I can be perfect one))))

rs-emetemunoy.rhcloud.com

also I use medium gear, before I use small gear and this issue be too, but after magic manipulation server been started.
Now magic is not work, today moon phase 96% in Ukraine.

when I created this app, maybe ~ 5 times start restart with magic combinations worked for me, now always return errors.
Comment 1 Ihor 2013-05-31 08:51:40 EDT
ssh
[rs-emetemunoy.rhcloud.com 51a88672e0b8cddc6c00004a]\> ctl_all start
Starting services
MySQL already running
httpd (pid 10065) already running
Timed out waiting for http listening port
Failed to start jbossas-7
[rs-emetemunoy.rhcloud.com 51a88672e0b8cddc6c00004a]\>
Comment 2 Rajat Chopra 2013-05-31 13:53:32 EDT
Can you issue ctl_all stop and then list the processes that are running on the gear? 'ps -aef'

Also need boot.log and server.log from the gear.
Comment 3 Ihor 2013-05-31 14:13:09 EDT
Created attachment 755330 [details]
logs

[rs-emetemunoy.rhcloud.com 51a88672e0b8cddc6c00004a]\> ctl_all stop
Stopping services
Application is already stopped
Waiting for stop to finish
Done
[rs-emetemunoy.rhcloud.com 51a88672e0b8cddc6c00004a]\> 


[rs-emetemunoy.rhcloud.com 51a88672e0b8cddc6c00004a]\> ps -aef
UID        PID  PPID  C STIME TTY          TIME CMD
1688       844   493  0 13:56 ?        00:00:00 sshd: 51a88672e0b8cddc6c00004a@pts/0
1688       845   844  0 13:56 pts/0    00:00:00 /bin/bash --init-file /usr/bin/rhcsh -i
1688      1765   845  0 13:57 pts/0    00:00:00 ps -aef
[rs-emetemunoy.rhcloud.com 51a88672e0b8cddc6c00004a]\> 

------------------------------------------------------------------

[rs-emetemunoy.rhcloud.com 51a88672e0b8cddc6c00004a]\> ctl_all start
Starting services
Timed out waiting for http listening port
Failed to start jbossas-7
[rs-emetemunoy.rhcloud.com 51a88672e0b8cddc6c00004a]\> ps -aef
UID        PID  PPID  C STIME TTY          TIME CMD
1688       844   493  0 13:56 ?        00:00:00 sshd: 51a88672e0b8cddc6c00004a@pts/0
1688       845   844  0 13:56 pts/0    00:00:00 /bin/bash --init-file /usr/bin/rhcsh -i
1688      1859     1  0 13:57 pts/0    00:00:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/var/lib/openshift/51a88672e0b8cddc6c0000
1688      1988  1859  0 13:57 pts/0    00:00:00 /usr/libexec/mysqld --defaults-file=/var/lib/openshift/51a88672e0b8cddc6c00004a//mysql
1688      2051     1  0 13:57 ?        00:00:00 /usr/sbin/httpd -C Include /var/lib/openshift/51a88672e0b8cddc6c00004a//metrics-0.1/co
1688      2052  2051  0 13:57 ?        00:00:00 /usr/sbin/rotatelogs /var/lib/openshift/51a88672e0b8cddc6c00004a/metrics-0.1/logs/erro
1688      2056  2051  0 13:57 ?        00:00:00 /usr/sbin/rotatelogs /var/lib/openshift/51a88672e0b8cddc6c00004a/metrics-0.1/logs/acce
1688      2068  2051  0 13:57 ?        00:00:00 /usr/sbin/httpd -C Include /var/lib/openshift/51a88672e0b8cddc6c00004a//metrics-0.1/co
1688      2757   845  0 13:59 pts/0    00:00:00 ps -aef
[rs-emetemunoy.rhcloud.com 51a88672e0b8cddc6c00004a]\> 


2013/05/31 15:12:38,580 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015870: Deploy of deployment "ROOT.war" was rolled back with failure message Operation cancelled
2013/05/31 15:12:38,683 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [300 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.
Comment 4 Bill DeCoste 2013-05-31 17:14:50 EDT
Thanks for the logs. The key lines are 

2013/05/31 15:12:38,580 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015870: Deploy of deployment "ROOT.war" was rolled back with failure message Operation cancelled
2013/05/31 15:12:38,683 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [300 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.

The deployment timeout is being hit. To increase the deployment timeout, edit deployment-timeout in standalone.xml:

<subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">
    <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000" deployment-timeout="300"/>
</subsystem>
Comment 5 Ihor 2013-06-01 00:16:16 EDT
when I open rs/.openshift/config/standalone.xml I see I use before this configuration.

<subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">
<deployment-scanner path="deployments" relative-to="jboss.server.base.dir"
scan-interval="5000" deployment-timeout="300" />
</subsystem>

60 ms/60 = 1 min default
300 ms/60 = 5 min
600 ms/60 = 10 min
now I increase deployment-timeout="600"

deployment-timeout 	 LONG 	 Timeout, in seconds, a deployment is allows to execute before being canceled. The default is 60 seconds.
Comment 6 Ihor 2013-06-01 01:07:25 EDT
please read all: ">>>"
>>>in console "tail" nothing new.

ihor@ihor-M50Vc:~$ rhc app restart -a rs
Password: ******
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists
please contact Red Hat support..
Reference ID: aa3e84690f33808ba1664e320907a39e

>>>^also not help.

>>>in IDE Eclipse:
restart_on_add=false
Executing Jenkins build.

You can track your build at https://jenkins-emetemunoy.rhcloud.com/job/rs-build

Waiting for build to schedule.......
**BUILD FAILED/CANCELLED**
Please see the Jenkins log for more details via 'rhc tail'
!!!!!!!!
Deployment Halted!
If the build failed before the deploy step, your previous
build is still running.  Otherwise, your application may be
partially deployed or inaccessible.
Fix the build and try again.
!!!!!!!!
*********************************************************************
>>>in Jenkins output:
'ctl_all start'
    
    Type "help" for more info.
Starting services
WARNING: This ssh terminal was started without a tty.
          It is highly recommended to login with: ssh -t
MySQL already running
Timed out waiting for http listening port
Failed to start jbossas-7
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE

*********************************************************************
ihor@ihor-M50Vc:~$ rhc app restart -a rs
Password: ******
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists
please contact Red Hat support..
Reference ID: aa3e84690f33808ba1664e320907a39e
ihor@ihor-M50Vc:~$ rhc app force-stop rs
Password: ******

RESULT:
rs force stopped
ihor@ihor-M50Vc:~$ rhc cartridge start mysql-5.1 -a rs
Password: ******

RESULT:
mysql-5.1 started
ihor@ihor-M50Vc:~$ rhc app start -a rs
Password: ******
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists
please contact Red Hat support..
Reference ID: 683ae901782b2a5efd2df6d4c609b6b7
ihor@ihor-M50Vc:~$ 

>>> Looked on the date, force-stop, start, restart, not help AppServer have no command to start AppServer

2013/05/31 15:16:15,879 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015877: Stopped deployment ROOT.war in 217297ms
....
15:24:20,780 INFO  [org.jboss.as] JBAS015950: JBoss AS 7.1.1.Final "Brontes" stopped in 2691ms

>>> Looks like something is freeze on the some stage.
Comment 7 Ihor 2013-06-04 01:34:24 EDT
[rs-emetemunoy.rhcloud.com 51a88672e0b8cddc6c00004a]\> /bin/bash -e /usr/libexec/openshift/cartridges/jbossas-7/info/bin/app_ctl_impl.sh start
>Timed out waiting for http listening port


>>>When I run this command via SSH connection:
[rs-emetemunoy.rhcloud.com 51a88672e0b8cddc6c00004a]\> /bin/sh /var/lib/openshift/51a88672e0b8cddc6c00004a//jbossas-7/jbossas-7/bin/standalone.sh 
Removing existing modules
Removing existing standalone.xml
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /var/lib/openshift/51a88672e0b8cddc6c00004a/jbossas-7/jbossas-7

  JAVA: /usr/lib/jvm/jre-1.7.0/bin/java

  JAVA_OPTS:  -server -XX:+TieredCompilation -Xmx512m -XX:MaxPermSize=205m -XX:+AggressiveOpts -Dorg.apache.tomcat.util.LOW_MEMORY=true -Dorg.jboss.resolver.warning=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -Djava.net.preferIPv4Stack=true -Djboss.node.name=rs-emetemunoy.rhcloud.com -Djgroups.bind_addr=127.3.76.1 -Dorg.apache.coyote.http11.Http11Protocol.COMPRESSION=on

=========================================================================

01:18:12,691 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
01:18:14,991 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
..................

>>>Server is started and I can open my http://www.zinrad.com all is work good

>>> But when I close my SSH console, server is going down.

>>> How I can run my server without ssh access or in new independent thread?
Comment 8 Ihor 2013-06-04 02:06:58 EDT
> Also how I can increase this wonderful HTTP timeout?
> HardCode: while [ ${count} -lt 24 ]
cat /usr/libexec/openshift/cartridges/jbossas-7/info/bin/app_ctl_impl.sh
function start_app() {
    if [ -f "${OPENSHIFT_REPO_DIR}/.openshift/markers/enable_jpda" ]; then
       ENABLE_JPDA=1
    fi

    _state=`get_app_state`
    if [ -f $CART_DIR/run/stop_lock -o idle = "$_state" ]; then
        echo "Application is explicitly stopped!  Use 'rhc app start -a ${OPENSHIFT_APP_NAME}' to start back up." 1>&2
    else
        # Check for running app
        if isrunning; then
            echo "Application is already running" 1>&2
        else
            src_user_hook pre_start_${cartridge_type}
            set_app_state started
            # Start
            jopts="${JAVA_OPTS}"
            [ "${ENABLE_JPDA:-0}" -eq 1 ] && jopts="-Xdebug -Xrunjdwp:transport=dt_socket,address=$OPENSHIFT_INTERNAL_IP:8787,server=y,suspend=n ${JAVA_OPTS}"
            JAVA_OPTS="${jopts}" $APP_JBOSS_BIN_DIR/standalone.sh > ${APP_JBOSS_TMP_DIR}/${cartridge_type}.log 2>&1 &
            PROCESS_ID=$!
            echo $PROCESS_ID > $JBOSS_PID_FILE
            if ! ishttpup; then
                echo "Timed out waiting for http listening port"
                exit 1
            fi
            run_user_hook post_start_${cartridge_type}
        fi
    fi
}

# Check if the server http port is up
function ishttpup() {
    let count=0
    while [ ${count} -lt 24 ]
    do
        if /usr/sbin/lsof -P -n -i "@${OPENSHIFT_INTERNAL_IP}:8080" | grep "(LISTEN)" > /dev/null; then
            echo "Found ${OPENSHIFT_INTERNAL_IP}:8080 listening port"
            return 0
        fi
        let count=${count}+1
        sleep 2
    done
    return 1
}

:w !sudo tee %
/bin/bash: /usr/bin/sudo: Permission denied

shell returned 126

Press ENTER or type command to continue
Comment 9 Bill DeCoste 2013-06-04 07:58:03 EDT
To determine the cause of the Jenkins build failure please look in the jenkins.log file. There are 2 steps in a jenkins build

1) A Builder gear is created of the same type as your application. It looks from the logs that the builder gear failed to create. jenkins.log will show the cause
2) After the builder gear is created then the maven build itself will start on the builder gear. A failure at this stage is available in the build log.
Comment 10 Ihor 2013-06-04 08:08:49 EDT
(In reply to Bill DeCoste from comment #9)
> To determine the cause of the Jenkins build failure please look in the
> jenkins.log file. There are 2 steps in a jenkins build
> 
> 1) A Builder gear is created of the same type as your application. It looks
> from the logs that the builder gear failed to create. jenkins.log will show
> the cause
> 2) After the builder gear is created then the maven build itself will start
> on the builder gear. A failure at this stage is available in the build log.

>> "the cause of the Jenkins build failure"
> because "Timed out waiting for http listening port"


even if I remove Jenkins, server is also can't to be started.
I have try this before. I remove and create my rs application more than 7 times. I discover many of variants.


Now I connect via SSH and manually run my application with command:
/bin/sh /var/lib/openshift/51a88672e0b8cddc6c00004a//jbossas-7/jbossas-7/bin/standalone.sh 

but when I close my SSH terminal, Application Server going down on the your side.
Comment 11 Bill DeCoste 2013-06-04 08:17:08 EDT
Can you attach the latest server.log after you increased the deployment-timeout? The http error is saying that AS/EAP is not listening on 8080. 

Is this Origin or Enterprise?
Comment 12 Ihor 2013-06-04 08:57:24 EDT
Created attachment 756754 [details]
slogs

(In reply to Bill DeCoste from comment #11)
> Can you attach the latest server.log after you increased the
> deployment-timeout? The http error is saying that AS/EAP is not listening on
> 8080. 
> 
> Is this Origin or Enterprise?

This is Origin.

> Can you attach the latest server.log after

please look slogs.tar.gz
Comment 13 Paul Morie 2013-09-30 15:23:21 EDT
Is this still an issue?
Comment 14 Ihor 2013-09-30 16:20:10 EDT
ihor@ihor-M50Vc:~$ rhc app tidy -a rs
Your authorization token has expired. Please sign in now to continue.
Password: ******
Unable to complete the requested operation due to: Failed to correctly execute
all parallel operations.
Reference ID: 6b4932c7f9017247b47a78fd62d78d2a
ihor@ihor-M50Vc:~$ rhc app tidy -a rs
^CInterrupted
ihor@ihor-M50Vc:~$ rhc -d app tidy -a rs
DEBUG: Using config file /home/ihor/.openshift/express.conf
DEBUG: Authenticating with RHC::Auth::Token
DEBUG: Connecting to https://openshift.redhat.com/broker/rest/api
DEBUG: Getting all domains
DEBUG: Client supports API versions 1.1, 1.2, 1.3, 1.4, 1.5
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/api
DEBUG:    code 200  981 ms
DEBUG: Server supports API versions 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6
DEBUG: Client API version 1.5 is not current. Refetching API
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/api
DEBUG:    code 200  978 ms
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains
DEBUG:    code 200 1057 ms
DEBUG: Getting all domains
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains/emetemunoy/applications/rs
DEBUG:    code 200  314 ms
DEBUG: Starting application rs
DEBUG: Using token authentication
DEBUG: Request POST https://openshift.redhat.com/broker/rest/domains/emetemunoy/applications/rs/events
DEBUG:    code 503 50344 ms
Unable to perform action. Another operation is already running.
ihor@ihor-M50Vc:~$
Comment 15 Paul Morie 2013-10-01 10:09:19 EDT
Broker team, please take a look.
Comment 16 Rajat Chopra 2013-10-01 18:14:59 EDT
There is a lock timeout of 30minutes. Possible that an 'admin' maintenance job was working on it at the same time.

Could you convey the time of the incident, if you can recall?

Meanwhile, could I invite you to #openshift channel on irc/freenode? It will help us get down to the problem much faster.
Comment 17 Ihor 2013-10-10 04:20:28 EDT
ihor@ihor-M50Vc:~$ rhc app start rs
Your authorization token has expired. Please sign in now to continue.
Password: ******
Unable to complete the requested operation due to: Failed to correctly execute
all parallel operations.
Reference ID: bb7146f000069ec93517f304a6fb14f2
ihor@ihor-M50Vc:~$ 

again

>>> could I invite you to #openshift channel on irc/freenode?
yes but my timezone +2
Comment 18 Rajat Chopra 2013-10-10 13:38:42 EDT
Its back to 
"Failed to execute: 'control start' for /var/lib/openshift/51a88672e0b8cddc6c00004a/jbossas\nStarting jbossas cartridge\nTimed out waiting for http listening port\n\n"
Comment 19 Rajat Chopra 2013-10-11 16:19:38 EDT
Update: 
We are trying to see if an increase in the timeout for the cartridge helps with starting the app. That will help in isolating if the timeout message is masking the real error.
Will report with the results.
Comment 20 Paul Morie 2013-10-24 10:58:46 EDT
Ihor-

I had operations start your app without a timeout, but your application was already running.  Is this still an issue?  Please let us know.
Comment 21 Ihor 2013-10-24 14:18:58 EDT
All time when I try to git push I have:
Repository ssh://51a88672e0b8cddc6c00004a@rs-emetemunoy.rhcloud.com/~/git/rs.git/

Executing Jenkins build.

You can track your build at https://jenkins-emetemunoy.rhcloud.com/job/rs-build

ERROR - Job not available: http://jenkins-emetemunoy.rhcloud.com/job/rs-build
!!!!!!!!
Deployment Halted!
If the build failed before the deploy step, your previous
build is still running.  Otherwise, your application may be
partially deployed or inaccessible.
Fix the build and try again.
!!!!!!!!
An error occurred executing 'gear postreceive' (exit code: 1)
Error message: Failed to execute: 'control post-receive' for /var/lib/openshift/51a88672e0b8cddc6c00004a/jenkins-client

For more details about the problem, try running the command again with the '--trace' option.

========================================================================

[INFO] --- maven-resources-plugin:2.4.3:testResources (default-testResources) @ rs ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ rs ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 5 source files to /var/lib/openshift/52695d0b500446afa10000be/app-root/runtime/repo/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.7.2:test (default-test) @ rs ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-war-plugin:2.4:war (default-war) @ rs ---
[INFO] Packaging webapp
[INFO] Assembling webapp [rs] in [/var/lib/openshift/52695d0b500446afa10000be/app-root/runtime/repo/target/rs]
[INFO] Processing war project
[INFO] Copying webapp resources [/var/lib/openshift/52695d0b500446afa10000be/app-root/runtime/repo/src/main/webapp]
[INFO] Webapp assembled in [4520 msecs]
[INFO] Building war: /var/lib/openshift/52695d0b500446afa10000be/app-root/runtime/repo/deployments/ROOT.war
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 53.143s
[INFO] Finished at: Thu Oct 24 13:49:31 EDT 2013
[INFO] Final Memory: 34M/178M
[INFO] ------------------------------------------------------------------------
+ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 51a88672e0b8cddc6c00004a@rs-emetemunoy.rhcloud.com 'gear stop --conditional'
Stopping gear...
Stopping MySQL cartridge
Stopping jbossas cartridge
Sending SIGTERM to jboss:28386 ...
+ rsync --delete-after -azO -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/52695d0b500446afa10000be//.m2/ '51a88672e0b8cddc6c00004a@rs-emetemunoy.rhcloud.com:~/.m2/'
+ rsync --delete-after -azO -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/52695d0b500446afa10000be/app-root/runtime/repo/deployments/ '51a88672e0b8cddc6c00004a@rs-emetemunoy.rhcloud.com:${OPENSHIFT_REPO_DIR}deployments/'
+ rsync --delete-after -azO -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/52695d0b500446afa10000be/app-root/runtime/repo/.openshift/ '51a88672e0b8cddc6c00004a@rs-emetemunoy.rhcloud.com:${OPENSHIFT_REPO_DIR}.openshift/'
+ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 51a88672e0b8cddc6c00004a@rs-emetemunoy.rhcloud.com 'gear remotedeploy'
Starting application rs
Starting MySQL cartridge
Deploying JBoss
Starting jbossas cartridge
Timed out waiting for http listening port
An error occurred executing 'gear remotedeploy' (exit code: 1)
Error message: Failed to execute: 'control start' for /var/lib/openshift/51a88672e0b8cddc6c00004a/jbossas

For more details about the problem, try running the command again with the '--trace' option.
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE

===========================================

when I try to open http://www.zinrad.com/ua/index.html
and have:

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Apache/2.2.15 (Red Hat) Server at www.zinrad.com Port 80
===========================================
ihor@ihor-M50Vc:~$ sudo gem update rhc
[sudo] password for ihor: 
Updating installed gems
Nothing to update
ihor@ihor-M50Vc:~$ 




ihor@ihor-M50Vc:~$ rhc -d app start rs
DEBUG: Using config file /home/ihor/.openshift/express.conf
DEBUG: Authenticating with RHC::Auth::Token
DEBUG: Connecting to https://openshift.redhat.com/broker/rest/api
DEBUG: Getting all domains
DEBUG: Client supports API versions 1.1, 1.2, 1.3, 1.4, 1.5
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/api
DEBUG:    code 200 5425 ms
DEBUG: Server supports API versions 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6
DEBUG: Client API version 1.5 is not current. Refetching API
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/api
DEBUG:    code 200 1904 ms
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains
DEBUG:    code 200 2026 ms
DEBUG: Getting all domains
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains/emetemunoy/applications/rs
DEBUG:    code 200  539 ms
DEBUG: Starting application rs
DEBUG: Using token authentication
DEBUG: Request POST https://openshift.redhat.com/broker/rest/domain/emetemunoy/application/rs/events
DEBUG:    code 500 135088 ms
Unable to complete the requested operation due to: Failed to correctly execute
all parallel operations.
Reference ID: ed5fb830f94a2879ff425837759e32ee
ihor@ihor-M50Vc:~$ 



ihor@ihor-M50Vc:~$ rhc -d app restart rs
DEBUG: Using config file /home/ihor/.openshift/express.conf
DEBUG: Authenticating with RHC::Auth::Token
DEBUG: Connecting to https://openshift.redhat.com/broker/rest/api
DEBUG: Getting all domains
DEBUG: Client supports API versions 1.1, 1.2, 1.3, 1.4, 1.5
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/api
DEBUG:    code 200 1523 ms
DEBUG: Server supports API versions 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6
DEBUG: Client API version 1.5 is not current. Refetching API
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/api
DEBUG:    code 200 1799 ms
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains
DEBUG:    code 200 5039 ms
DEBUG: Getting all domains
DEBUG: Using token authentication
DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains/emetemunoy/applications/rs
DEBUG:    code 200 1392 ms
DEBUG: Restarting application rs
DEBUG: Using token authentication
DEBUG: Request POST https://openshift.redhat.com/broker/rest/domain/emetemunoy/application/rs/events
DEBUG:    code 500 240635 ms
Unable to complete the requested operation due to: Failed to correctly execute all parallel operations.
Reference ID: a763cb7926b5d1eae9155cbef9ecd771
ihor@ihor-M50Vc:~$ 

when I use rhc app restart rs - all work - but I need doit this all time when I push some changes, Or you(openshift) have some issue or upgrading servers. My app not start automatically.
Comment 22 Paul Morie 2013-10-28 16:50:32 EDT
Ihor-

I had operations restart your gear without a timeout, and it looks like the restart is taking longer than our current timeout for operations on the node (240s).  For now, we cannot increase the timeout, but the upcoming scheduler work will allow us to increase it in the future.
Comment 23 Ihor 2013-10-29 03:43:40 EDT
Paul-
Also, I add hot deploy marker to my project, and now without the server restart - all work, without any error messages - in IDE (git push) and Jenkins.

I didn't try console command: restart or start an application. (in terminal)

Note You need to log in before you can comment on or make changes to this bug.