Bug 1035160

This is fixed with the packages listed on the latest pad: http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-11-27

This bug still could be reproduced with 
openshift-enterprise-release-2.0.0c-1.git.2.acc7a39.el6op.noarch
openshift-enterprise-upgrade-node-2.0.0c-1.git.2.acc7a39.el6op.noarch

1. Setup ose-1.2 env
2. Create scalable jbosseap-6.0 app on the node
3. Upgrade env from 1.2 to 2.0
4. Try to restart the jbosseap-6 app
[root@broker openshift]# rhc app restart app2
Unable to complete the requested operation due to: Failed to correctly execute all parallel operations.
Reference ID: da8df8412a6f8c930b723ae7786d8b7c

The app is not available, no java process for the app


Error logs in /var/log/ruby193-mcollective.log
E, [2013-11-28T02:08:22.288491 #1397] ERROR -- : openshift.rb:310:in `rescue in with_container_from_args' undefined method `values' for nil:NilClass
E, [2013-11-28T02:08:22.288670 #1397] ERROR -- : openshift.rb:311:in `rescue in with_container_from_args' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1050:in `with_gear_rotation'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1156:in `restart'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:956:in `block in oo_restart'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:299:in `with_container_from_args'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:955:in `oo_restart'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:139:in `execute_action'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:195:in `block in execute_parallel_action'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:188:in `each'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:188:in `execute_parallel_action'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'


Error logs in jbosseap/logs/server.log of the jbosseap app
2013/11/28 02:00:15,397 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/]] (ServerService Thread Pool -- 67) JBWEB000306: Exception sending context destroyed event to listener instance of class com.sun.faces.config.ConfigureListener: java.lang.ExceptionInInitializerError
        at com.sun.faces.config.ConfigManager.releaseFactories(ConfigManager.java:718) [jsf-impl.jar:2.1.19-redhat-1]
        at com.sun.faces.config.ConfigManager.destroy(ConfigManager.java:401) [jsf-impl.jar:2.1.19-redhat-1]
        at com.sun.faces.config.ConfigureListener.contextDestroyed(ConfigureListener.java:333) [jsf-impl.jar:2.1.19-redhat-1]
        at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3427) [jbossweb.jar:7.2.2.Final-redhat-1]
        at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3920) [jbossweb.jar:7.2.2.Final-redhat-1]
        at org.jboss.as.web.deployment.WebDeploymentService.doStop(WebDeploymentService.java:171) [jboss-as-web.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.web.deployment.WebDeploymentService.access$100(WebDeploymentService.java:60) [jboss-as-web.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.web.deployment.WebDeploymentService$2.run(WebDeploymentService.java:113) [jboss-as-web.jar:7.2.1.Final-redhat-10]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.6.0_28]
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.6.0_28]
        at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.6.0_28]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) [rt.jar:1.6.0_28]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.6.0_28]
        at java.lang.Thread.run(Thread.java:701) [rt.jar:1.6.0_28]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: java.util.MissingResourceException: Can't find javax.faces.LogStrings bundle
        at java.util.logging.Logger.setupResourceInfo(Logger.java:1488) [rt.jar:1.6.0_28]
        at java.util.logging.Logger.getLogger(Logger.java:423) [rt.jar:1.6.0_28]
        at javax.faces.FactoryFinder.<clinit>(FactoryFinder.java:296) [jboss-jsf-api_2.1_spec.jar:2.1.19.1.Final-redhat-1]
        ... 15 more

2013/11/28 02:00:15,435 INFO  [org.hornetq.ra] (MSC service thread 1-4) HQ151003: HornetQ resource adaptor stopped
2013/11/28 02:00:15,448 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment ROOT.war (runtime-name: ROOT.war) in 170ms
2013/11/28 02:00:15,481 INFO  [org.hornetq.core.server] (MSC service thread 1-3) HQ221002: HornetQ Server version 2.3.5.Final-redhat-2 (Monster Bee, 123) [109496a9-5809-11e3-aa46-85346bfff765] stopped

This is fixed in the latest upgrade puddle: http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-02

Please note that this pad has changed the upgrade steps to use a puddle, so there are a few workarounds that need to be done to make sure the right bits are installed during the upgrade.

I tested with jbosseap-6 in both scalable and non-scalable applications.

I missed a package in the previous pad that caused the gear upgrades to be skipped

Rest this bug following http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-02, still failed.

Now I already set up an ose-1.2 test bed for you, I also created all kinds of cartridge app, you could use it for upgrade testing to reproduce this bug.

Broker: 10.4.59.154 root/redhat
Node: 10.4.59.132 root/redhat

See the same error from /var/lib/openshift/52958455aeb9059466000020/.tmp/52958455aeb9059466000020/jbosseap.log

Error: Unable to access jarfile /var/lib/openshift/52958455aeb9059466000020/jbosseap/jboss-modules.jar

After doing the upgrade steps in http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-02 on the Broker and Node that you provided, I was not able to reproduce this bug.  (On a side note, I did update the pad later in the evening US time, so it may be that you tested against one of the earlier and incomplete puddles)

The only error I see in the logs after the migration is:
2013/12/03 11:32:00,534 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/]] (ServerService Thread Pool -- 68) JBWEB000306: Exception sending context dest
royed event to listener instance of class com.sun.faces.config.ConfigureListener: java.lang.ExceptionInInitializerError
        at com.sun.faces.config.ConfigManager.releaseFactories(ConfigManager.java:718) [jsf-impl.jar:2.1.19-redhat-1]
and that is on container shutdown, the logs pertaining to the startup after that are clean.

For the symlink, it is broken upstream as well, but does not appear to be affecting functionality (although an upstream bug should probably be filed to clean it up).

Verified this bug following http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-04, and PASS.


For the symlink, after upgrade, it is linked to correct target.