Hide Forgot
Description of problem: In ose-1.2, jbosseap cartridge is referencing some jbosseap-6.0 file, but after upgrade, jbosseap-6.0 is replaced by jbosseap-6, that will cause jbosseap app is broken. Version-Release number of selected component (if applicable): openshift-enterprise-upgrade-node-2.0.0b-1.git.16.a0bff65.el6op.noarch openshift-enterprise-release-2.0.0b-1.git.16.a0bff65.el6op.noarch How reproducible: Always Steps to Reproduce: 1. Setup ose-1.2 env 2. Create jbosseap-6.0 app on the node 3. Upgrade env from 1.2 to 2.0 Actual results: jbosseap app is not available, found no java process for jbosseap app. Then try to start jbosseap manuall via rhc client. Seen from app log - /var/lib/openshift/52958455aeb9059466000020/.tmp/52958455aeb9059466000020/jbosseap.log Removing existing standalone.xml ========================================================================= JBoss Bootstrap Environment JBOSS_HOME: /var/lib/openshift/52958455aeb9059466000020/jbosseap JAVA: /etc/alternatives/java_sdk_1.7.0/bin/java JAVA_OPTS: -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xmx256m -XX:MaxPermSize=102m -XX:+AggressiveOpts -Dorg.apache.tomcat.util.LOW_MEMORY=true -DOPENSHIFT_APP_UUID=52958455aeb9059466000020 -Dorg.jboss.resolver.warning=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -Djava.net.preferIPv4Stack=true -Djboss.node.name=jbeapscal-jialiu.my-ose.com.cn -Djgroups.bind_addr=127.1.244.129 -Dorg.apache.coyote.http11.Http11Protocol.COMPRESSION=on ========================================================================= Error: Unable to access jarfile /var/lib/openshift/52958455aeb9059466000020/jbosseap/jboss-modules.jar # ll /var/lib/openshift/52958455aeb9059466000020/jbosseap/jboss-modules.jar lrwxrwxrwx. 1 52958455aeb9059466000020 52958455aeb9059466000020 48 Nov 27 00:34 /var/lib/openshift/52958455aeb9059466000020/jbosseap/jboss-modules.jar -> /etc/alternatives/jbosseap-6.0/jboss-modules.jar # ll /etc/alternatives/jbosseap-6.0/jboss-modules.jar ls: cannot access /etc/alternatives/jbosseap-6.0/jboss-modules.jar: No such file or directory In app dir, jboss-modules.jar and modules are symbolic link file to jbosseap-6.0, but after upgrade, jbosseap-6.0 is upgraded to jbosseap-6, these symbolic link is broken. Expected results: After upgrade, jbosseap should be still available. Additional info:
This is fixed with the packages listed on the latest pad: http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-11-27
This bug still could be reproduced with openshift-enterprise-release-2.0.0c-1.git.2.acc7a39.el6op.noarch openshift-enterprise-upgrade-node-2.0.0c-1.git.2.acc7a39.el6op.noarch 1. Setup ose-1.2 env 2. Create scalable jbosseap-6.0 app on the node 3. Upgrade env from 1.2 to 2.0 4. Try to restart the jbosseap-6 app [root@broker openshift]# rhc app restart app2 Unable to complete the requested operation due to: Failed to correctly execute all parallel operations. Reference ID: da8df8412a6f8c930b723ae7786d8b7c The app is not available, no java process for the app Error logs in /var/log/ruby193-mcollective.log E, [2013-11-28T02:08:22.288491 #1397] ERROR -- : openshift.rb:310:in `rescue in with_container_from_args' undefined method `values' for nil:NilClass E, [2013-11-28T02:08:22.288670 #1397] ERROR -- : openshift.rb:311:in `rescue in with_container_from_args' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1050:in `with_gear_rotation' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1156:in `restart' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:956:in `block in oo_restart' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:299:in `with_container_from_args' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:955:in `oo_restart' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:139:in `execute_action' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:195:in `block in execute_parallel_action' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:188:in `each' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:188:in `execute_parallel_action' /opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg' /opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch' /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout' /opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch' Error logs in jbosseap/logs/server.log of the jbosseap app 2013/11/28 02:00:15,397 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/]] (ServerService Thread Pool -- 67) JBWEB000306: Exception sending context destroyed event to listener instance of class com.sun.faces.config.ConfigureListener: java.lang.ExceptionInInitializerError at com.sun.faces.config.ConfigManager.releaseFactories(ConfigManager.java:718) [jsf-impl.jar:2.1.19-redhat-1] at com.sun.faces.config.ConfigManager.destroy(ConfigManager.java:401) [jsf-impl.jar:2.1.19-redhat-1] at com.sun.faces.config.ConfigureListener.contextDestroyed(ConfigureListener.java:333) [jsf-impl.jar:2.1.19-redhat-1] at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3427) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3920) [jbossweb.jar:7.2.2.Final-redhat-1] at org.jboss.as.web.deployment.WebDeploymentService.doStop(WebDeploymentService.java:171) [jboss-as-web.jar:7.2.1.Final-redhat-10] at org.jboss.as.web.deployment.WebDeploymentService.access$100(WebDeploymentService.java:60) [jboss-as-web.jar:7.2.1.Final-redhat-10] at org.jboss.as.web.deployment.WebDeploymentService$2.run(WebDeploymentService.java:113) [jboss-as-web.jar:7.2.1.Final-redhat-10] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.6.0_28] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.6.0_28] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.6.0_28] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) [rt.jar:1.6.0_28] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.6.0_28] at java.lang.Thread.run(Thread.java:701) [rt.jar:1.6.0_28] at org.jboss.threads.JBossThread.run(JBossThread.java:122) Caused by: java.util.MissingResourceException: Can't find javax.faces.LogStrings bundle at java.util.logging.Logger.setupResourceInfo(Logger.java:1488) [rt.jar:1.6.0_28] at java.util.logging.Logger.getLogger(Logger.java:423) [rt.jar:1.6.0_28] at javax.faces.FactoryFinder.<clinit>(FactoryFinder.java:296) [jboss-jsf-api_2.1_spec.jar:2.1.19.1.Final-redhat-1] ... 15 more 2013/11/28 02:00:15,435 INFO [org.hornetq.ra] (MSC service thread 1-4) HQ151003: HornetQ resource adaptor stopped 2013/11/28 02:00:15,448 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment ROOT.war (runtime-name: ROOT.war) in 170ms 2013/11/28 02:00:15,481 INFO [org.hornetq.core.server] (MSC service thread 1-3) HQ221002: HornetQ Server version 2.3.5.Final-redhat-2 (Monster Bee, 123) [109496a9-5809-11e3-aa46-85346bfff765] stopped
This is fixed in the latest upgrade puddle: http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-02 Please note that this pad has changed the upgrade steps to use a puddle, so there are a few workarounds that need to be done to make sure the right bits are installed during the upgrade. I tested with jbosseap-6 in both scalable and non-scalable applications. I missed a package in the previous pad that caused the gear upgrades to be skipped
Rest this bug following http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-02, still failed. Now I already set up an ose-1.2 test bed for you, I also created all kinds of cartridge app, you could use it for upgrade testing to reproduce this bug. Broker: 10.4.59.154 root/redhat Node: 10.4.59.132 root/redhat
See the same error from /var/lib/openshift/52958455aeb9059466000020/.tmp/52958455aeb9059466000020/jbosseap.log Error: Unable to access jarfile /var/lib/openshift/52958455aeb9059466000020/jbosseap/jboss-modules.jar
After doing the upgrade steps in http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-02 on the Broker and Node that you provided, I was not able to reproduce this bug. (On a side note, I did update the pad later in the evening US time, so it may be that you tested against one of the earlier and incomplete puddles) The only error I see in the logs after the migration is: 2013/12/03 11:32:00,534 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/]] (ServerService Thread Pool -- 68) JBWEB000306: Exception sending context dest royed event to listener instance of class com.sun.faces.config.ConfigureListener: java.lang.ExceptionInInitializerError at com.sun.faces.config.ConfigManager.releaseFactories(ConfigManager.java:718) [jsf-impl.jar:2.1.19-redhat-1] and that is on container shutdown, the logs pertaining to the startup after that are clean. For the symlink, it is broken upstream as well, but does not appear to be affecting functionality (although an upstream bug should probably be filed to clean it up).
Verified this bug following http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-04, and PASS. For the symlink, after upgrade, it is linked to correct target.