Attachment 950949 Details for Bug 1139735 – server1 upgrade log

server1 upgrade log

sever1Upgrade.out (text/plain), 16.37 KB, created by Filip Brychta on 2014-10-27 11:09:46 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Filip Brychta

Created: 2014-10-27 11:09:46 UTC

Size: 16.37 KB

patch

obsolete

>[hudson@fbr-ha ~]$ jon-server-3.3.0.ER05/bin/rhqctl upgrade --from-server-dir /home/hudson/jon-server-3.2.0.GA/
>05:05:39,212 INFO  [org.jboss.modules] JBoss Modules version 1.3.3.Final-redhat-1
>05:05:39,596 INFO  [org.rhq.server.control.command.Upgrade] Stopping any running RHQ components...
>05:05:40,510 INFO  [org.jboss.modules] JBoss Modules version 1.2.2.Final-redhat-1
>05:05:40,726 INFO  [org.rhq.server.control.command.Upgrade] The old installation components have been stopped
>Starting RHQ Storage Installer ...
>05:06:11,887 INFO  [org.jboss.modules] JBoss Modules version 1.3.3.Final-redhat-1
>05:06:12,144 INFO  [org.rhq.storage.installer.StorageInstaller] Running RHQ Storage Node installer...
>05:06:12,176 INFO  [org.rhq.cassandra.Deployer] Unzipping storage node to /home/hudson/jon-server-3.3.0.ER05/rhq-storage
>05:06:12,545 INFO  [org.rhq.cassandra.Deployer] Applying configuration changes to /home/hudson/jon-server-3.3.0.ER05/rhq-storage/conf/cassandra.yaml
>05:06:12,662 INFO  [org.rhq.cassandra.Deployer] Applying configuration changes to /home/hudson/jon-server-3.3.0.ER05/rhq-storage/conf/log4j-server.properties
>05:06:12,670 INFO  [org.rhq.cassandra.Deployer] Applying configuration changes to /home/hudson/jon-server-3.3.0.ER05/rhq-storage/conf/cassandra-jvm.properties
>05:06:12,679 INFO  [org.rhq.cassandra.Deployer] Updating file permissions in /home/hudson/jon-server-3.3.0.ER05/rhq-storage/bin
>05:06:13,009 INFO  [org.rhq.storage.installer.StorageInstaller] Updating rhq-server.properties...
>05:06:13,598 INFO  [org.rhq.storage.installer.StorageInstaller] Starting RHQ Storage Node
>05:06:26,129 INFO  [org.rhq.storage.installer.StorageInstaller] RHQ Storage Node is up and running and ready to service client requests
>05:06:26,137 INFO  [org.rhq.storage.installer.StorageInstaller] Installation of the storage node has completed successfully.
>05:06:26,167 INFO  [org.rhq.server.control.command.Upgrade] The storage node upgrade has finished with an exit value of 0
>05:06:26,769 INFO  [org.rhq.server.control.command.Upgrade] Looks like the new server already has an Oracle driver: /home/hudson/jon-server-3.3.0.ER05/modules/org/rhq/oracle/main/ojdbc6-12.1.0.1.0.jar
>05:06:26,856 INFO  [org.rhq.server.control.command.Upgrade] The RHQ Server must be started to complete its installation. Starting the RHQ server in preparation of running the server installer...
>05:06:26,872 INFO  [org.rhq.server.control.command.Upgrade] Waiting for the RHQ Server to start in preparation of running the server installer...
>Trying to start the RHQ Server...
>RHQ Server                     (pid 15188  ) is starting
>05:06:33,561 INFO  [org.jboss.modules] JBoss Modules version 1.3.3.Final-redhat-1
>05:06:33,786 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] The server is preconfigured and ready for auto-install.
>05:06:33,866 INFO  [org.xnio] XNIO Version 3.0.10.GA-redhat-1
>05:06:33,875 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.10.GA-redhat-1
>05:06:33,908 INFO  [org.jboss.remoting] JBoss Remoting version (unknown)
>05:06:34,556 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] Installing into app server version [7.4.0.Final-redhat-19]
>05:06:34,750 INFO  [org.rhq.enterprise.server.installer.ServerInstallUtil] Database is supported: postgresql8.4(postgresql:8.4)
>05:06:34,781 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] This [[name=fbr-ha.bc.jonqe.lab.eng.bos.redhat.com, address=10.16.23.105, port=7080, secureport=7443]] will be considered a reinstallation of an existing server [[name=fbr-ha.bc.jonqe.lab.eng.bos.redhat.com, address=10.16.23.105, port=7080, secureport=7443]]
>05:06:34,782 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] The app server where the installation will go is found at: /home/hudson/jon-server-3.3.0.ER05
>05:06:34,783 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] It looks like everything is OK and you can start the installation.
>05:06:34,788 INFO  [org.rhq.server.control.command.Upgrade] The RHQ Server is ready to be upgraded by the server installer.
>05:06:34,790 INFO  [org.rhq.server.control.command.Upgrade] Upgrading RHQ Server
>05:06:34,791 INFO  [org.rhq.server.control.command.Upgrade] The RHQ Server installer is running
>05:06:35,041 INFO  [org.xnio] XNIO Version 3.0.10.GA-redhat-1
>05:06:35,071 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.10.GA-redhat-1
>05:06:35,112 INFO  [org.jboss.remoting] JBoss Remoting version (unknown)
>05:06:35,737 INFO  [org.jboss.modules] JBoss Modules version 1.3.3.Final-redhat-1
>05:06:35,891 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] The server is preconfigured and ready for auto-install.
>05:06:35,966 INFO  [org.xnio] XNIO Version 3.0.10.GA-redhat-1
>05:06:35,975 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.10.GA-redhat-1
>05:06:35,998 INFO  [org.jboss.remoting] JBoss Remoting version (unknown)
>05:06:36,492 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] Installing into app server version [7.4.0.Final-redhat-19]
>05:06:36,772 INFO  [org.rhq.enterprise.server.installer.ServerInstallUtil] Database is supported: postgresql8.4(postgresql:8.4)
>05:06:36,803 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] Database setup log file directory: /home/hudson/jon-server-3.3.0.ER05/logs
>05:06:36,841 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] Database schema exists - it will now be updated.
>05:06:37,354 INFO  [org.rhq.cassandra.schema.VersionManager] Preparing to check storage schema compatibility.
>05:06:37,651 INFO  [org.rhq.cassandra.schema.VersionManager] Completed storage schema compatibility check.
>05:06:37,651 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] Install RHQ schema along with updates to storage nodes.
>05:06:37,653 INFO  [org.rhq.cassandra.schema.VersionManager] Preparing to install storage schema
>05:06:37,654 INFO  [org.rhq.cassandra.schema.AbstractManager] Shutting down existing cluster connections
>05:06:37,683 INFO  [org.rhq.cassandra.schema.AbstractManager] Shutting down existing cluster connections
>05:06:37,710 ERROR [org.rhq.cassandra.schema.VersionManager] com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host fbr-ha.bc.jonqe.lab.eng.bos.redhat.com/10.16.23.105: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
>05:06:37,712 INFO  [org.rhq.cassandra.schema.AbstractManager] Shutting down existing cluster connections
>05:06:37,718 ERROR [org.rhq.enterprise.server.installer.InstallerServiceImpl] Could not complete storage cluster schema installation: Authentication error on host fbr-ha.bc.jonqe.lab.eng.bos.redhat.com/10.16.23.105: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM: java.lang.RuntimeException: com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host fbr-ha.bc.jonqe.lab.eng.bos.redhat.com/10.16.23.105: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
>	at org.rhq.cassandra.schema.VersionManager.create(VersionManager.java:132) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.cassandra.schema.VersionManager.install(VersionManager.java:89) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.cassandra.schema.SchemaManager.install(SchemaManager.java:123) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.enterprise.server.installer.InstallerServiceImpl.prepareStorageSchema(InstallerServiceImpl.java:700) [rhq-installer-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.enterprise.server.installer.InstallerServiceImpl.prepareDatabase(InstallerServiceImpl.java:644) [rhq-installer-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.enterprise.server.installer.InstallerServiceImpl.install(InstallerServiceImpl.java:409) [rhq-installer-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.enterprise.server.installer.Installer.doInstall(Installer.java:134) [rhq-installer-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.enterprise.server.installer.Installer.main(Installer.java:59) [rhq-installer-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65]
>	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65]
>	at org.jboss.modules.Module.run(Module.java:312) [jboss-modules.jar:1.3.3.Final-redhat-1]
>	at org.jboss.modules.Main.main(Main.java:460) [jboss-modules.jar:1.3.3.Final-redhat-1]
>Caused by: com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host fbr-ha.bc.jonqe.lab.eng.bos.redhat.com/10.16.23.105: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
>	at com.datastax.driver.core.Connection.initializeTransport(Connection.java:169) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Connection.<init>(Connection.java:130) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Connection$Factory.open(Connection.java:417) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:204) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:167) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:80) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:839) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Cluster$Manager.access$100(Cluster.java:763) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Cluster.<init>(Cluster.java:93) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:732) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at org.rhq.cassandra.util.ClusterBuilder.build(ClusterBuilder.java:130) [rhq-cassandra-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.cassandra.schema.SessionManager.initSession(SessionManager.java:33) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.cassandra.schema.AbstractManager.initClusterSession(AbstractManager.java:100) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.cassandra.schema.VersionManager.create(VersionManager.java:115) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	... 13 more
>
>05:06:37,739 ERROR [org.rhq.enterprise.server.installer.Installer] The installer will now exit due to previous errors: java.lang.Exception: Could not complete storage cluster schema installation: Authentication error on host fbr-ha.bc.jonqe.lab.eng.bos.redhat.com/10.16.23.105: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
>	at org.rhq.enterprise.server.installer.InstallerServiceImpl.prepareStorageSchema(InstallerServiceImpl.java:734) [rhq-installer-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.enterprise.server.installer.InstallerServiceImpl.prepareDatabase(InstallerServiceImpl.java:644) [rhq-installer-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.enterprise.server.installer.InstallerServiceImpl.install(InstallerServiceImpl.java:409) [rhq-installer-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.enterprise.server.installer.Installer.doInstall(Installer.java:134) [rhq-installer-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.enterprise.server.installer.Installer.main(Installer.java:59) [rhq-installer-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65]
>	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65]
>	at org.jboss.modules.Module.run(Module.java:312) [jboss-modules.jar:1.3.3.Final-redhat-1]
>	at org.jboss.modules.Main.main(Main.java:460) [jboss-modules.jar:1.3.3.Final-redhat-1]
>Caused by: java.lang.RuntimeException: com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host fbr-ha.bc.jonqe.lab.eng.bos.redhat.com/10.16.23.105: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
>	at org.rhq.cassandra.schema.VersionManager.create(VersionManager.java:132) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.cassandra.schema.VersionManager.install(VersionManager.java:89) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.cassandra.schema.SchemaManager.install(SchemaManager.java:123) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.enterprise.server.installer.InstallerServiceImpl.prepareStorageSchema(InstallerServiceImpl.java:700) [rhq-installer-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	... 10 more
>Caused by: com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host fbr-ha.bc.jonqe.lab.eng.bos.redhat.com/10.16.23.105: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
>	at com.datastax.driver.core.Connection.initializeTransport(Connection.java:169) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Connection.<init>(Connection.java:130) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Connection$Factory.open(Connection.java:417) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:204) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:167) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:80) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:839) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Cluster$Manager.access$100(Cluster.java:763) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Cluster.<init>(Cluster.java:93) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:732) [cassandra-driver-core-1.0.8-jboss-1.jar:]
>	at org.rhq.cassandra.util.ClusterBuilder.build(ClusterBuilder.java:130) [rhq-cassandra-util-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.cassandra.schema.SessionManager.initSession(SessionManager.java:33) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.cassandra.schema.AbstractManager.initClusterSession(AbstractManager.java:100) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	at org.rhq.cassandra.schema.VersionManager.create(VersionManager.java:115) [rhq-cassandra-schema-4.12.0.JON330ER05.jar:4.12.0.JON330ER05]
>	... 13 more
>
>Trying to stop the RHQ Server...
>RHQ Server (pid=15188) is stopping...
>RHQ Server has stopped.
>RHQ Server                     (no pid file) is âdown
>Stopping RHQ storage node...
>RHQ storage node (pid=14942) is stopping...
>RHQ storage node has stopped
>05:06:49,110 ERROR [org.rhq.server.control.RHQControl] Installer failed with code 2, shut down server
>05:06:49,111 WARN  [org.rhq.server.control.command.Upgrade] UNDO: Removing server-installed marker file and management user and reverting to original standalone-full.xml
>05:06:49,113 WARN  [org.rhq.server.control.command.Upgrade] UNDO: Stopping component: --server
>RHQ Server                     (no pid file) is âdown
>05:06:49,206 WARN  [org.rhq.server.control.command.Upgrade] UNDO: Stopping component: --storage
>05:06:49,207 WARN  [org.rhq.server.control.command.Upgrade] UNDO: Removing new storage node install directory
>Starting RHQ Storage Installer ...
>05:06:50,291 INFO  [org.jboss.modules] JBoss Modules version 1.3.3.Final-redhat-1
>05:06:50,541 INFO  [org.rhq.storage.installer.StorageInstaller] Running RHQ Storage Node installer...
>05:06:50,548 INFO  [org.rhq.storage.installer.StorageInstaller] Undoing storage node version stamp...
>05:06:51,577 WARN  [org.rhq.server.control.command.Upgrade] UNDO: Reverting server properties file
>

Actions: View

Attachments on bug 1139735: 950949 | 950950