Description of problem: rhevh upgrade fails on java.io.IOException. 2012-07-16 09:23:08,868 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-44) [5b5b5b66] Installation of white-vdsf.eng.lab.tlv.redhat.com. Received message: Preparing ISO file. FYI. (Stage: Upload upgrade ISO to oVirt Node) 2012-07-16 09:23:08,868 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-44) [5b5b5b66] VDS message: Preparing ISO file 2012-07-16 09:23:08,871 ERROR [org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper] (pool-4-thread-44) uploadLargeFile: Unable to copy local file: : java.io.IOExce ption: Failed to create directory /null/deploy/ROOT.war at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.fastFileCopy(MinaInstallWrapper.java:1009) [engine-utils.jar:] at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.uploadLargeFile(MinaInstallWrapper.java:546) [engine-utils.jar:] at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.UploadFile(MinaInstallWrapper.java:411) [engine-utils.jar:] at org.ovirt.engine.core.bll.OVirtInstaller.RunStage(OVirtInstaller.java:48) [engine-bll.jar:] at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:234) [engine-bll.jar:] at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] at org.ovirt.engine.core.bll.CommandBase.ExecuteWithoutTransaction(CommandBase.java:805) [engine-bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:897) [engine-bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1204) [engine-bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] at org.ovirt.engine.core.bll.CommandBase.Execute(CommandBase.java:912) [engine-bll.jar:] at org.ovirt.engine.core.bll.CommandBase.ExecuteAction(CommandBase.java:269) [engine-bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_03-icedtea] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_03-icedtea] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_03-icedtea] Version-Release number of selected component (if applicable): si10 How reproducible: Always Steps to Reproduce: 1. Try to upgrade rhevh from the UI 2. 3. Actual results: Expected results: Additional info:
Created attachment 598375 [details] logs
From MinaInstallWrapper.java - try { callbackAddMessage("Preparing ISO file"); copyDest = Path.Combine( Path.Combine(Path.Combine(System.getProperty("jboss.server.home.dir"), "deploy"), "ROOT.war"), For some reason - jboss.server.home.dir is null. Juan, do you have any insights?
This hasn't been touched since JBoss 5 times. Let me take a look.
If we need to solve this quickly we need to replace with the following: Path.Combine(Path.Combine(System.getProperty("jboss.server.base.dir"), "deployments/engine.ear"), "root.war") But this needs a deeper change, as writing files to the /usr directory is not a good practice.
Sorry, the previous comment was wrong. The correct way to do this is: Path.Combine(Path.Combine(System.getProperty("jboss.server.data.dir"), "deployments/engine.ear"), "root.war")
The suggested changes to fix this bug are the following: http://gerrit.ovirt.org/6483 - Move servlet utilities to utils module http://gerrit.ovirt.org/6484 - Add BLOB servlet http://gerrit.ovirt.org/6485 - Use BLOB servlet to serve .iso files With these three changes the engine can serve the node (or RHEV-H) .iso files without copying them to the root web application (without copying them at all in fact), but there are still issues to make this work upstream. First issue is that I needed to update some options in the database as they use "rhevh" instead of "ovirt-node": update vdc_options set option_value = 'ovirt-node-iso' where option_name = 'OvirtIsoPrefix' update vdc_options set option_value = '2.5.0' where option_name = 'OvirtInitialSupportedIsoVersion' update vdc_options set option_value = '/usr/share/ovirt-node-iso' where option_name = 'oVirtISOsRepositoryPath' Second issue is that the logic in the engine that decides if the version of the .iso is suitable for the operating system doesn't work because the version of the .iso is 2.5.x and the version of the operating system is 17.0.x. Any suggestion on how to fix these two issues? Should we fix them in this bug?
Juan, I think that if you need to fix these issues in order to make it work upstream, then yes - it should be a part of your patches. About the database - why not provide just an upgrade script? About 2nd issue - maybe we should hold some sort of mapping between ISO version and matching OS? Doron, can you provide more information on the 2nd issue?
See bug 732096 for the specifics. The general idea is that the file name has the relevant release and version, which are also stored in a matching txt file. The backend should only do a basic sanity of "can't upgrade rhel 5 to 6", etc.
This additional change fixes the database options and the version number comparison: http://gerrit.ovirt.org/6555
With the changes suggested the updated .iso is downloaded to the node correctly, but it still fails to install due to an issue with the vdsm-upgrade script. See the details in bug 842775.
Fix[1] is available, files are no longer copied before transmit. [1] http://gerrit.ovirt.org/#/c/6722/ new implementation for apache-sshd usage
*** Bug 822925 has been marked as a duplicate of this bug. ***
commit 8d7d8ecb07cca8c47ae539115b581a7124923235 Author: Alon Bar-Lev <alonbl> Date: Wed Jul 25 16:21:07 2012 +0300 bootstrap: new implementation for apache-sshd usage Major changes: 1. Do not use temporary files for compression/decompression. 2. Do not use wget to pull large files, use ssh for all transfers. 3. One pass on files for digest, compress/decompress, send/receive. 4. Do not pull every 1 second for bytes/status. 5. Test for command status code. 6. File transfer using ssh and md5sum at same session, md5sum written to stderr. 7. Limit buffer size when reading remote output, so we won't exhaust all memory. 8. Do not echo back whole file content when sending file. 9. Consistent error, exception handling and debugging information. 10. More unit tests. Split between pure ssh implementation[1] and application logic[2]. Unit tests now have their own dedicated generic sshd[3], for proper work in embedded mode ssh apache-ssh-0.7.0 is required. Separate unit tests dedicated to OVirtSSH implementation, by default embedded apache-sshd is used, this can be overridden by setting java system properties, see[4]. As unit tests takes long time, use -Penable-ssh-tests to activate. [1] org.ovirt.engine.core.utils.ssh.OVirtSSH [2] org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper [3] org.ovirt.engine.core.utils.ssh.SSHD [4] org.ovirt.engine.core.utils.ssh.TestCommon Change-Id: I50ba60f2db364114907485da3074feb714615e0c Signed-off-by: Alon Bar-Lev <alonbl> http://gerrit.ovirt.org/#/c/6722/