Red Hat Bugzilla – Bug 840356
[ovirt-engine-backend] rhevh upgrade fails on java.io.IOException (Failed to create directory /null/deploy/ROOT.war)
Last modified: 2016-02-10 14:27:11 EST
Description of problem:
rhevh upgrade fails on java.io.IOException.
2012-07-16 09:23:08,868 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-44) [5b5b5b66] Installation of white-vdsf.eng.lab.tlv.redhat.com. Received message:
Preparing ISO file. FYI. (Stage: Upload upgrade ISO to oVirt Node)
2012-07-16 09:23:08,868 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-44) [5b5b5b66] VDS message: Preparing ISO file
2012-07-16 09:23:08,871 ERROR [org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper] (pool-4-thread-44) uploadLargeFile: Unable to copy local file: : java.io.IOExce
ption: Failed to create directory /null/deploy/ROOT.war
at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.fastFileCopy(MinaInstallWrapper.java:1009) [engine-utils.jar:]
at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.uploadLargeFile(MinaInstallWrapper.java:546) [engine-utils.jar:]
at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.UploadFile(MinaInstallWrapper.java:411) [engine-utils.jar:]
at org.ovirt.engine.core.bll.OVirtInstaller.RunStage(OVirtInstaller.java:48) [engine-bll.jar:]
at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:234) [engine-bll.jar:]
at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.ExecuteWithoutTransaction(CommandBase.java:805) [engine-bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:897) [engine-bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1204) [engine-bll.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:]
at org.ovirt.engine.core.bll.CommandBase.Execute(CommandBase.java:912) [engine-bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.ExecuteAction(CommandBase.java:269) [engine-bll.jar:]
at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:]
at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:]
at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:]
at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_03-icedtea]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_03-icedtea]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_03-icedtea]
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Try to upgrade rhevh from the UI
Created attachment 598375 [details]
From MinaInstallWrapper.java -
callbackAddMessage("Preparing ISO file");
copyDest = Path.Combine(
Path.Combine(Path.Combine(System.getProperty("jboss.server.home.dir"), "deploy"), "ROOT.war"),
For some reason - jboss.server.home.dir is null.
Juan, do you have any insights?
This hasn't been touched since JBoss 5 times. Let me take a look.
If we need to solve this quickly we need to replace with the following:
Path.Combine(Path.Combine(System.getProperty("jboss.server.base.dir"), "deployments/engine.ear"), "root.war")
But this needs a deeper change, as writing files to the /usr directory is not a good practice.
Sorry, the previous comment was wrong. The correct way to do this is:
Path.Combine(Path.Combine(System.getProperty("jboss.server.data.dir"), "deployments/engine.ear"), "root.war")
The suggested changes to fix this bug are the following:
http://gerrit.ovirt.org/6483 - Move servlet utilities to utils module
http://gerrit.ovirt.org/6484 - Add BLOB servlet
http://gerrit.ovirt.org/6485 - Use BLOB servlet to serve .iso files
With these three changes the engine can serve the node (or RHEV-H) .iso files without copying them to the root web application (without copying them at all in fact), but there are still issues to make this work upstream.
First issue is that I needed to update some options in the database as they use "rhevh" instead of "ovirt-node":
update vdc_options set option_value = 'ovirt-node-iso' where option_name = 'OvirtIsoPrefix'
update vdc_options set option_value = '2.5.0' where option_name = 'OvirtInitialSupportedIsoVersion'
update vdc_options set option_value = '/usr/share/ovirt-node-iso' where option_name = 'oVirtISOsRepositoryPath'
Second issue is that the logic in the engine that decides if the version of the .iso is suitable for the operating system doesn't work because the version of the .iso is 2.5.x and the version of the operating system is 17.0.x.
Any suggestion on how to fix these two issues? Should we fix them in this bug?
I think that if you need to fix these issues in order to make it work upstream, then yes - it should be a part of your patches.
About the database - why not provide just an upgrade script?
About 2nd issue - maybe we should hold some sort of mapping between ISO version and matching OS?
Doron, can you provide more information on the 2nd issue?
See bug 732096 for the specifics.
The general idea is that the file name has the relevant release and version,
which are also stored in a matching txt file. The backend should only do a basic
sanity of "can't upgrade rhel 5 to 6", etc.
This additional change fixes the database options and the version number comparison:
With the changes suggested the updated .iso is downloaded to the node correctly, but it still fails to install due to an issue with the vdsm-upgrade script. See the details in bug 842775.
Fix is available, files are no longer copied before transmit.
 http://gerrit.ovirt.org/#/c/6722/ new implementation for apache-sshd usage
*** Bug 822925 has been marked as a duplicate of this bug. ***
Author: Alon Bar-Lev <email@example.com>
Date: Wed Jul 25 16:21:07 2012 +0300
bootstrap: new implementation for apache-sshd usage
1. Do not use temporary files for compression/decompression.
2. Do not use wget to pull large files, use ssh for all transfers.
3. One pass on files for digest, compress/decompress, send/receive.
4. Do not pull every 1 second for bytes/status.
5. Test for command status code.
6. File transfer using ssh and md5sum at same session, md5sum written to
7. Limit buffer size when reading remote output, so we won't exhaust all
8. Do not echo back whole file content when sending file.
9. Consistent error, exception handling and debugging information.
10. More unit tests.
Split between pure ssh implementation and application logic.
Unit tests now have their own dedicated generic sshd, for proper
work in embedded mode ssh apache-ssh-0.7.0 is required.
Separate unit tests dedicated to OVirtSSH implementation, by default
embedded apache-sshd is used, this can be overridden by setting
java system properties, see.
As unit tests takes long time, use -Penable-ssh-tests to activate.
Signed-off-by: Alon Bar-Lev <firstname.lastname@example.org>