Bug 840356 - [ovirt-engine-backend] rhevh upgrade fails on java.io.IOException (Failed to create directory /null/deploy/ROOT.war)
[ovirt-engine-backend] rhevh upgrade fails on java.io.IOException (Failed to ...
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
x86_64 Linux
urgent Severity unspecified
: ---
: ---
Assigned To: Alon Bar-Lev
Pavel Stehlik
: 822925 (view as bug list)
Depends On:
Blocks: 728154
  Show dependency treegraph
Reported: 2012-07-16 02:29 EDT by Kiril Nesenko
Modified: 2016-02-10 14:27 EST (History)
16 users (show)

See Also:
Fixed In Version: si16
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2012-12-04 14:58:28 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
logs (221.82 KB, application/x-gzip)
2012-07-16 02:29 EDT, Kiril Nesenko
no flags Details

  None (edit)
Description Kiril Nesenko 2012-07-16 02:29:09 EDT
Description of problem:
rhevh upgrade fails on java.io.IOException. 

2012-07-16 09:23:08,868 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-44) [5b5b5b66] Installation of white-vdsf.eng.lab.tlv.redhat.com. Received message: 
Preparing ISO file. FYI. (Stage: Upload upgrade ISO to oVirt Node)
2012-07-16 09:23:08,868 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-44) [5b5b5b66] VDS message: Preparing ISO file
2012-07-16 09:23:08,871 ERROR [org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper] (pool-4-thread-44) uploadLargeFile: Unable to copy local file: : java.io.IOExce
ption: Failed to create directory /null/deploy/ROOT.war
        at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.fastFileCopy(MinaInstallWrapper.java:1009) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.uploadLargeFile(MinaInstallWrapper.java:546) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.UploadFile(MinaInstallWrapper.java:411) [engine-utils.jar:]
        at org.ovirt.engine.core.bll.OVirtInstaller.RunStage(OVirtInstaller.java:48) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:234) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.ExecuteWithoutTransaction(CommandBase.java:805) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:897) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1204) [engine-bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.Execute(CommandBase.java:912) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.ExecuteAction(CommandBase.java:269) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:]
        at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_03-icedtea]
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_03-icedtea]
        at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_03-icedtea]

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Try to upgrade rhevh from the UI 
Actual results:

Expected results:

Additional info:
Comment 1 Kiril Nesenko 2012-07-16 02:29:38 EDT
Created attachment 598375 [details]
Comment 2 Yair Zaslavsky 2012-07-19 07:58:56 EDT
From MinaInstallWrapper.java - 

  try {
            callbackAddMessage("Preparing ISO file");
            copyDest = Path.Combine(
                    Path.Combine(Path.Combine(System.getProperty("jboss.server.home.dir"), "deploy"), "ROOT.war"),

For some reason - jboss.server.home.dir is null.

Juan, do you have any insights?
Comment 3 Juan Hernández 2012-07-19 08:26:35 EDT
This hasn't been touched since JBoss 5 times. Let me take a look.
Comment 4 Juan Hernández 2012-07-19 08:49:02 EDT
If we need to solve this quickly we need to replace with the following:

Path.Combine(Path.Combine(System.getProperty("jboss.server.base.dir"), "deployments/engine.ear"), "root.war")

But this needs a deeper change, as writing files to the /usr directory is not a good practice.
Comment 5 Juan Hernández 2012-07-19 08:52:29 EDT
Sorry, the previous comment was wrong. The correct way to do this is:

Path.Combine(Path.Combine(System.getProperty("jboss.server.data.dir"), "deployments/engine.ear"), "root.war")
Comment 6 Juan Hernández 2012-07-20 10:35:35 EDT
The suggested changes to fix this bug are the following:

http://gerrit.ovirt.org/6483 - Move servlet utilities to utils module
http://gerrit.ovirt.org/6484 - Add BLOB servlet
http://gerrit.ovirt.org/6485 - Use BLOB servlet to serve .iso files

With these three changes the engine can serve the node (or RHEV-H) .iso files without copying them to the root web application (without copying them at all in fact), but there are still issues to make this work upstream.

First issue is that I needed to update some options in the database as they use "rhevh" instead of "ovirt-node":

update vdc_options set option_value = 'ovirt-node-iso' where option_name = 'OvirtIsoPrefix'
update vdc_options set option_value = '2.5.0' where option_name = 'OvirtInitialSupportedIsoVersion'
update vdc_options set option_value = '/usr/share/ovirt-node-iso' where option_name = 'oVirtISOsRepositoryPath'

Second issue is that the logic in the engine that decides if the version of the .iso is suitable for the operating system doesn't work because the version of the .iso is 2.5.x and the version of the operating system is 17.0.x.

Any suggestion on how to fix these two issues? Should we fix them in this bug?
Comment 7 Yair Zaslavsky 2012-07-22 03:05:15 EDT
I think that if you need to fix these issues in order to make it work upstream, then yes - it should be a part of your patches.

About the database - why not provide just an upgrade script?

About 2nd issue - maybe we should hold some sort of mapping between ISO version and matching OS?

Doron, can you provide more information on the 2nd issue?
Comment 8 Doron Fediuck 2012-07-22 09:16:24 EDT
See bug 732096 for the specifics.
The general idea is that the file name has the relevant release and version,
which are also stored in a matching txt file. The backend should only do a basic
sanity of "can't upgrade rhel 5 to 6", etc.
Comment 11 Juan Hernández 2012-07-24 10:05:11 EDT
This additional change fixes the database options and the version number comparison:

Comment 12 Juan Hernández 2012-07-24 11:08:56 EDT
With the changes suggested the updated .iso is downloaded to the node correctly, but it still fails to install due to an issue with the vdsm-upgrade script. See the details in bug 842775.
Comment 13 Alon Bar-Lev 2012-08-06 16:47:38 EDT
Fix[1] is available, files are no longer copied before transmit.

[1] http://gerrit.ovirt.org/#/c/6722/ new implementation for apache-sshd usage
Comment 14 Douglas Schilling Landgraf 2012-08-16 10:06:10 EDT
*** Bug 822925 has been marked as a duplicate of this bug. ***
Comment 16 Alon Bar-Lev 2012-08-22 04:28:33 EDT
commit 8d7d8ecb07cca8c47ae539115b581a7124923235
Author: Alon Bar-Lev <alonbl@redhat.com>
Date:   Wed Jul 25 16:21:07 2012 +0300

    bootstrap: new implementation for apache-sshd usage
    Major changes:
      1. Do not use temporary files for compression/decompression.
      2. Do not use wget to pull large files, use ssh for all transfers.
      3. One pass on files for digest, compress/decompress, send/receive.
      4. Do not pull every 1 second for bytes/status.
      5. Test for command status code.
      6. File transfer using ssh and md5sum at same session, md5sum written to
      7. Limit buffer size when reading remote output, so we won't exhaust all
      8. Do not echo back whole file content when sending file.
      9. Consistent error, exception handling and debugging information.
     10. More unit tests.
    Split between pure ssh implementation[1] and application logic[2].
    Unit tests now have their own dedicated generic sshd[3], for proper
    work in embedded mode ssh apache-ssh-0.7.0 is required.
    Separate unit tests dedicated to OVirtSSH implementation, by default
    embedded apache-sshd is used, this can be overridden by setting
    java system properties, see[4].
    As unit tests takes long time, use -Penable-ssh-tests to activate.
    [1] org.ovirt.engine.core.utils.ssh.OVirtSSH
    [2] org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper
    [3] org.ovirt.engine.core.utils.ssh.SSHD
    [4] org.ovirt.engine.core.utils.ssh.TestCommon
    Change-Id: I50ba60f2db364114907485da3074feb714615e0c
    Signed-off-by: Alon Bar-Lev <alonbl@redhat.com>


Note You need to log in before you can comment on or make changes to this bug.