Bug 840356 - [ovirt-engine-backend] rhevh upgrade fails on java.io.IOException (Failed to create directory /null/deploy/ROOT.war)
Summary: [ovirt-engine-backend] rhevh upgrade fails on java.io.IOException (Failed to ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.1.0
Hardware: x86_64
OS: Linux
urgent
unspecified
Target Milestone: ---
: ---
Assignee: Alon Bar-Lev
QA Contact: Pavel Stehlik
URL:
Whiteboard: infra
: 822925 (view as bug list)
Depends On:
Blocks: 728154
TreeView+ depends on / blocked
 
Reported: 2012-07-16 06:29 UTC by Kiril Nesenko
Modified: 2016-02-10 19:27 UTC (History)
16 users (show)

Fixed In Version: si16
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-12-04 19:58:28 UTC
oVirt Team: Infra
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
logs (221.82 KB, application/x-gzip)
2012-07-16 06:29 UTC, Kiril Nesenko
no flags Details

Description Kiril Nesenko 2012-07-16 06:29:09 UTC
Description of problem:
rhevh upgrade fails on java.io.IOException. 

2012-07-16 09:23:08,868 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-44) [5b5b5b66] Installation of white-vdsf.eng.lab.tlv.redhat.com. Received message: 
Preparing ISO file. FYI. (Stage: Upload upgrade ISO to oVirt Node)
2012-07-16 09:23:08,868 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-44) [5b5b5b66] VDS message: Preparing ISO file
2012-07-16 09:23:08,871 ERROR [org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper] (pool-4-thread-44) uploadLargeFile: Unable to copy local file: : java.io.IOExce
ption: Failed to create directory /null/deploy/ROOT.war
        at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.fastFileCopy(MinaInstallWrapper.java:1009) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.uploadLargeFile(MinaInstallWrapper.java:546) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper.UploadFile(MinaInstallWrapper.java:411) [engine-utils.jar:]
        at org.ovirt.engine.core.bll.OVirtInstaller.RunStage(OVirtInstaller.java:48) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:234) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.ExecuteWithoutTransaction(CommandBase.java:805) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:897) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1204) [engine-bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.Execute(CommandBase.java:912) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.ExecuteAction(CommandBase.java:269) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:]
        at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_03-icedtea]
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_03-icedtea]
        at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_03-icedtea]



Version-Release number of selected component (if applicable):
si10

How reproducible:
Always

Steps to Reproduce:
1. Try to upgrade rhevh from the UI 
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Kiril Nesenko 2012-07-16 06:29:38 UTC
Created attachment 598375 [details]
logs

Comment 2 Yair Zaslavsky 2012-07-19 11:58:56 UTC
From MinaInstallWrapper.java - 

  try {
            callbackAddMessage("Preparing ISO file");
            copyDest = Path.Combine(
                    Path.Combine(Path.Combine(System.getProperty("jboss.server.home.dir"), "deploy"), "ROOT.war"),

For some reason - jboss.server.home.dir is null.

Juan, do you have any insights?

Comment 3 Juan Hernández 2012-07-19 12:26:35 UTC
This hasn't been touched since JBoss 5 times. Let me take a look.

Comment 4 Juan Hernández 2012-07-19 12:49:02 UTC
If we need to solve this quickly we need to replace with the following:

Path.Combine(Path.Combine(System.getProperty("jboss.server.base.dir"), "deployments/engine.ear"), "root.war")

But this needs a deeper change, as writing files to the /usr directory is not a good practice.

Comment 5 Juan Hernández 2012-07-19 12:52:29 UTC
Sorry, the previous comment was wrong. The correct way to do this is:

Path.Combine(Path.Combine(System.getProperty("jboss.server.data.dir"), "deployments/engine.ear"), "root.war")

Comment 6 Juan Hernández 2012-07-20 14:35:35 UTC
The suggested changes to fix this bug are the following:

http://gerrit.ovirt.org/6483 - Move servlet utilities to utils module
http://gerrit.ovirt.org/6484 - Add BLOB servlet
http://gerrit.ovirt.org/6485 - Use BLOB servlet to serve .iso files

With these three changes the engine can serve the node (or RHEV-H) .iso files without copying them to the root web application (without copying them at all in fact), but there are still issues to make this work upstream.

First issue is that I needed to update some options in the database as they use "rhevh" instead of "ovirt-node":

update vdc_options set option_value = 'ovirt-node-iso' where option_name = 'OvirtIsoPrefix'
update vdc_options set option_value = '2.5.0' where option_name = 'OvirtInitialSupportedIsoVersion'
update vdc_options set option_value = '/usr/share/ovirt-node-iso' where option_name = 'oVirtISOsRepositoryPath'

Second issue is that the logic in the engine that decides if the version of the .iso is suitable for the operating system doesn't work because the version of the .iso is 2.5.x and the version of the operating system is 17.0.x.

Any suggestion on how to fix these two issues? Should we fix them in this bug?

Comment 7 Yair Zaslavsky 2012-07-22 07:05:15 UTC
Juan,
I think that if you need to fix these issues in order to make it work upstream, then yes - it should be a part of your patches.

About the database - why not provide just an upgrade script?


About 2nd issue - maybe we should hold some sort of mapping between ISO version and matching OS?


Doron, can you provide more information on the 2nd issue?

Comment 8 Doron Fediuck 2012-07-22 13:16:24 UTC
See bug 732096 for the specifics.
The general idea is that the file name has the relevant release and version,
which are also stored in a matching txt file. The backend should only do a basic
sanity of "can't upgrade rhel 5 to 6", etc.

Comment 11 Juan Hernández 2012-07-24 14:05:11 UTC
This additional change fixes the database options and the version number comparison:

http://gerrit.ovirt.org/6555

Comment 12 Juan Hernández 2012-07-24 15:08:56 UTC
With the changes suggested the updated .iso is downloaded to the node correctly, but it still fails to install due to an issue with the vdsm-upgrade script. See the details in bug 842775.

Comment 13 Alon Bar-Lev 2012-08-06 20:47:38 UTC
Fix[1] is available, files are no longer copied before transmit.

[1] http://gerrit.ovirt.org/#/c/6722/ new implementation for apache-sshd usage

Comment 14 Douglas Schilling Landgraf 2012-08-16 14:06:10 UTC
*** Bug 822925 has been marked as a duplicate of this bug. ***

Comment 16 Alon Bar-Lev 2012-08-22 08:28:33 UTC
commit 8d7d8ecb07cca8c47ae539115b581a7124923235
Author: Alon Bar-Lev <alonbl>
Date:   Wed Jul 25 16:21:07 2012 +0300

    bootstrap: new implementation for apache-sshd usage
    
    Major changes:
    
      1. Do not use temporary files for compression/decompression.
    
      2. Do not use wget to pull large files, use ssh for all transfers.
    
      3. One pass on files for digest, compress/decompress, send/receive.
    
      4. Do not pull every 1 second for bytes/status.
    
      5. Test for command status code.
    
      6. File transfer using ssh and md5sum at same session, md5sum written to
         stderr.
    
      7. Limit buffer size when reading remote output, so we won't exhaust all
         memory.
    
      8. Do not echo back whole file content when sending file.
    
      9. Consistent error, exception handling and debugging information.
    
     10. More unit tests.
    
    Split between pure ssh implementation[1] and application logic[2].
    
    Unit tests now have their own dedicated generic sshd[3], for proper
    work in embedded mode ssh apache-ssh-0.7.0 is required.
    
    Separate unit tests dedicated to OVirtSSH implementation, by default
    embedded apache-sshd is used, this can be overridden by setting
    java system properties, see[4].
    
    As unit tests takes long time, use -Penable-ssh-tests to activate.
    
    [1] org.ovirt.engine.core.utils.ssh.OVirtSSH
    [2] org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper
    [3] org.ovirt.engine.core.utils.ssh.SSHD
    [4] org.ovirt.engine.core.utils.ssh.TestCommon
    
    Change-Id: I50ba60f2db364114907485da3074feb714615e0c
    Signed-off-by: Alon Bar-Lev <alonbl>

http://gerrit.ovirt.org/#/c/6722/


Note You need to log in before you can comment on or make changes to this bug.