Bug 1436749 - [3.6 ONLY] Host upgrade fails, "Step: ovirt-node-upgrade; Details: RuntimeError: Bootloader Installation Failed" [NEEDINFO]
Summary: [3.6 ONLY] Host upgrade fails, "Step: ovirt-node-upgrade; Details: RuntimeErr...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-host-deploy
Version: 3.6.10
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ovirt-3.6.11
: ---
Assignee: Douglas Schilling Landgraf
QA Contact: Pavel Stehlik
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-28 14:36 UTC by Robert McSwain
Modified: 2020-05-14 15:50 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-20 19:44:21 UTC
oVirt Team: Node
Target Upstream Version:
Embargoed:
dougsland: needinfo? (rmcswain)

Attachments (Terms of Use)

Description Robert McSwain 2017-03-28 14:36:00 UTC 
Description of problem:
Host fails to upgrade when initiated from RHEV-M, citing 
"Step: ovirt-node-upgrade; Details: RuntimeError: Bootloader Installation Failed"

Version-Release number of selected component (if applicable):
RHEV-M 3.6.10 
rhevh-7.3-20170118.0.el6ev.iso

How reproducible:
100% in this instance

Steps to Reproduce:
Initiate an upgrade from the manager to rhevh-7.3-20170118.0.el6ev.iso, and it fails.

Additional info:

the audit table in the database shows:

 2017-03-23 15:20:40.409-04    | 6ffd31b0       | Host [hostname] upgrade was started (User: admin@internal).
 2017-03-23 15:20:41.449-04    | 77f04cdc       | Installing Host [hostname]. Connected to host 10.35.34.28 with SSH key fingerprint: SHA256:PfJH0+Qw4cjPtxsMUZ3SXZ27X1y2R/G5+KWX/c/SJ8Y.
 2017-03-23 15:20:41.521-04    | 77f04cdc       | Installing Host [hostname]. Sending file /usr/share/rhev-hypervisor/rhevh-7.3-20170118.0.el6ev.iso to /data/updates/ovirt-node-image.iso.
 2017-03-23 15:21:11.814-04    | 77f04cdc       | Installing Host [hostname]. Executing /usr/share/vdsm-reg/vdsm-upgrade.
 2017-03-23 15:21:11.973-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Temporary Directory is: /data/tmpUS0pUx
 2017-03-23 15:21:11.997-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.LockFile: INFO     Acquiring Lock
 2017-03-23 15:21:12.019-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Checking iso md5
 2017-03-23 15:21:13.745-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Running pre-upgrade hooks
 2017-03-23 15:21:13.77-04     | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Running: 01-vdsm
 2017-03-23 15:21:26.832-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Running: 03-persist-multipath-wwids
 2017-03-23 15:21:26.855-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     pre-upgrade hooks completed
 2017-03-23 15:21:26.877-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Running pre-upgrade hooks
 2017-03-23 15:21:26.902-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Running: 01-vdsm
 2017-03-23 15:21:30.389-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Running: 03-persist-multipath-wwids
 2017-03-23 15:21:30.437-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     pre-upgrade hooks completed
 2017-03-23 15:21:30.458-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     hooks: False
 2017-03-23 15:21:33.57-04     | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Installing Bootloader
 2017-03-23 15:21:35.973-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: ERROR    Error: Upgrade Failed: Bootloader Installation Failed
 2017-03-23 15:21:35.996-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: Traceback (most recent call last):
 2017-03-23 15:21:36.094-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: RuntimeError: Bootloader Installation Failed
 2017-03-23 15:21:36.113-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: Error: Upgrade Failed: Bootloader Installation Failed
 2017-03-23 15:21:36.136-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: Traceback (most recent call last):
 2017-03-23 15:21:36.233-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: RuntimeError: Bootloader Installation Failed
 2017-03-23 15:21:36.252-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Running rollback hooks
 2017-03-23 15:21:36.271-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     rollback hooks completed
 2017-03-23 15:21:36.289-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Running rollback hooks
 2017-03-23 15:21:36.308-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     rollback hooks completed
 2017-03-23 15:21:36.327-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Upgrade Failed, Rollback Completed
 2017-03-23 15:21:36.345-04    | 77f04cdc       | Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Cleaning up temporary directory
 2017-03-23 15:21:36.364-04    | 77f04cdc       | Failed to install Host [hostname]. Step: ovirt-node-upgrade; Details: Upgraded Failed .
 2017-03-23 15:21:36.383-04    | 77f04cdc       | Failed to install Host [hostname]. Step: RHEV_INSTALL.
 2017-03-23 15:21:36.427-04    | 77f04cdc       | Host [hostname] installation failed. Upgrade failed, please refer to logs for further information.


The engine shows the following about this

(15:58) $ less /var/log/ovirt-engine/engine.log|egrep 77f04cdc         
...
2017-03-23 14:21:33,589 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO     Installing Bootloader
2017-03-23 14:21:35,970 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: ERROR    Error: Upgrade Failed: Bootloader Installation Failed&#10;"/>
2017-03-23 14:21:35,972 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: ERROR    Error: Upgrade Failed: Bootloader Installation Failed
2017-03-23 14:21:35,992 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: ERROR    Error: Upgrade Failed: Bootloader Installation Failed
2017-03-23 14:21:35,992 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message="Traceback (most recent call last):&#10;"/>
2017-03-23 14:21:35,993 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details: Traceback (most recent call last):
2017-03-23 14:21:36,015 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details: Traceback (most recent call last):
2017-03-23 14:21:36,015 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message='  File "/usr/sbin/ovirt-node-upgrade", line 371, in run&#10;'/>
2017-03-23 14:21:36,016 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details:   File "/usr/sbin/ovirt-node-upgrade", line 371, in run
2017-03-23 14:21:36,035 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details:   File "/usr/sbin/ovirt-node-upgrade", line 371, in run
2017-03-23 14:21:36,035 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message="    self._run_upgrade()&#10;"/>
2017-03-23 14:21:36,036 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details:     self._run_upgrade()
2017-03-23 14:21:36,054 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details:     self._run_upgrade()
2017-03-23 14:21:36,054 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message='  File "/usr/sbin/ovirt-node-upgrade", line 256, in _run_upgrade&#10;'/>
2017-03-23 14:21:36,055 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details:   File "/usr/sbin/ovirt-node-upgrade", line 256, in _run_upgrade
2017-03-23 14:21:36,073 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details:   File "/usr/sbin/ovirt-node-upgrade", line 256, in _run_upgrade
2017-03-23 14:21:36,073 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message='    raise RuntimeError("Bootloader Installation Failed")&#10;'/>
2017-03-23 14:21:36,074 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details:     raise RuntimeError("Bootloader Installation Failed")
2017-03-23 14:21:36,093 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details:     raise RuntimeError("Bootloader Installation Failed")
2017-03-23 14:21:36,093 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message="RuntimeError: Bootloader Installation Failed&#10;"/>
2017-03-23 14:21:36,094 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details: RuntimeError: Bootloader Installation Failed
2017-03-23 14:21:36,112 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details: RuntimeError: Bootloader Installation Failed
2017-03-23 14:21:36,112 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message="Error: Upgrade Failed: Bootloader Installation Failed&#10;"/>
2017-03-23 14:21:36,113 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details: Error: Upgrade Failed: Bootloader Installation Failed
2017-03-23 14:21:36,132 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details: Error: Upgrade Failed: Bootloader Installation Failed
2017-03-23 14:21:36,132 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message="Traceback (most recent call last):&#10;"/>
2017-03-23 14:21:36,133 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details: Traceback (most recent call last):
2017-03-23 14:21:36,133 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-5-thread-1) [77f04cdc] SSH error running command root.34.28:'/usr/share/vdsm-reg/vdsm-upgrade': Command returned failure code 1 during SSH session 'root.34.28'
2017-03-23 14:21:36,133 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-5-thread-1) [77f04cdc] Exception: java.io.IOException: Command returned failure code 1 during SSH session 'root.34.28'
2017-03-23 14:21:36,153 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details: Traceback (most recent call last):
2017-03-23 14:21:36,153 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message='  File "/usr/sbin/ovirt-node-upgrade", line 371, in run&#10;'/>
2017-03-23 14:21:36,154 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details:   File "/usr/sbin/ovirt-node-upgrade", line 371, in run
2017-03-23 14:21:36,172 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details:   File "/usr/sbin/ovirt-node-upgrade", line 371, in run
2017-03-23 14:21:36,173 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message="    self._run_upgrade()&#10;"/>
2017-03-23 14:21:36,174 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details:     self._run_upgrade()
2017-03-23 14:21:36,192 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details:     self._run_upgrade()
2017-03-23 14:21:36,192 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status="OK" message='  File "/usr/sbin/ovirt-node-upgrade", line 256, in _run_upgrade&#10;'/>
2017-03-23 14:21:36,193 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallerMessages] (OVirtNodeUpgrade) [77f04cdc] Installation '10.35.34.28': Step: ovirt-node-upgrade; Details:   File "/usr/sbin/ovirt-node-upgrade", line 256, in _run_upgrade
2017-03-23 14:21:36,213 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (OVirtNodeUpgrade) [77f04cdc] Correlation ID: 77f04cdc, Call Stack: null, Custom Event ID: -1, Message: Installing Host [hostname]. Step: ovirt-node-upgrade; Details:   File "/usr/sbin/ovirt-node-upgrade", line 256, in _run_upgrade
2017-03-23 14:21:36,213 INFO  [org.ovirt.engine.core.bll.hostdeploy.OVirtNodeUpgrade] (OVirtNodeUpgrade) [77f04cdc] update from host '10.35.34.28': <BSTRAP component="ovirt-node-upgrade" status

I found this bug https://bugzilla.redhat.com/show_bug.cgi?id=1260551 which has the string "Error: Upgrade Failed: Bootloader Installation Failed"

however your version is shown to be 7.3 in the logs from

"Sending file /usr/share/rhev-hypervisor/rhevh-7.3-20170118.0.el6ev.iso to /data/updates/ovirt-node-image.iso."

I also found https://bugzilla.redhat.com/show_bug.cgi?id=1192786 which shows an issue with UEFI, however this instance is using legacy BIOS mode. 

Additionally, the install works when from ISO directly to the blade, but always fails when the upgrade is initiated from the manager.
Comment 2 Sandro Bonazzola 2017-03-31 10:59:34 UTC 
Douglas, can you please investigate?
Note You need to log in before you can comment on or make changes to this bug.