Description of problem: Situation: vdsm-upgrade of RHEV 3.5, from rhev-hypervisor7-7.0-20141218.0 to rhev-hypervisor7-7.0-20150106.0, failed, because of the engine fqdn was written in /etc/default/ovirt, in double quotes. It is not clear where the double quotes came from. It might be that it was written in quotes in the RHEV TUI in quotes, and then additional quotes were added. Problem: Though the upgrade failure, at the end of the printout, there was this message: "Upgraded Succeeded. Rebooting" There was niether reboot afterwards, nor the rhev-h was upgraded. * The problem was solved by editing the /etc/default/ovirt from double quotes into single one. Console log: [root@coda-vdsb ~]# /usr/share/vdsm-reg/vdsm-upgrade <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Temporary Directory is: /data/tmpUHEMW_ "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.LockFile: INFO Acquiring Lock "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Running pre-upgrade hooks "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Running: 01-vdsm "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO pre-upgrade hooks completed "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Running pre-upgrade hooks "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Running: 01-vdsm "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO pre-upgrade hooks completed "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO hooks: False "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Installing Bootloader "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="/etc/default/ovirt: line 1: https://dhcp160-207.scl.lab.tlv.redhat.com:443: No such file or directory "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="/etc/default/ovirt: line 1: https://dhcp160-207.scl.lab.tlv.redhat.com:443: No such file or directory "/> <BSTRAP component="ovirt-node-upgrade" status="OKtatus="OK" message="ovirt-node-upgrade.LockFile: INFO Acquiring Lock "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Running pre-upgrade hooks "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Running: 01-vdsm "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO pre-upgrade hooks completed "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Running pre-upgrade hooks "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Running: 01-vdsm "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO pre-upgrade hooks completed "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO hooks: False "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Installing Bootloader "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="/etc/default/ovirt: line 1: https://dhcp160-207.scl.lab.tlv.redhat.com:443: No such file or directory "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="/etc/default/ovirt: line 1: https://dhcp160-207.scl.lab.tlv.redhat.com:443: No such file or directory "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Installing Bootloader "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: ERROR Error: Upgrade Failed: Unable to save to file! "/> <BSTRAP component="ovirt-node-upgrade" status="" mesmessage="ovirt-node-upgrade.UpgradeTool: INFO Installing Bootloader "/> <BSTRAP component="ovirt-node-upgrade" status="OK" smessage="ovirt-node-upgrade.UpgradeTool: INFO Installing Bootloader "/> <BSTRAP component="ovirt-node-upgrade" status="OK" amessage="ovirt-node-upgrade.UpgradeTool: INFO Installing Bootloader "/> <BSTRAP component="ovirt-node-upgrade" status="OK" gmessage="ovirt-node-upgrade.UpgradeTool: INFO Installing Bootloader "/> <BSTRAP component="ovirt-node-upgrade" status="OK" e=message="ovirt-node-upgrade.UpgradeTool: INFO Installing Bootloader "/> <BSTRAP component="ovirt-node-upgrade" status="OK" "ovirt-node-upgrade.UpgradeTool: ERROR Error: Upgrade Failed: Unable to save to file! "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="Traceback (most recent call last): "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=' File "/usr/sbin/ovirt-node-upgrade", line 364, in run '/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=" self._run_upgrade() "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=' File "/usr/sbin/ovirt-node-upgrade", line 255, in _run_upgrade '/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=" if not upgrade.ovirt_boot_setup(): "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=' File "/usr/lib/python2.7/site-packages/ovirtnode/install.py", line 701, in ovirt_boot_setup '/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=' File "/usr/lib/python2.7/site-packages/ovirtnode/ovirtfunctions.py", line 371, in disable_firstboot '/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=' File "/usr/lib/python2.7/site-packages/augeas.py", line 385, in save '/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="IOError: Unable to save to file! "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="Error: Upgrade Failed: Unable to save to file! "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="Traceback (most recent call last): "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=' File "/usr/sbin/ovirt-node-upgrade", line 364, in run '/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=" self._run_upgrade() "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=' File "/usr/sbin/ovirt-node-upgrade", line 255, in _run_upgrade '/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=" if not upgrade.ovirt_boot_setup(): "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=' File "/usr/lib/python2.7/site-packages/ovirtnode/install.py", line 701, in ovirt_boot_setup '/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=' File "/usr/lib/python2.7/site-packages/ovirtnode/ovirtfunctions.py", line 371, in disable_firstboot '/> <BSTRAP component="ovirt-node-upgrade" status="OK" message=' File "/usr/lib/python2.7/site-packages/augeas.py", line 385, in save '/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="IOError: Unable to save to file! "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Running rollback hooks "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO rollback hooks completed "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Running rollback hooks "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO rollback hooks completed "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Upgrade Failed, Rollback Completed "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="ovirt-node-upgrade.UpgradeTool: INFO Cleaning up temporary directory "/> <BSTRAP component="ovirt-node-upgrade" status="OK" message="Upgraded Succeeded. Rebooting"/> <BSTRAP component="RHEV_INSTALL" status="OK" /> [root@coda-vdsb ~]# How reproducible: 100% Expected results: The upgrade printout should reflect the upgrade failure.
The doubles quotes in /etc/default/ovirt are a known bug (bug 1177216) and will be addressed in the RC builds. Could it be that a beta was installed on the host before? If so, then we can close this bug.
(In reply to comment #1): This host was installed from USB with rhev-hypervisor7-7.0-20141218.0, Then was upgraded (by vdsm-upgrade) to rhev-hypervisor7-7.0-20150106.0. Just to clarify, this bug, unlike bug 1177216, is NOT about having the double quotes, but rather on how such problems are handled. The bug is that though we have a problem with the engine fqdn (for example double quotas): A. It is not failing the upgrade. It writes Installed ok, rebooting, though it is not upgraded, and not rebooting. B. The error itself is not repoted correctly: message="/etc/default/ovirt: line 1: https://dhcp160-207.scl.lab.tlv.redhat.com:443: No such file or directory "/> While the real problem is that we have the fqdn in double quotes.
Test vesion: rhev-hypervisor7-7.0-20141218.0 rhev-hypervisor7-7.0-20150106.0 Red Hat Enterprise Virtualization Manager Version: 3.5.0-0.33.el6ev Test steps: 1. Install rhev-hypervisor7-7.0-20141218.0 2. Register to rhevm3.5.0-0.33.el6ev 3. Upgrade to rhev-hypervisor7-7.0-20150106.0 through rhevm 4. Check the events in rhevm. Test results: 1. Upgrade is not succeed. 2. The events in rhevm portal reports upgraded succeed as follows. 2015-Mar-12, 17:32 Host dhcp-11-98.nay.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. 2015-Mar-12, 17:27 Host dhcp-11-98.nay.redhat.com installed 2015-Mar-12, 17:27 Installing Host dhcp-11-98.nay.redhat.com. Step: RHEV_INSTALL. 2015-Mar-12, 17:27 Installing Host dhcp-11-98.nay.redhat.com. Step: ovirt-node-upgrade; Details: Upgraded Succeeded. Rebooting . So virtQE can reproduce this issue. The RHEVM events reports upgrade succeed while it's not actually.
Moved to POST because this is not yet merged in downstream.
As Bug 1270177 block this issue verify on build rhev-hypervisor7-7.2-20150928.0, we'd verify this issue after Bug 1270177 fix.
Test vesion: rhev-hypervisor7-7.0-20141218.0 rhev-hypervisor7-7.2-20151112.1.el7ev ovirt-node-3.6.0-0.20.20151103git3d3779a.el7ev.noarch Red Hat Enterprise Virtualization Manager Version: 3.5.6.2-0.1.el6ev Test steps: 1. Install rhev-hypervisor7-7.0-20141218.0 2. Register to rhevm-3.5.6.2-0.1.el6ev 3. Upgrade to rhev-hypervisor7-7.2-20151112.1.el7ev through rhevm 4. Check the events in rhevm. Test results: 1. After step4, it shows host upgraded succeed as follows. Host dhcp-10-46.nay.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. Installing Host dhcp-10-46.nay.redhat.com. Step: RHEV_INSTALL. Installing Host dhcp-10-46.nay.redhat.com. Step: ovirt-node-upgrade; Details: Upgraded Succeeded. Rebooting . Installing Host dhcp-10-46.nay.redhat.com. Step: ovirt-node-upgrade; Details: ovirt-node-upgrade.UpgradeTool: INFO Cleaning up temporary directory . So this issue is not fixed in ovirt-node-3.6.0-0.20.20151103git3d3779a.el7ev.noarch .
Please attach the logs from the engine and host side.
Created attachment 1099582 [details] Attach engine.log correspond to comment#10
Created attachment 1099583 [details] Attach host log correspond to comment#10
Based on the logs attached, this is a different bug from the original. 2015-11-27 02:05:50,354 - INFO - ovirt-node-upgrade - Temporary Directory is: /data/tmpFZy01u 2015-11-27 02:05:50,355 - DEBUG - ovirt-node-upgrade - {'iso_file': '/data/updates/ovirt-node-image.iso', 'skip_existing_hooks': False, 'reboot': 1} 2015-11-27 02:05:50,355 - INFO - ovirt-node-upgrade - Acquiring Lock 2015-11-27 02:05:50,356 - DEBUG - ovirt-node-upgrade - ('mount', '-o', 'loop', '/data/updates/ovirt-node-image.iso', '/live') 2015-11-27 02:05:50,405 - DEBUG - ovirt-node-upgrade - [] 2015-11-27 02:05:50,405 - DEBUG - ovirt-node-upgrade - mount: /dev/loop3 is write-protected, mounting read-only 2015-11-27 02:05:50,405 - DEBUG - ovirt-node-upgrade - ('mount', '-o', 'loop', '/live/LiveOS/squashfs.img', '/data/tmpFZy01u/squashfs') 2015-11-27 02:05:50,410 - DEBUG - ovirt-node-upgrade - [] 2015-11-27 02:05:50,411 - DEBUG - ovirt-node-upgrade - 2015-11-27 02:05:50,411 - DEBUG - ovirt-node-upgrade - ('mount', '-o', 'loop', '/data/tmpFZy01u/squashfs/LiveOS/ext3fs.img', '/data/tmpFZy01u/rootfs') 2015-11-27 02:05:50,446 - DEBUG - ovirt-node-upgrade - [] 2015-11-27 02:05:50,447 - DEBUG - ovirt-node-upgrade - 2015-11-27 02:05:50,447 - INFO - ovirt-node-upgrade - Running pre-upgrade hooks 2015-11-27 02:05:50,447 - INFO - ovirt-node-upgrade - Running: 01-vdsm 2015-11-27 02:05:50,447 - DEBUG - ovirt-node-upgrade - ('/usr/libexec/ovirt-node/hooks/pre-upgrade/01-vdsm',) 2015-11-27 02:05:58,848 - DEBUG - ovirt-node-upgrade - [u'/usr/libexec/ovirt-node/hooks/pre-upgrade/01-vdsm: Stopping vdsmd to upgrade'] 2015-11-27 02:05:58,848 - DEBUG - ovirt-node-upgrade - 2015-11-27 02:05:58,848 - INFO - ovirt-node-upgrade - pre-upgrade hooks completed 2015-11-27 02:05:58,855 - INFO - ovirt-node-upgrade - Running pre-upgrade hooks 2015-11-27 02:05:58,855 - INFO - ovirt-node-upgrade - Running: 01-vdsm 2015-11-27 02:05:58,855 - DEBUG - ovirt-node-upgrade - ('/data/tmpFZy01u/rootfs/usr/libexec/ovirt-node/hooks/pre-upgrade/01-vdsm',) 2015-11-27 02:05:59,829 - DEBUG - ovirt-node-upgrade - [u'/data/tmpFZy01u/rootfs/usr/libexec/ovirt-node/hooks/pre-upgrade/01-vdsm: Stopping vdsmd to upgrade'] 2015-11-27 02:05:59,829 - DEBUG - ovirt-node-upgrade - 2015-11-27 02:05:59,829 - INFO - ovirt-node-upgrade - Running: 03-persist-multipath-wwids 2015-11-27 02:05:59,829 - DEBUG - ovirt-node-upgrade - ('/data/tmpFZy01u/rootfs/usr/libexec/ovirt-node/hooks/pre-upgrade/03-persist-multipath-wwids',) 2015-11-27 02:06:00,258 - DEBUG - ovirt-node-upgrade - [] 2015-11-27 02:06:00,258 - DEBUG - ovirt-node-upgrade - 2015-11-27 02:06:00,258 - INFO - ovirt-node-upgrade - pre-upgrade hooks completed 2015-11-27 02:06:00,258 - INFO - ovirt-node-upgrade - hooks: False 2015-11-27 02:06:01,411 - DEBUG - ovirt-node-upgrade - <module 'install' from '/data/tmpFZy01u/ovirtnode/install.pyc'> 2015-11-27 02:06:01,421 - INFO - ovirt-node-upgrade - Installing Bootloader 2015-11-27 02:06:52,105 - ERROR - ovirt-node-upgrade - Error: Upgrade Failed: Unable to save to file! Traceback (most recent call last): File "/usr/sbin/ovirt-node-upgrade", line 364, in run self._run_upgrade() File "/usr/sbin/ovirt-node-upgrade", line 255, in _run_upgrade if not upgrade.ovirt_boot_setup(): File "/data/tmpFZy01u/ovirtnode/install.py", line 775, in ovirt_boot_setup _functions.disable_firstboot() File "/data/tmpFZy01u/ovirtnode/ovirtfunctions.py", line 371, in disable_firstboot aug.save() File "/data/tmpFZy01u/rootfs/usr/lib/python2.7/site-packages/augeas.py", line 488, in save raise IOError("Unable to save to file!") IOError: Unable to save to file! 2015-11-27 02:06:52,106 - INFO - ovirt-node-upgrade - Running rollback hooks 2015-11-27 02:06:52,106 - INFO - ovirt-node-upgrade - rollback hooks completed 2015-11-27 02:06:52,106 - INFO - ovirt-node-upgrade - Running rollback hooks 2015-11-27 02:06:52,106 - INFO - ovirt-node-upgrade - rollback hooks completed 2015-11-27 02:06:52,106 - INFO - ovirt-node-upgrade - Upgrade Failed, Rollback Completed 2015-11-27 02:06:52,106 - INFO - ovirt-node-upgrade - Cleaning up temporary directory 2015-11-27 02:06:52,107 - DEBUG - ovirt-node-upgrade - ('/usr/bin/umount', '/data/tmpFZy01u/rootfs') 2015-11-27 02:06:52,115 - DEBUG - ovirt-node-upgrade - [] 2015-11-27 02:06:52,115 - DEBUG - ovirt-node-upgrade - 2015-11-27 02:06:52,115 - DEBUG - ovirt-node-upgrade - ('/usr/bin/umount', '/data/tmpFZy01u/squashfs') 2015-11-27 02:06:52,121 - DEBUG - ovirt-node-upgrade - [] 2015-11-27 02:06:52,122 - DEBUG - ovirt-node-upgrade - 2015-11-27 02:06:52,122 - DEBUG - ovirt-node-upgrade - ('/usr/bin/umount', '/live') 2015-11-27 02:06:52,144 - DEBUG - ovirt-node-upgrade - [] 2015-11-27 02:06:52,144 - DEBUG - ovirt-node-upgrade - "ovirt-node-upgrade.log" 60L, 4836C
(In reply to Douglas Schilling Landgraf from comment #14) > Based on the logs attached, this is a different bug from the original. Answering myself, actually it's the same report as original. The issue is that the testing must not happen from the image rhev-hypervisor7-7.0-20141218.0 to rhev-hypervisor7-7.2-20151112.1 because the old image doesn't contain the patch attached to the bug. We included the patch in ovirt-node-3.2.3-2. To test the patch I recommend test from: rhevh-7.1-20150521.0 (which includes ovirt-node-3.2.3-2) to higher version, like rhev-hypervisor7-7.2-20151123.0.iso. If something goes wrong in the upgrade, it will not show as successfully. About "ERROR Error: Upgrade Failed: Unable to save to file!" that's also seen in this report. This one was fixed in ovirt-node-plugin-vdsm-0.2.0-17, any image that contains this ovirt-node-plugin-vdsm trying to upgrade to a higher should not show this error anymore. Example: rhevh-7.0-20150114.0.el7ev.iso to rhev-hypervisor7-7.2-20151123.0.iso @wanghui, could you please re-test using the images I have pointed and if everything goes well remove the FailedQA flag? Thanks!
Due to this issue is target to fixed in rhev3.6. So I need to verified this issue is ovirt-node-3.6. Test version: rhev-hypversior7-7.2-20151112.1.el7ev ovirt-node-3.6.0-0.20.20151103git3d3779a.el7ev.noarch Upgrade to rhev-hypervisor7-7.2-20151201.2.el7ev Red Hat Enterprise Virtualization Manager Version: 3.6.1-0.2.el6 Test steps: 1. Install rhev-hypversior7-7.2-20151112.1.el7ev 2. Register to rhevm-3.6.1-0.2.el6 3. To trigger the upgrade error as original reported one, add double quota in /etc/default/ovirt like follows. MANAGED_BY=""RHEV-M https://10.66.73.120:443"" 3. Upgrade to rhev-hypervisor7-7.2-20151112.1.el7ev through rhevm 4. Check the events in rhevm. Test result: 1. After step4, upgrade failed in rhevh. And the rhevm events reported upgrade failed alos. Failed to upgrade Host localhost (User: admin@internal). Host localhost installation failed. Upgrade failed, please refer to logs for further information. Failed to install Host localhost. Step: RHEV_INSTALL. Failed to install Host localhost. Step: ovirt-node-upgrade; Details: Upgraded Failed . So this issue is fixed in ovirt-node-3.6.0-0.20.20151103git3d3779a.el7ev.noarch. Changed the status to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0378.html