Bug 1305177

Summary: [RHEV-H][beta3] Cannot upgrade RHEV-H to beta3 (from beta2)
Product: Red Hat Enterprise Virtualization Manager Reporter: Marina Kalinin <mkalinin>
Component: rhev-hypervisorAssignee: Fabian Deutsch <fdeutsch>
Status: CLOSED CANTFIX QA Contact: Huijuan Zhao <huzhao>
Severity: urgent Docs Contact:
Priority: medium    
Version: 3.6.3CC: cshao, fdeutsch, gklein, huiwa, huzhao, lsurette, mkalinin, pstehlik, shipatil, ycui, yeylon, ykaul
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Cause: If RHEV-H 3.5.5 or later was installed fresh on a host with EFI support and CSM fallback and now an upgrade of that host to 3.6 is performed Consequence: The user will encounter this error Workaround (if any): Disable EFI support in BIOS or completely reinstall the host with RHEV-H 3.6 Result: The upgrade or reinstallation will work flawlessly
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-04 20:06:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
The error
none
the pre-error screen
none
upgrade exception after disabling uefi
none
upgrade exception screen after disabling uefi none

Description Marina Kalinin 2016-02-05 22:54:16 UTC
Description of problem:
Cannot upgrade RHEV-H to beta3.
After "Update Admin Password" screen, I get this error and nothing else is available - "esc" does not work, nor "F2". Only reboot. And then if I try upgrading again, same problem occurs.

The error:
mount, wrong fs type, bad option, bad superblock ...
See attached screenshot for the full error.


Version-Release number of selected component (if applicable):
Old: rhev-hypervisor7-7.2-20160120.0
New: rhev-hypervisor7-7.2-20160126.0


How reproducible:
Always, so far.

Steps to Reproduce:
1. Provide the ISO with the new version.
2. Restart the box and boot from the iso.
3. Click "Update" on the "Update Admin Password" page.

Actual results:
The host seems to be locked in this error screen. Only reboot available.


Expected results:
The upgrade should work.
And if it does not work, it should at least let hitting the "esc" as advised".


Additional info:
Attaching screenshots.

Comment 1 Marina Kalinin 2016-02-05 22:55:45 UTC
Created attachment 1121525 [details]
The error

Comment 2 Marina Kalinin 2016-02-05 22:56:15 UTC
Created attachment 1121526 [details]
the pre-error screen

Comment 3 Fabian Deutsch 2016-02-08 09:52:36 UTC
Looking at the bug I assume that the system you are using is first trying to bot using EFI or fallback to BIOS otherwise.
The old rhev-hypervisor7-7.2-20160120.0 did not have EFI support, and was likely installed in BIOS mode.

The new rhev-hypervisor7-7.2-20160126.0 does support EFI, and thus the system booted the ISO in EFI mode.

I see two options:

1. Disable EFI on the system you are using
2. Keep EFI enabled and reinstall the host

Comment 5 Marina Kalinin 2016-02-26 17:15:30 UTC
Created attachment 1130867 [details]
upgrade exception after disabling uefi

Comment 6 Marina Kalinin 2016-02-26 17:16:49 UTC
Created attachment 1130869 [details]
upgrade exception screen after disabling uefi

Comment 7 Marina Kalinin 2016-02-26 17:31:33 UTC
Hi Fabian,
I apologize it took me such a long time to reply.
I disabled UEFI and tried to upgrade again.
After reboot it brought me to the upgrade page again, so I believe the upgrade failed. So do the logs. From ovirt-node.log:

~~~
2016-02-26 17:03:57,351      ERROR Installer transaction failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt/node/installer/core/progress_page.py", line 126, in __run
    self.progress_plugin.dry_or(do_commit)
  File "/usr/lib/python2.7/site-packages/ovirt/node/plugins.py", line 188, in dry_or
    return func()
  File "/usr/lib/python2.7/site-packages/ovirt/node/installer/core/progress_page.py", line 120, in do_commit
    tx_element.commit()
  File "/usr/lib/python2.7/site-packages/ovirt/node/installer/core/progress_page.py", line 284, in commit
    boot_setup = install.ovirt_boot_setup()
  File "/usr/lib/python2.7/site-packages/ovirtnode/install.py", line 667, in ovirt_boot_setup
    if not self.grub2_install():
  File "/usr/lib/python2.7/site-packages/ovirtnode/install.py", line 339, in grub2_install
    _functions.remove_efi_entry(self.efi_name)
  File "/usr/lib/python2.7/site-packages/ovirtnode/ovirtfunctions.py", line 1805, in remove_efi_entry
    for entry in efi.list_entries():
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/system.py", line 899, in list_entries
    lines = self._efibootmgr([("verbose", None)])
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/system.py", line 870, in _efibootmgr
    return self._call(cmd)
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/system.py", line 873, in _call
    return process.check_output(cmd)
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/process.py", line 159, in check_output
    stdout = subprocess.check_output(*args, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['efibootmgr', '--verbose']' returned non-zero exit status 2
~~~

Comment 8 Marina Kalinin 2016-02-26 17:51:11 UTC
Fabian, let me know what should I do next.
I'll keep the host as is for now.

I believe I can reinstall, but question is if this problem would not occur upgrading 3.5 hosts to 3.6 because of this problem.

Comment 9 Marina Kalinin 2016-03-03 21:13:40 UTC
I just tried upgrading with beta3 and same problem is there. Upgrade fails with same error.

Comment 10 Fabian Deutsch 2016-03-04 10:55:33 UTC
Marina, could you please try reinstalling the host with 3.5 and then upgrade to 3.6 beta3 to see if the bug exists?

Comment 11 Fabian Deutsch 2016-03-04 11:00:01 UTC
Can the following scenario be tested:

1. Install RHEV-H 3.5 on a host with EFI and CSM enabled (fallback to BIOS if EFI is not found)
2. Media upgrade to RHEV-H 3.6 rc1 or GA

Comment 12 Huijuan Zhao 2016-03-04 11:32:43 UTC
we tested the following for upgrade, all passed:

1. UEFI machine upgrade
  1.1 Upgrade from rhev-hypervisor7-7.1-20150420.0.iso to rhevh-7.2-20160301.1.el7ev.iso on UEFI machine - PASS
  1.2 Upgrade from rhev-hypervisor7-7.1-20150603.0.iso to rhevh-7.2-20160126.0.el7ev.iso on UEFI machine(hp-bl460cg9) - PASS

2. BIOS machine upgrade
  2.1 Upgrade from 7.2_3.6 Beta3 rhevh-7.2-20160126.0.el7ev.iso and RC1(rhevh-7.2-20160225.0.el7ev.iso) to rhevh-7.2-20160301.1.el7ev.iso - PASS
  2.2 Upgrade from 7.2_3.5.8 to rhevh-7.2-20160301.1.el7ev.iso - PASS
  2.3 Upgrade from 7.1_3.5.z latest rhevh-7.1-20151015.0 to rhevh-7.2-20160301.1.el7ev.iso - PASS

Comment 13 Fabian Deutsch 2016-03-04 11:34:28 UTC
Thanks for the clarification.

But how is the EFi machine configured?
Is EFi only configured or EFI with CSM fallback?

Comment 14 Huijuan Zhao 2016-03-04 13:09:33 UTC
UEFI machine upgrade on hp-bl460cg9 server:

1. Both enable UEFI and CSM, all the original testing base on this scenario: 
   1.1 Upgrade from rhev-hypervisor7-7.1-20150420.0.iso to rhevh-7.2-20160301.1.el7ev.iso on UEFI machine - PASS
   1.2 Upgrade from rhev-hypervisor7-7.1-20150603.0.iso to rhevh-7.2-20160126.0.el7ev.iso on UEFI machine(hp-bl460cg9) - PASS

2. Only enbale UEFI, CSM disable, I tested this scenario just now:
  Upgrade from 7.2-20160126.0.el7ev.iso to rhevh-7.2-20160301.1.el7ev.iso on UEFI machine(hp-bl460cg9) - PASS

Comment 15 Marina Kalinin 2016-03-04 17:35:51 UTC
(In reply to Fabian Deutsch from comment #11)
> Can the following scenario be tested:
> 
> 1. Install RHEV-H 3.5 on a host with EFI and CSM enabled (fallback to BIOS
> if EFI is not found)
> 2. Media upgrade to RHEV-H 3.6 rc1 or GA

I enabled both UEFI and CSM, installed 3.5.8 and then tried upgrading - same error as the one that I opened the bug for.

Comment 16 Marina Kalinin 2016-03-04 18:11:22 UTC
<rbarry_> You need to wipe it (reinstall/uninstall from an ISO)
 If it's booted in EFI the 2nd time, it's going to try to mount the EFI system partition, which won't exist
 So the upgrade needs to be legacy boot as well
 We'll have to tell customers to reinstall legacy systems upgraded to EFI (or just upgrade from a legacy boot), but I think that already got documented when EFI was added the first time
<mku> rbarry_, so, I tried with legacy only and it failed for another reason
 rbarry_, it is ok with me
 rbarry_, do you have any idea where it was documented maybe?
<rbarry_> Probably a kbase article. I'll see if I can find one
<mku> fabiand_afk, following this conversation, are you ok if I close the bug with a statement that if UEFI is in use, reinstall is needed?


I'll close the bug then.

Comment 17 Marina Kalinin 2016-03-04 20:06:58 UTC
https://access.redhat.com/solutions/2190591