Bug 1502681 - [Docs][Upgrade] Document the steps needed to recover from failed upgrades to NIST-800 RHVH versions, or to back out of such an upgrade
Summary: [Docs][Upgrade] Document the steps needed to recover from failed upgrades to ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: Documentation
Version: 4.1.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.1.9
: ---
Assignee: Avital Pinnick
QA Contact: Billy Burmester
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-16 12:19 UTC by Huijuan Zhao
Modified: 2019-05-07 12:50 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-08 12:43:08 UTC
oVirt Team: Docs
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
All logs from host (10.21 MB, application/x-gzip)
2017-10-16 12:19 UTC, Huijuan Zhao
no flags Details

Description Huijuan Zhao 2017-10-16 12:19:49 UTC
Created attachment 1339197 [details]
All logs from host

Description of problem:
Failure when trying to upgrade a non-nist system to a nist system and then upgrading from the non-nist system to a different nist system.


Version-Release number of selected component (if applicable):
Build 1: rhvh-4.1-0.20170522.0 (non-nist system)
Build 2: rhvh-4.1-0.20170706.0 (nist system)
Build 3: rhvh-4.1-0.20171012.0 (nist system)

How reproducible:
100%


Steps to Reproduce:
1. Install rhvh-4.1-0.20170522.0
2. Add rhvh to rhvm-4.1, and attach nfs storage domain
3. Upgrade rhvh-4.1-0.20170522.0 to rhvh-4.1-0.20170706.0 via "yum install" in host.
Upgrade successful, can enter to new layer rhvh-4.1-0.20170706.0.
4. Reboot and login old layer rhvh-4.1-0.20170522.0, setup local repos in host(upgrade to rhvh-4.1-0.20171012.0), and upgrade rhvh in rhvm side.


Actual results:
After step 4, upgrade failed.
Due to nist partition(such as /home) is not mounted in non-nist system(4.1-20170522.0), and its LV exists, so there is an exception.


Expected results:
After step 4, should upgrade successful.


Additional info:

Comment 1 Ryan Barry 2017-10-16 12:29:20 UTC
Changing basically every field here...

In general, the flow listed in comment#1 is recommended against anyway, because there's a risk of data loss.

But if a customer encounters a failed upgrade to NIST-800-compatible RHVH, or decides that they want to go back to an older version to resolve some regression and upgrade later, future upgrades will fail.

The RHVH upgrade process essentially checks:

Are NIST-800 partitions mounted? -> skip that block
If not -> create appropriate LVs and mount targets

We do not check:

Do NIST LVs exist but are unmounted? -> remove them

Because of the risk of data loss.

The process to do thsi should be documented.

In general, there are 4 LVs/mounts to worry about:

/tmp
/home
/var/log
/var/log/audit

"lvm lvs" will show something like rhvh_var_log_audit

Which is the LV for /var/log/audit, and needs to be removed.

We should recommend removal of all 4 of these manually. Any logs which may be necessary should be recovered from /var/log and /var/log/audit before removing the LVs.

Comment 8 Avital Pinnick 2018-03-08 12:43:08 UTC
Updated document


Note You need to log in before you can comment on or make changes to this bug.