Bug 1281845 - [RFE] Preserve local Storage Domains when installing RHEV-H
[RFE] Preserve local Storage Domains when installing RHEV-H
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: rhev-hypervisor-ng (Show other bugs)
3.6.0
All Linux
medium Severity medium
: ovirt-4.0.2
: ---
Assigned To: Douglas Schilling Landgraf
cshao
: FutureFeature, TestOnly
Depends On: 1356087
Blocks: ovirt-node-ng 1362310
  Show dependency treegraph
 
Reported: 2015-11-13 10:13 EST by Alexandros Gkesos
Modified: 2017-08-17 20:26 EDT (History)
19 users (show)

See Also:
Fixed In Version: rhev-hypervisor7-ng-3.6-20160429.0
Doc Type: Enhancement
Doc Text:
You can preserve disk content, including the local storage domain, during a reinstallation of RHVH. Additional configuration during the setup is required.
Story Points: ---
Clone Of:
: 1362310 (view as bug list)
Environment:
Last Closed: 2016-08-23 17:04:21 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Node
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
cshao: testing_plan_complete+


Attachments (Terms of Use)
anaconda-auto-manual-partition (124.03 KB, image/png)
2016-06-17 03:16 EDT, cshao
no flags Details
activate.png (20.17 KB, image/png)
2016-08-18 06:43 EDT, Yihui Zhao
no flags Details
data_terminal.png (12.02 KB, image/png)
2016-08-18 06:43 EDT, Yihui Zhao
no flags Details
local_rhevm.png (20.93 KB, image/png)
2016-08-18 06:44 EDT, Yihui Zhao
no flags Details
maintenance.png (23.45 KB, image/png)
2016-08-18 06:45 EDT, Yihui Zhao
no flags Details
rhevm.data.png (23.47 KB, image/png)
2016-08-18 06:46 EDT, Yihui Zhao
no flags Details
rhevm_import.png (38.06 KB, image/png)
2016-08-18 06:46 EDT, Yihui Zhao
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2804081 None None None 2016-12-14 00:41 EST

  None (edit)
Description Alexandros Gkesos 2015-11-13 10:13:20 EST
3. What is the nature and description of the request?
-
 4. Why does the customer need this? (List the business requirements here) 
Yes, it's truly important.

5. How would the customer like to achieve this? (List the functional requirements here) 
* The RHEV-H installer should detect the presence of a local storate domain. If it exists should show an option to preserve it (enabled by default).

6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented. 
* Simply perform an installation of a RHEV-H node with a local storage domain and see if it has not been clobered by the installation routine.

7. Is there already an existing RFE upstream or in Red Hat Bugzilla? 
AFAIK, no.

8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL5, RHEL6)? 
RHEV 3.6, if possible.

9. Is the sales team involved in this request and do they have any additional input? 
No.

10. List any affected packages or components. 
RHEV-H installer.

11. Would the customer be able to assist in testing this functionality if implemented? 
Absolutely yes.
Comment 4 Fabian Deutsch 2016-03-23 05:52:13 EDT
The RHEV-H Next installer will provide the flexibility to choose a custom layout which only needs to suite a few requirements.

With this flexibility it will be possible to choose a custom partitioning layout where the local storage domain can be preserved upon reinstallation of RHEV-H.
I.e. the local storage domain can be put onto a separate LV.

However, this is not done automatically, it needs to be considered during the initial installation.

Example:
1. During the first installation
   - Use a thon LVVM layout
   - Create a separate volume for /localsd
2. Install
3. Reinstall
   - Clear volumes, except for the volume used for /localsd

Does this flow meet the requirements?
Comment 5 Fabian Deutsch 2016-04-07 11:09:14 EDT
This can now be tested with upstream oVirt Node Next builds.
Comment 6 Alexandros Gkesos 2016-04-11 11:05:57 EDT
That's the reply from the customer:

This new feature is a very interesting improvement towards the feature we are requesting.

We think also that would be very useful that the installer warn you before overwriting a local storage, as this kind of operations could be critical in environments using local storage.
Comment 7 Fabian Deutsch 2016-04-12 04:15:15 EDT
Thanks Alexandros, that sound slike another RFE.

Let's take this RFE to track that there is the ability to keep local SDs.
An additional RFE should be filed to get a warning during reinstallation, if there is the risk of destroying an local SD.
Comment 8 Alexandros Gkesos 2016-04-14 10:58:33 EDT
Thank you for the reply Fabian.
I have filed:

Bug 1327248 - [RFE] Warning during rhev-h reinstallation, if there is the risk of destroying a local SD.
https://bugzilla.redhat.com/show_bug.cgi?id=1327248
Comment 10 cshao 2016-06-17 03:16:00 EDT
Hi Fabian,

According #c4 & 6, I did the following testing.

1. Anaconda interactive installation on local machine, automatic partition, installer will pop-up warn info before overwriting a local storage like attachment "anaconda-auto-manual-partition" show.

2. Anaconda interactive installation on local machine, manual partition, installer still can pop-up warn info before overwriting a local storage.

3. ks installation on local machine, no warning info, but I don't think it is necessary.

Test version:
rhev-hypervisor7-ng-4.0-20160616.0.x86_64
imgbased-0.7.0-0.1.el7ev.noarch
redhat-release-rhev-hypervisor-4.0-0.7.el7.x86_64

So can I verify this bug with above test scenarios?

Thanks.
Comment 11 cshao 2016-06-17 03:16 EDT
Created attachment 1168944 [details]
anaconda-auto-manual-partition
Comment 12 Fabian Deutsch 2016-06-17 04:47:07 EDT
Chen, to me the testing in comment 10 rather looks like the verification of bug 1327248, see comment 8.


To verify this bug, you the test steps described in comment 4.

In other words again (and a bit different than comment 4):

1. Install NGN manually
   Create one VG for NGN installatin
   Create another separate VG with an LV for /localsd
2. Add NGN to Engine
3. Create local SD on /localsd
4. Ensure setup is working
5. Reinstall NGN
   - Boot from ISO
   - Remove the VG used for NGN installation
   - Create a new VG for NGN installation
   - KEEP THE /localsd VG (but no mountpoint set)
6. Add host to engine
7. Setup /localsd
8. Host into maintenance
9. Mount LV from separate VG to /localsd
10. Host tp up again

After 10: DC with localsd should be UP
Comment 13 cshao 2016-07-13 07:50:21 EDT
(In reply to Fabian Deutsch from comment #12)
> Chen, to me the testing in comment 10 rather looks like the verification of
> bug 1327248, see comment 8.
> 
> 
> To verify this bug, you the test steps described in comment 4.
> 
> In other words again (and a bit different than comment 4):
> 
> 1. Install NGN manually
>    Create one VG for NGN installatin
>    Create another separate VG with an LV for /localsd
> 2. Add NGN to Engine
> 3. Create local SD on /localsd
> 4. Ensure setup is working
> 5. Reinstall NGN
>    - Boot from ISO
>    - Remove the VG used for NGN installation
>    - Create a new VG for NGN installation
>    - KEEP THE /localsd VG (but no mountpoint set)
> 6. Add host to engine
> 7. Setup /localsd
> 8. Host into maintenance
> 9. Mount LV from separate VG to /localsd
> 10. Host tp up again
> 
> After 10: DC with localsd should be UP

This bug is blocked by bug 1356087. I will verify this bug after 1356087 fixed.
Comment 14 Ying Cui 2016-07-15 05:18:42 EDT
Fabian, let's do not consider the bug 1356087 in comment 13, just consider the comment 12.

For step 3 in comment 12, the host initialized the meta data on this LV.
For step 7 in comment 12, I don't quite understand what the specific operation is, does it need to attach the /localsd to new host? 
For step 9 in comment 12, still not sure what should do.
For "After 10:" in comment 12, base on step 7 and step 9, I don't think DC with /localsd can be up unless the metadata is cleared. but if the meta data is cleared, why we preserved this LV during installation?

Let's try to clarify the requirement here:
Would the customer like to _re-use_ the local domain with initial metadata? Does that means if during installation, the local storage domain is preserved, after installation, customer can UP the previous local storage domain and all initial meta data can be recognized?
Comment 16 Ying Cui 2016-08-01 00:57:20 EDT
According to comment 14, I have to reassign this bug to get the attention.
Comment 17 Fabian Deutsch 2016-08-01 08:10:17 EDT
Let me clarify my steps and also reply to comment 14:

1. Install NGN manually
   Create one VG for NGN installatin
   Create another separate VG with an LV for OR use a separate disk for /localsd
2. Create a new cluster (for local SD) and add NGN to to this cluster
3. Create local SD on /localsd
4. Ensure this setup is working
5. Reinstall NGN
   - Boot from ISO
   - Remove the VG used for NGN installation
   - Create a new VG for NGN installation
   - KEEP THE /localsd VG (but no mountpoint set)
6. Create a new cluster and add the reinstalled host to this cluster
7. Recover /localsd using:
7.a Mount the existing localsds VG OR separate device to /localsd_old
7.b mkdir /localsd
7.c Import the /localsd_old into /localsd as described here: http://www.ovirt.org/develop/release-management/features/storage/importstoragedomain/

This procedur should
a) Keep/Not delete the plain data on the localsd VG or the separate device
b) Allow th euser to use the localsd again after reinstallation

Ying, does this make sense?

Julie, I wonder if this is something for the docs, or rather for a kbase.
Comment 18 Ying Cui 2016-08-01 11:56:45 EDT
(In reply to Fabian Deutsch from comment #17)
> Let me clarify my steps and also reply to comment 14:
> 
> 1. Install NGN manually
>    Create one VG for NGN installatin
>    Create another separate VG with an LV for OR use a separate disk for
> /localsd
> 2. Create a new cluster (for local SD) and add NGN to to this cluster
> 3. Create local SD on /localsd
> 4. Ensure this setup is working
> 5. Reinstall NGN
>    - Boot from ISO
>    - Remove the VG used for NGN installation
>    - Create a new VG for NGN installation
>    - KEEP THE /localsd VG (but no mountpoint set)
> 6. Create a new cluster and add the reinstalled host to this cluster
> 7. Recover /localsd using:
> 7.a Mount the existing localsds VG OR separate device to /localsd_old
> 7.b mkdir /localsd
> 7.c Import the /localsd_old into /localsd as described here:
> http://www.ovirt.org/develop/release-management/features/storage/
> importstoragedomain/

Thanks Fabian.
I knew we could export to NFS domain and import the VM/template again before. 
But step 7, I still need to consider it tomorrow. 


> 
> This procedur should
> a) Keep/Not delete the plain data on the localsd VG or the separate device
> b) Allow th euser to use the localsd again after reinstallation
> 
> Ying, does this make sense?

The expectations are correct, keep the plain data and user can reuse the data again after reinstallation.
Comment 21 Douglas Schilling Landgraf 2016-08-17 17:16:22 EDT
Hi, 

Here the steps I made:

Create a virtual machine with 2 hard-disks

#1) Installed redhat-virtualization-host-4.0-20160812.0.x86_64.liveimg.squashfs via kickstart

    Boot the iso: RHEL-7.2-20151030.0-Server-x86_64-boot.iso added to the grub 
    lines:
              ks=http://192.168.125.1/user.cfg

#2) Created directory in the Hypervisor
    # mkdir /data

    # fdisk /dev/sdb (created partition 1)
    # mkfs.ext4 /dev/sdb1
    # mount -t ext4 /dev/sdb1 /data
    (Added as well in fstab to auto-mount)

    # chown 36:36 /data

#3) In the RHEV Admin page:
     - Created Datacenter/Cluster for local storage
     - Registered/Approved the Host to the new Datacenter/Cluster
     - Created Local Storage specifying /data in the Storage tab

* At this point Host is UP and Storage is UP

Rebooted the host to do the re-installation:

Manual re-install:
======================
- Anaconda complains about having storage already used (/dev/sda and /dev/sdb) and asked to click in the storage icon to resolve it. I have clicked and DESELECT the /dev/sdb storage to avoid any issue with my local storage.


Automatic re-install:
=======================
-   Boot the iso: RHEL-7.2-20151030.0-Server-x86_64-boot.iso added to the grub 
    lines:
              ks=http://192.168.125.1/user.cfg


   * Now, I have changed the kickstart to change clearpart option to only touch 
     the sda partitions.
     <snip>
     clearpart --all --drives=sda
     </snip>

After the reinstall
========================
I can see the local storage data available in the /dev/sdb1:

# mount /dev/sdb1 /mnt
# ls /mnt/
__DIRECT_IO_TEST__  e44d9940-9fde-47c6-be0d-d78cf3f0adfd  lost+found


==== user.cfg ==================
#Authconfig set
authconfig --enableshadow --passalgo=md5

#Keyboard type set
keyboard us

#Default language set
lang en_US
liveimg --url=http://192.168.125.1/redhat-virtualization-host-4.0-20160812.0.x86_64.liveimg.squashfs

#Bootloader location set
bootloader --location=mbr

#Root password set
rootpw --plaintext redhat

#Clear partitions before disk part
clearpart --all

#Auto partition in thinp mode
autopart --type=thinp

%post --erroronfail
imgbase layout --init
imgbase --experimental volume --create /var 4G
%end

#Text mode for installer
text

#Reboot system after install
reboot
Comment 22 Douglas Schilling Landgraf 2016-08-17 17:22:50 EDT
To import the existing domain:

- In the node, prepare the /data path again:

  # mkdir /data
  # mount /dev/sdb1 /data
  <The previous storage still there>
  # chown 36:36 /data


- Registered the node into RHEVM (4.0.2.6-0.1.el7ev)
- Created Datacenter/Cluster for local storage
- Create a new local storage in /mnt 
  (I couldn't import directly the previous one)

< At this moment everything should be up >

- Now in RHEVM, go to storage and click import and provide /data
  and later active it in the DataCenter tab -> Storage.

Finally, if you put the temporary local storage domain from /mnt in maintenance mode it will change the Data (master) storage to the imported domain.

I hope this test helps to clarify.
Comment 23 Ying Cui 2016-08-17 23:55:37 EDT
Thanks Douglas for detail on this bug, we will check this TestOnly bug on 4.0 again to make sure all works well or not during QE testing, the ETA: Aug 21.
Comment 24 Ying Cui 2016-08-18 02:16:42 EDT
Chen, the steps in comment 21 and comment 22 are very clearly steps to us, could we try to this today? It is RFE bug, we would like to make it into GA once pass QE. 

For comment 21 and comment 22, you also need create VM on previous local storage /data, and _after_ reinstalled, and /data storage is up on Host again, then try to start this VM, if VM can be started after reinstallaion host, then we can verify this bug if we do not consider the bug 1356087 in comment 13.
Comment 25 Yihui Zhao 2016-08-18 06:39:45 EDT
Hi Douglas,
   Thank you for detail with steps to reproduce.And follow your steps,i tested successfully.

Here the steps I made:
     My testing environment is the machine with storage "/dev/sda" and the USB drive "/dev/sdb".

#1) Installed redhat-virtualization-host-4.0-20160817.0.x86_64.liveimg.squashfs via GUI with storage "/dev/sda",and "/dev/sdb" keep.

  
#2) Created directory in the Hypervisor
    # mkdir /data

    # fdisk /dev/sdb (created partition 1)
    # mkfs.ext4 /dev/sdb1
    # mount -t ext4 /dev/sdb1 /data
    (Added as well in fstab to auto-mount)

    # chown 36:36 /data
     

#3) In the RHEV Admin page:
     - Created Datacenter/Cluster for local storage
     - Registered/Approved the Host to the new Datacenter/Cluster
     - Created Local Storage specifying /data in the Storage tab

* At this point Host is UP and Storage is UP,and the output of using cmd "ls /data":
[root@dhcp-8-194 /]# ls /data                                                   
0f26c6a8-7617-4ba6-8ab4-1a12cfa12ddf  __DIRECT_IO_TEST__  lost+found 

#4)Rebooted the host to do the re-installation:
     Reinstalled redhat-virtualization-host-4.0-20160817.0.x86_64.liveimg.squashfs via GUI with storage "/dev/sda",and "/dev/sdb" keep.


#5)import the existing domain:

- In the node, prepare the /data path again:

  # mkdir /data
  # mount /dev/sdb1 /data
  <The previous storage still there>
  # chown 36:36 /data

The output of using cmd "ls /data":
[root@dhcp-8-194 /]# ls /data                                                   
0f26c6a8-7617-4ba6-8ab4-1a12cfa12ddf  __DIRECT_IO_TEST__  lost+found 


- Registered the node into RHEVM (4.0.2.6-0.1.el7ev)
- Created Datacenter/Cluster for local storage
- Create a new local storage in /mnt (chown 36:36 /mnt)
  
< At this moment everything should be up >

Now in RHEVM, go to storage and click import and provide /data
  and it will display some information about storage,then the import storage is maintenance,later active it in the DataCenter tab -> Storage.

Finally, i put the temporary local storage domain from /mnt in maintenance mode it will change the Data (master) storage to the imported domain.

Add:some pictures about storage information listed on attachment.

Thank you for all efforts.
Yihui Zhao
Comment 26 Yihui Zhao 2016-08-18 06:43 EDT
Created attachment 1191844 [details]
activate.png
Comment 27 Yihui Zhao 2016-08-18 06:43 EDT
Created attachment 1191846 [details]
data_terminal.png
Comment 28 Yihui Zhao 2016-08-18 06:44 EDT
Created attachment 1191847 [details]
local_rhevm.png
Comment 29 Yihui Zhao 2016-08-18 06:45 EDT
Created attachment 1191848 [details]
maintenance.png
Comment 30 Yihui Zhao 2016-08-18 06:46 EDT
Created attachment 1191849 [details]
rhevm.data.png
Comment 31 Yihui Zhao 2016-08-18 06:46 EDT
Created attachment 1191850 [details]
rhevm_import.png
Comment 32 Fabian Deutsch 2016-08-18 06:58:20 EDT
Just a note, the bug in comment 13 is considered to be an enhancement.
But in today's state it is possible - as described in comment 21 - to setup storage to allow keeping local storage domain data.
Comment 33 cshao 2016-08-18 07:33:57 EDT
Thanks yzhao for the detail testing, cancel the needinfo flag.
Comment 34 cshao 2016-08-18 08:29:10 EDT
Verify this bug according #c25.
Comment 37 errata-xmlrpc 2016-08-23 17:04:21 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1702.html
Comment 38 Alexandros Gkesos 2016-09-16 09:50:33 EDT
Hello Douglas/All,

Thanks for the Enhancement. Customer is happy but came up with one more request.
This enhancement is working for RHVN to RHVN, but is it working from RHEV-H (3.6) to RHVN? Please let me know if not so i can open a Bug (and if there is any possibility that we will implement it to help users migrate easier from 3.6 to 4)
Comment 39 Douglas Schilling Landgraf 2016-09-20 03:51:37 EDT
(In reply to Alexandros Gkesos from comment #38)
> Hello Douglas/All,
> 
> Thanks for the Enhancement. Customer is happy but came up with one more
> request.
> This enhancement is working for RHVN to RHVN, but is it working from RHEV-H
> (3.6) to RHVN? Please let me know if not so i can open a Bug (and if there
> is any possibility that we will implement it to help users migrate easier
> from 3.6 to 4)

From 3.6 to 4 we must check a valid path and test it. We are currently looking at a few options, but it will take some time before there is a result. There is a specific bug report, please see: bz#1320556.
Comment 41 Marina 2017-04-10 14:54:34 EDT
Hi Douglas,

Can you please review current documentation we have on the subject and confirm it is accurate?

First, we have the admin guide section about local storage, with the 2 notes on top related to this bug:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/administration_guide/sect-preparing_and_adding_local_storage

Second, we have a KCS, that the first note is pointing to:
https://access.redhat.com/solutions/2804081

Thank you!
Comment 42 Douglas Schilling Landgraf 2017-04-24 10:16:51 EDT
(In reply to Marina from comment #41)
> Hi Douglas,
> 
> Can you please review current documentation we have on the subject and
> confirm it is accurate?
> 
> First, we have the admin guide section about local storage, with the 2 notes
> on top related to this bug:
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/
> html/administration_guide/sect-preparing_and_adding_local_storage
> 
> Second, we have a KCS, that the first note is pointing to:
> https://access.redhat.com/solutions/2804081
> 
Looks good Marina, we are working to document the steps via:
https://bugzilla.redhat.com/show_bug.cgi?id=1421437

Note You need to log in before you can comment on or make changes to this bug.