Bug 1850378 - Installation of node will not quit when mountpoint has existing domain (VMs)
Summary: Installation of node will not quit when mountpoint has existing domain (VMs)
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-node
Classification: oVirt
Component: Installation & Update
Version: 4.4.1
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ovirt-4.4.2
: 4.4.2
Assignee: Nir Levy
QA Contact: peyu
URL:
Whiteboard:
: 1850278 (view as bug list)
Depends On:
Blocks: 1852721 1863045
TreeView+ depends on / blocked
 
Reported: 2020-06-24 07:48 UTC by Nir Levy
Modified: 2023-09-14 06:02 UTC (History)
15 users (show)

Fixed In Version: ovirt-node-ng-image-update-4.4.2-0.5.rc5
Doc Type: Known Issue
Doc Text:
When you upgrade Red Hat Virtualization from 4.3 to 4.4 with a storage domain that is locally mounted on / (root), the upgrade fails. Specifically, on the host it appears that the upgrade is successful, but the host's status on the Administration Portal, is `NonOperational`. Local storage should always be defined on a file system that is separate from / (root). Use a separate logical volume or disk, to prevent possible loss of data during upgrades. If you are using / (root) as the locally mounted storage domain, migrate your data to a separate logical volume or disk prior to upgrading.
Clone Of:
: 1852721 1863045 (view as bug list)
Environment:
Last Closed: 2020-09-18 07:13:29 UTC
oVirt Team: Node
Embargoed:
sbonazzo: ovirt-4.4?
mtessun: blocker-
peyu: testing_plan_complete+
mtessun: planning_ack+
sbonazzo: devel_ack+
mavital: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 109935 0 master MERGED spec: pre script 2021-01-15 03:21:11 UTC
oVirt gerrit 110086 0 ovirt-4.4 MERGED spec: pre script 2021-01-15 03:21:11 UTC
oVirt gerrit 110628 0 master MERGED spec: pre script 2021-01-15 03:21:11 UTC
oVirt gerrit 110699 0 ovirt-4.4 MERGED spec: pre script 2021-01-15 03:21:11 UTC

Description Nir Levy 2020-06-24 07:48:49 UTC
Description of problem:
rpm host installing rely on depracated implementation
'pre' install searches for  __DIRECT_IO_TEST__
which is no longer set by vdsm gerrit.ovirt.org/#/c/103526/

Version-Release number of selected component (if applicable):
ovirt-node-ng-image-update-4.4.1-0.5.rc4.el8.noarch.rpm 

How reproducible:
installation on disk that has pre-existing domain will run over.

Steps to Reproduce:
1.use disk with has existing domain
2.install node on this disk
3.

Actual results:
instalation succeed

Expected results:
installing will fail with proper massage

Additional info:
VDSM current implementation rely on existance of mountpoint/*/dom_md
we can either mimic this or look for use a stateless non vdsmd api if available or made available easily.

Comment 1 Nir Levy 2020-06-24 07:50:17 UTC
*** Bug 1850278 has been marked as a duplicate of this bug. ***

Comment 2 Sandro Bonazzola 2020-06-24 16:33:31 UTC
proposing this as blocker for 4.4.1 since upgrade from 4.4.0 may end up with data loss.

Comment 3 Michal Skrivanek 2020-06-25 03:03:46 UTC
Is that a concern for local SDs? Gluster? Something else?

Comment 6 Michal Skrivanek 2020-07-02 12:08:51 UTC
patch attached to this buug is merged, is it the last one? if so please move bug to MODIFIED

Comment 8 Michal Skrivanek 2020-07-14 18:12:58 UTC
so upgrades with local storage SD is going to fail? Then it should be in Known Issues.

Comment 9 Sandro Bonazzola 2020-07-27 14:48:37 UTC
(In reply to Michal Skrivanek from comment #8)
> so upgrades with local storage SD is going to fail? Then it should be in
> Known Issues.

With local mounted storage SD. And yes, should be in Known Issues. Steve, can you handle?

Comment 10 Steve Goodman 2020-08-03 09:09:45 UTC
Please review doc_text for the Known Issue.

Is there a workaround? Can you detach a locally mounted storage domain before upgrading? Would that work around the problem?

Comment 11 Steve Goodman 2020-08-03 14:14:08 UTC
After speaking with Nir, I understand that the issue is that the upgrade does not prevent you from upgrading if you have data on your root (/) LV. This current bug fixes this issue by preventing that. So the Known Issue is not for this current bug, which target 4.4.2. 

The Known Issue should appear in the RHV 4.4.1 (GA) release notes, so a separate bug is required targeting 4.4.1 GA.

Comment 12 peyu 2020-09-01 07:11:17 UTC
This issue is solved, but an engine bug(Downstream has solved this issue) was found in Test 1.

Test version:
ovirt-node-ng-installer-4.4.1-2020072310.el8.iso
ovirt-node-ng-image-update-4.4.2-0.5.rc5.el8.noarch.rpm
oVirt Open Virtualization Manager: 4.4.1.4-1.el8



Test 1: 

Test steps: 
Refer to Comment7

Test results:
1. Host is not upgraded, but the engine shows success and host status is up after reboot. This issue refer to Bug 1770893



Test 2: 

Test steps:
1. Install ovirt-node-ng-installer-4.4.1-2020072310.el8.iso
2. Setup local repos in host and point to "ovirt-node-ng-image-update-4.4.2-0.5.rc5.el8.noarch.rpm"
3. Add host to upstream engine
4. Login to host, create a local storage directory under / , and modify the permissions and ownership
   # mkdir /local-storage
   # chown 36:36 /local-storage
   # chmod 0755 /local-storage
5. Adding Local Storage via engine
6. Create a VM on local storage
7. Upgrade host from host side
   # yum update

Test results:
1. Host upgrade was successfully blocked. The information is as follows:
~~~~~~
# yum update
Red Hat update to latest                                                                             164 kB/s | 1.0 kB     00:00    
Dependencies resolved.
=====================================================================================================================================
 Package                                    Architecture           Version                              Repository              Size
=====================================================================================================================================
Installing:
 ovirt-node-ng-image-update                 noarch                 4.4.2-0.5.rc5.el8                    update                 782 M
     replacing  ovirt-node-ng-image-update-placeholder.noarch 4.4.1.4-1.el8

Transaction Summary
=====================================================================================================================================
Install  1 Package

Total download size: 782 M
Is this ok [y/N]: y
Downloading Packages:
ovirt-node-ng-image-update-4.4.2-0.5.rc5.el8.noarch.rpm                                               84 MB/s | 782 MB     00:09    
-------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                 84 MB/s | 782 MB     00:09     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                             1/1 
  Running scriptlet: ovirt-node-ng-image-update-4.4.2-0.5.rc5.el8.noarch                                                         1/2 
Local storage domains were found on the same filesystem as / ! Please migrate the data to a new LV before upgrading, or you will lose the VMs
See: https://bugzilla.redhat.com/show_bug.cgi?id=1550205#c3
Storage domains were found in:
	/local-storage/45b21993-bc2f-4c71-a442-c5982d8b3113/dom_md
error: %prein(ovirt-node-ng-image-update-4.4.2-0.5.rc5.el8.noarch) scriptlet failed, exit status 1

Error in PREIN scriptlet in rpm package ovirt-node-ng-image-update
  Obsoleting       : ovirt-node-ng-image-update-placeholder-4.4.1.4-1.el8.noarch                                                 2/2 
error: ovirt-node-ng-image-update-4.4.2-0.5.rc5.el8.noarch: install failed

  Verifying        : ovirt-node-ng-image-update-4.4.2-0.5.rc5.el8.noarch                                                         1/2 
  Verifying        : ovirt-node-ng-image-update-placeholder-4.4.1.4-1.el8.noarch                                                 2/2 
Unpersisting: ovirt-node-ng-image-update-placeholder-4.4.1.4-1.el8.noarch.rpm

Failed:
  ovirt-node-ng-image-update-4.4.2-0.5.rc5.el8.noarch                                                                                

Error: Transaction failed
~~~~~~



Test 3: 

Test steps: 
1. Install ovirt-node-ng-installer-4.4.1-2020072310.el8.iso
2. Setup local repos in host and point to "ovirt-node-ng-image-update-4.4.2-0.5.rc5.el8.noarch.rpm"
3. Add host to RHVM
4. Login to host, create local storage directory and mount it
   # mkdir /local-storage
   # lvcreate -L 20G onn -n data
   # mkfs.ext4 /dev/mapper/onn-data
   # echo "/dev/mapper/onn-data /local-storage ext4 defaults,discard 1 2" >> /etc/fstab
   # mount /local-storage
   # mount -a
   # chown 36:36 /local-storage
   # chmod 0755 /local-storage
5. Add Local Storage via RHVM
6. Create a VM on local storage
7. Upgrade host via RHVM

Test results:
1. Host upgrade is successful, local storage starts normally after upgrade.



QE will move the bug status to "VERIFIED".

Comment 13 Sandro Bonazzola 2020-09-18 07:13:29 UTC
This bugzilla is included in oVirt 4.4.2 release, published on September 17th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

Comment 14 Red Hat Bugzilla 2023-09-14 06:02:45 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.