Bug 1614971

Summary: Upgrading RHV-H from 4.0.X to 4.2 is failing during migrate_var
Product: Red Hat Enterprise Virtualization Manager Reporter: nijin ashok <nashok>
Component: imgbasedAssignee: Yuval Turgeman <yturgema>
Status: CLOSED ERRATA QA Contact: Yaning Wang <yaniwang>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.2.5CC: abpatil, chyan, cshao, dfediuck, huzhao, mtessun, peyu, qiyuan, rbarry, sbonazzo, sgoodman, weiwang, yaniwang, ycui, yturgema
Target Milestone: ovirt-4.2.7Keywords: ZStream
Target Release: ---Flags: peyu: testing_plan_complete+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: imgbased-1.0.25 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-05 15:00:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description nijin ashok 2018-08-10 21:36:31 UTC
Description of problem:

The 4.0 RHV-H is not having glusterfs-server installed. So the file "/usr/libexec/glusterfs/glusterfind/S57glusterfind-delete-post.py" doesn't exist in 4.0 which come from this package.

While upgrading from 4.0 to 4.2, we will sync items present in the new 4.2 /var/ to the existing /var/ filesystem. Since "/var/lib/glustersd" directory is not present in the existing var filesystem, it will be synced.

===
2018-08-10 19:53:18,391 [DEBUG] (migrate_var) Copying /tmp/mnt.6fSmc//var/lib/glusterd to /var/lib/glusterd
===

We have symlink "/var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post" which points to "/usr/libexec/glusterfs/glusterfind/S57glusterfind-delete-post.py" under this directory. Since file S57glusterfind-delete-post.py doesn't exist in the 4.0 server, this symlink will be a often/broken.

===
[root@ ~]# imgbase w
[INFO] You are on rhvh-4.0-0.20170104.0+1

[root@ ~]# mount /dev/rhvh/rhvh-4.2.5.1-0.20180801.0+1 /tmp/rhvh-4.2
[root@ ~]# cd /tmp/rhvh-4.2
[root@ rhvh-4.2]# ls -l var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post
lrwxrwxrwx. 1 root root 64 Aug  1 14:05 var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post -> /usr/libexec/glusterfs/glusterfind/S57glusterfind-delete-post.py

rhvh-4.2]# python
Python 2.7.5 (default, Aug  2 2016, 04:20:16) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path.exists("/tmp/rhvh-4.2/var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post")
False
====

As per https://gerrit.ovirt.org/#/c/92376/, we are syncing files as well after syncing the directory. Although the link 
"/var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post" is already copied to existing /var/ as a part of "directory" sync, the imgbased will try to sync it again since it will fail the "os.path.exists" check as it's a broken link. This copy will fail with "No such file or directory:" since source file doesn't exist.


===
2018-08-10 19:53:18,601 [DEBUG] (migrate_var) Copying /tmp/mnt.6fSmc//var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post to /var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post

2018-08-10 19:53:29,742 [DEBUG] (MainThread) Traceback (most recent call last):
  File "/tmp/tmp.ybG3syJOJ9/usr/lib/python2.7/site-packages/imgbased/utils.py", line 1491, in run
    self._function(*self._args, **self._kwargs)
  File "/tmp/tmp.ybG3syJOJ9/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 245, in migrate_var
    shutil.copy2(newlv_path, realpath)
  File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'/tmp/mnt.6fSmc//var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post'
====

Hence the upgrade will fail.


Version-Release number of selected component (if applicable):

imgbased-1.0.22

How reproducible:

100%

Steps to Reproduce:

Upgrade RHV-H 4.0 directly to 4.2.

Actual results:

The upgrade is failing during "var" migration

Expected results:

The upgrade should work fine.

Additional info:

Comment 2 Yuval Turgeman 2018-08-15 08:32:56 UTC
Thanks for the detailed description, Nijin !

Comment 3 cshao 2018-09-13 12:11:04 UTC
Test version:
rhvh-4.0-0.20170104.0+1
rhvh-4.2.5.1-0.20180801.0

Test steps:
1. Install rhvh-4.0-0.20170104.0+1
2. Upgrade to rhvh-4.2.5.1-0.20180801.0
3. Reboot and login.

Test result:
The upgrade is failing, only can login to RHVH 4.0.


# imgbase  w 
[INFO] You are on rhvh-4.0-0.20170104.0+1
# mount /dev/rhvh/rhvh-4.2.5.1-0.20180801.0+1 /tmp/rhvh-4.2
# cd /tmp/rhvh-4.2
# ls -l var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post
lrwxrwxrwx. 1 root root 64 Aug  1 04:35 var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post -> /usr/libexec/glusterfs/glusterfind/S57glusterfind-delete-post.py



# python
Python 2.7.5 (default, Aug  2 2016, 04:20:16) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path.exists("/tmp/rhvh-4.2/var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post")
False
>>>

Comment 5 Yaning Wang 2018-09-25 08:57:11 UTC
Test version:

rhvh-4.0-0.20170104.0+1
rhvh-4.2.7.0-0.20180918.0+1

Test steps:
1. Install rhvh-4.0-0.20170104.0+1
2. Upgrade to rhvh-4.2.5.1-0.20180801.0
3. Reboot and login.

Test result:
upgrade success


[root@dhcp-8-129 ~]# imgbase w
You are on rhvh-4.2.7.0-0.20180918.0+1
[root@dhcp-8-129 ~]# imgbase layout
rhvh-4.0-0.20170104.0
 +- rhvh-4.0-0.20170104.0+1
rhvh-4.2.7.0-0.20180918.0
 +- rhvh-4.2.7.0-0.20180918.0+1

Comment 8 errata-xmlrpc 2018-11-05 15:00:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3470

Comment 9 Daniel Gur 2019-08-28 13:13:00 UTC
sync2jira

Comment 10 Daniel Gur 2019-08-28 13:17:12 UTC
sync2jira