Bug 1690702

Summary: Live migration fail with unsafe error when GPFS is used as shared filesystem
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Han Han <hhan>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: gaojianan <jgao>
Severity: high Docs Contact:
Priority: high    
Version: 8.0CC: bugproxy, chhu, dyuan, fjin, hannsj_uhl, jdenemar, jsuchane, knoel, lmen, pkrempa, xuzhang, yalzhang
Target Milestone: rcKeywords: OtherQA, Patch, Upstream
Target Release: 8.1Flags: knoel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-5.0.0-10.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1698133 (view as bug list) Environment:
Last Closed: 2019-08-07 10:41:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1679528    
Bug Blocks: 1624641, 1698133    

Description Han Han 2019-03-20 03:28:23 UTC 
This bug was initially created as a copy of Bug #1679528

I am copying this bug because: 
IBM GPFS is a shared fs supported in rhos: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html/configuration_reference/ch_configuring-openstack-shared-file-systems

However, libvirt in rhel doesn't vm migration in GPFS directly. It is better to support it in RHEL. We have encounter 2 similiar issues before:
1. cephfs: https://bugzilla.redhat.com/show_bug.cgi?id=1672178
2. glusterfs.fuse: https://bugzilla.redhat.com/show_bug.cgi?id=1640465


Description of problem:

Live-migration fail with message "error: Unsafe migration: Migration without shared storage is unsafe" when IBM GPFS (Spectrum Scale) is used as shared file system.


Version-Release number of selected component: 4.5.0-10.el7_6.3.x86_64


How reproducible:

Install 2 nodes CentOS 7.6 with last release of libvirt

Install and mount on both nodes GPFS shared file system.

On first node create a vm with disk type file and source file on shared filesystem.

Live migrate the vm on second node.


Steps to Reproduce:
1. yum install libvirtd
2. virsh create vm.xml
3. virsh migrate --live vm  qemu+ssh://second.node/system

Actual results:
error: Unsafe migration: Migration without shared storage is unsafe

Expected results:
no error

Additional info:
If I use the option --unsafe the live-migration works fine. But it isn't the expected behavior.

This impact with our OpenStack environment base con Rocky release
Comment 1 Han Han 2019-03-20 03:31:49 UTC 
For testing, we don't have IBM GPFS servers. Do you know how to borrow some from IBM or other teams?
And do we need to clone this to rhel7.6.z or rhel8 slow train?
Comment 2 Xuesong Zhang 2019-03-25 07:39:16 UTC 
Hi, Hanns-Joachim Uhl,

We do not have the IBM GPFS servers in Red Hat, would you please fins the right person help to verify this BZ while it's ON_QA? Thx.

Note: I'm not sure if you are the correct person for asking help, I find you are tracking most of the IBM related BZs[1]. So I'm here for asking the help, please correct me and share me the correct contact if I'm wrong. Thx a lot.

[1] IBM RHEL8 BZs trackers:
Bug 1564587 (ibm8.0bugs, IBM8.0bugs) - IBM 8.0 Bug tracker
Bug 1570601 (ibm8.1feat, ibm8.1feature) - IBM RHEL 8.1 feature tracker
Bug 1624641 (ibm8.1bugs, IBM8.1bugs) - IBM 8.1 Bug tracker
Bug 1663685 (ibm8.2feat, ibm8.2feature) - IBM RHEL 8.2 feature tracker
Comment 3 IBM Bug Proxy 2019-04-04 06:00:23 UTC 
------- Comment From pthan.com 2019-04-04 01:56 EDT-------
We can reproduce this bug on rhel7.6:

[root@c20f2m5n04 ~]# virsh migrate --live vmip231 qemu+ssh://c20f2m5n03/system

[root@c20f2m5n04 ~]# uname -a
Linux c20f2m5n04.gpfs.net 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@c20f2m5n04 ~]# virsh list
Id    Name                           State
----------------------------------------------------
2     vmip231                        running

[root@c20f2m5n04 ~]#
Comment 4 IBM Bug Proxy 2019-04-08 07:52:29 UTC 
------- Comment From pthan.com 2019-04-08 03:45 EDT-------
Looks like we can reproduce this issue on RH 8 RC2:

[root@localhost ~]# virsh migrate --live vmip231 qemu+ssh://c20f2m5n03/system
error: Unsafe migration: Migration without shared storage is unsafe

[root@localhost ~]# uname -a
Linux localhost.localdomain 4.18.0-80.el8.x86_64 #1 SMP Wed Mar 13 12:02:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# rpm -qf `which virsh`
libvirt-client-4.5.0-23.module+el8+2800+2d311f65.x86_64
Comment 5 Jaroslav Suchanek 2019-04-09 16:18:12 UTC 
Per comment https://bugzilla.redhat.com/show_bug.cgi?id=1679528#c3 this is already pushed upstream.

commit d163b940a73a0d6b8277a4bccef2b60936933cf0
Author: Diego Michelotto <diego.michelotto.it>
Date:   Mon Feb 25 19:19:03 2019 +0100

    virfile: added GPFS as shared fs
    
    Added GPFS as shared file system recognized during live migration
    security checks.
    
    GPFS is 'IBM General Parallel File System' also called
    'IBM Spectrum Scale'
    
    BUG: https://bugzilla.redhat.com/show_bug.cgi?id=1679528
    
    Signed-off-by: Diego Michelotto <diego.michelotto.it>
    Signed-off-by: Peter Krempa <pkrempa>

I will clone this to rhel-8 as well.
Comment 8 gaojianan 2019-06-10 02:37:36 UTC 
For this bug,we can't setup the environment of GPFS,so i can't verify it.
Could you please help to verify it when the bug status changes to ON_QA.

Thank you ！
Comment 10 gaojianan 2019-07-01 06:36:22 UTC 
[Urgent]
Hi,could you help to verify this bug on latest version?
If you have any questions about setting env,you can give comment here.
If not,pls tell me and i will check it with sanity check.
Comment 11 gaojianan 2019-07-17 05:32:54 UTC 
Sanity checked in libvirt-5.0.0-11.module+el8.0.1
In /src/util/virfile.c
...

3619     if ((fstypes & VIR_FILE_SHFS_GPFS) &&
3620         (f_type == GPFS_SUPER_MAGIC))
3621         return 1;
3622 
3623     return 0;
3624 }
...
3785                                  VIR_FILE_SHFS_CEPH |
3786                                  VIR_FILE_SHFS_GPFS);
...

Have added code into it.
Comment 13 errata-xmlrpc 2019-08-07 10:41:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2395