Bug 1536222

Summary: [ESXi][RHEL7.5]Freezing filesystems on a VM that has bind-mounted filesystems fails with ENXIO
Product: Red Hat Enterprise Linux 7 Reporter: Alexandros Panagiotou <apanagio>
Component: open-vm-toolsAssignee: Cathy Avery <cavery>
Status: CLOSED ERRATA QA Contact: ldu <ldu>
Severity: high Docs Contact:
Priority: high    
Version: 7.4CC: boyang, cavery, jen, jherrman, jjarvis, jsavanyo, jsuchane, knoel, ldu, leiwang, mrezanin, mtessun, ravindrakumar, rjones, yacao
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: open-vm-tools-10.2.5-2.el7 Doc Type: If docs needed, set a value
Doc Text:
Previously, performing a quiesced snapshot of a guest virtual machine failed if the guest had a bind mount configured. With this update, the quiesce operation skips bound mount points, and a quiesced snapshot can be successfully performed on the affected guests.
Story Points: ---
Clone Of:
: 1582123 (view as bug list) Environment:
Last Closed: 2018-10-30 09:48:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1527233    
Bug Blocks: 1582123    
Attachments:
Description Flags
reproducer steps provided by customer none

Description Alexandros Panagiotou 2018-01-18 23:05:28 UTC
Created attachment 1383085 [details]
reproducer steps provided by customer

Description of problem:
Freezing filesystems on a VM that has bind-mounted filesystems fails with ENXIO.

Version-Release number of selected component (if applicable):
open-vm-tools-10.1.5-3.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Install haproxy and create the bind-mount:
   yum install haproxy
   mkdir /var/lib/haproxy/dev
   touch /var/lib/haproxy/dev/log
   mount -o bind /dev/log /var/lib/haproxy/dev/log
2. Configure haproxy to use /dev/log
3. Start haproxy: systemctl start haproxy
4. Run backup

A more detailed description can be found in the attached text file.

Actual results:
Freezing filesystems fails. The following logs are written in vmsvc logs:

...
[Dec 18 14:10:23.743] [   debug] [vmsvc] SyncDriver: failed to open '/var/lib/haproxy/dev/log': 6 (No such device or address)
...
[Dec 18 14:10:23.745] [ warning] [vmbackup] Error freezing filesystems.
...

Expected results:
Freezing filesystems succeeds.

Additional info:
Upstream bug report and pull request (accepted by VMware): https://github.com/vmware/open-vm-tools/pull/218

Comment 2 Richard W.M. Jones 2018-01-18 23:12:37 UTC
Fix is super-simple, but it's very late in the RHEL 7.5 cycle.
I'll leave this one up to Jeff to decide.  However even if we
don't take it for 7.5 GA, I think it should be in a 7.5 z-stream
release.

Comment 3 Junfeng Wang 2018-01-22 09:34:50 UTC
this issue can be reproduced in our environment

Comment 4 Cathy Avery 2018-01-22 12:09:44 UTC
(In reply to Richard W.M. Jones from comment #2)
> Fix is super-simple, but it's very late in the RHEL 7.5 cycle.
> I'll leave this one up to Jeff to decide.  However even if we
> don't take it for 7.5 GA, I think it should be in a 7.5 z-stream
> release.

Whats the fix? Is it upstream?

Comment 5 Richard W.M. Jones 2018-01-22 12:12:57 UTC
https://github.com/vmware/open-vm-tools/pull/218

It is NOT upstream as far as I can tell.

Comment 6 Ravindra Kumar 2018-01-22 19:05:13 UTC
Upstream fix is planned for the next minor update of open-vm-tools 10.2.x, which is couple of months away.

Comment 7 Cathy Avery 2018-01-24 14:39:41 UTC
OK then this bug should be targeted for 7.6.

Comment 8 Jeff Nelson 2018-01-24 16:38:33 UTC
Per comments 6 and 7, targeting for 7.6.

Comment 9 Alexandros Panagiotou 2018-04-20 18:09:47 UTC
Hello, 
From https://github.com/vmware/open-vm-tools/releases, I understand that open-vm-tools 10.2.5 is now released and as far as I can tell it contains the fix - therefore it is now upstream.

Would it make sense to add the "Patch" keyword? Also given #2, does it make sense to keep it waiting for 7.6?

Thanks and Regards,
Alexandros

Comment 10 Cathy Avery 2018-04-20 19:03:47 UTC
(In reply to Alexandros Panagiotou from comment #9)

I'm reassigning this to Jeff. The rebase is in his domain. I think the patch has to be in 7.6 before we can backport it to z-stream. Rich will correct me if I'm wrong.

Comment 12 Jeff Nelson 2018-04-20 23:19:08 UTC
Because the rebase is expected to fix the problem, I've set the status to POST. This is one of several conditions for requesting a backport to 7.5.z, so I've gone ahead and requested that too. It would help if someone could provide a justification for the backport.

Comment 17 Cathy Avery 2018-05-24 13:29:57 UTC
@Ravindra

Hi,

I'm going to need an upstream patch commit for this fix in order to insert it into the 7.5.z stream. I will also subsequently provide a build for you to test.

Thanks,

Cathy

Comment 18 Ravindra Kumar 2018-05-24 23:10:46 UTC
(In reply to Cathy Avery from comment #17)
> I'm going to need an upstream patch commit for this fix in order to insert
> it into the 7.5.z stream.

It is here - https://github.com/vmware/open-vm-tools/commit/6559b7dfd9f6f40c2df76e1b1e785fa3eb1c1da4.

Comment 19 Yaju Cao 2018-06-14 09:30:43 UTC
Change to 'ON_QA' since fix is included in rebase Bug 1527233, of which status is ON_QA.

Comment 20 ldu 2018-08-09 07:51:42 UTC
verify this bug with RHEL7.6 guest on VMware ESXi6.0

Verify version:
kernel-3.10.0-862.el7.x86_64
open-vm-tools-10.2.5-2.el7.x86_64

Verify steps:
1. Install haproxy and create the bind-mount:
   yum install haproxy
   mkdir /var/lib/haproxy/dev
   touch /var/lib/haproxy/dev/log
   mount -o bind /dev/log /var/lib/haproxy/dev/log
2. Configure haproxy to use /dev/log
3. Start haproxy: systemctl start haproxy
4. Take sanpshot with quiesce guest file system

The test result: Verified
The snapshot with quiesce guest file system succeeds and no error log found.

Comment 22 errata-xmlrpc 2018-10-30 09:48:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3104