Bug 1158036

Summary: Fail to save guest to the pre-create save file which locate in root_squash nfs server
Product: Red Hat Enterprise Linux 6 Reporter: zhenfeng wang <zhwang>
Component: libvirtAssignee: John Ferlan <jferlan>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.7CC: dyuan, gsun, jferlan, jherrman, lhuang, libvirt-maint, mzhan, rbalakri, virt-bugs, vivianzhang, wzhang, ydu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.10.2-49.el6 Doc Type: Bug Fix
Doc Text:
Prior to this update, when using the "virsh save" command to save a domain to an NFS client with the "root squash" access rights reduction while running the libvirtd daemon with a non-default owner:group configuration, saving the NFS client failed with a "Transport endpoint is not connected" error message. This update ensures that the chmod operation during the saving process correctly specifies the non-default owner:group configuration, and using "virsh save" in the described scenario works as expected.
Story Points: ---
Clone Of: 1158034 Environment:
Last Closed: 2015-07-22 05:48:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1158034    
Bug Blocks:    

Description zhenfeng wang 2014-10-28 11:41:43 UTC
+++ This bug was initially created as a clone of Bug #1158034 +++

Description of problem:
Fail to save guest to the pre-create save file which locate in root_squash nfs server

Version-Release number of selected component (if applicable):
libvirt-1.2.8-5.el7.x86_64
qemu-kvm-rhev-2.1.2-3.el7.x86_64
kernel-3.10.0-191.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare a nfs server with root_squash
# cat /etc/exports
/var/tmp/nfs *(rw,root_squash)

# ll /var/tmp/nfs -d
drwxrwxrwx. 2 root root 4096 Sep 29 16:08 /var/tmp/nfs

2.Pre-create the guest's save file and prepare a guest img on the nfs server
# ll /var/tmp/nfs/
total 5077000
-rw-r--r--. 1 root root 4984995840 Oct 28 12:23 rhel7.img
-rw-r--r--. 1 root root  213851864 Oct 28 12:23 rhel7.save

3.Change the guest's image and pre-create save file's group to qemu, so that
qemu group user can read&write the image and pre-create save file
#chmod 777 /var/tmp/nfs/rhel7.img
#chmod 777 /var/tmp/nfs/rhel7.save

#chown root:qemu /var/tmp/nfs/rhel7.img
#chown root:qemu /var/tmp/nfs/rhel7.save

# ll /var/tmp/nfs/
total 5077000
-rwxrwxrwx. 1 root qemu 4984995840 Oct 28 12:23 rhel7.img
-rwxrwxrwx. 1 root qemu  213851864 Oct 28 12:23 rhel7.save

4.keep the qemu.conf default configuration in nfs client
#vim /etc/libvirt/qemu.conf
user = "qemu"
group= "qemu"
dynamic_ownership = 1

5.create a nfs pool on the nfs client
#cat nfs.xml
<pool type='netfs'>
  <name>nfs</name>
  <uuid>08380649-cf25-4af7-b816-6f8494003f69</uuid>
  <capacity unit='bytes'>206423719936</capacity>
  <allocation unit='bytes'>121639010304</allocation>
  <available unit='bytes'>84784709632</available>
  <source>
    <host name='$nfs_server_addr'/>
    <dir path='/var/tmp/nfs'/>
    <format type='nfs'/>
  </source>
  <target>
    <path>/tmp/pl</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

#mkdir /tmp/pl
#virsh pool-define nfs.xml
#virsh pool-start nfs

5.Start a normal guest which guest's img locate in the nfs server
#virsh dumpxml rhel6
--
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/tmp/pl/rhel7.img'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
--

#virsh start rhel6
Domain rhel7 started

6.Save the guest to the pre-create save file, will fail to save the guest to the pre-create save file and report the following error
# virsh save rhel7 /tmp/pl/rhel7.save 
error: Failed to save domain rhel7 to /tmp/pl/rhel7.save
error: Error from child process creating '/tmp/pl/rhel7.save': Transport endpoint is not connected

7.Also hit this issue in rhel6.6 host

Actual results:
Fail to save guest to the pre-create save file which both guest's image and save file locate in the nfs share directory

Expected results:
Should save the guest successfully

Additional info:

Comment 1 zhenfeng wang 2014-10-28 11:44:42 UTC
Get different error info in upper step6 in rhel6.6
In rhel6.6 , I get the following error info
# virsh save rhel7 /tmp/pl/rhel7.save 
error: Failed to save domain rhel7 to /tmp/pl/rhel7.save
error: Error from child process creating '/tmp/pl/rhel7.save': Operation not permitted

Comment 5 vivian zhang 2015-02-27 10:29:06 UTC
Hi, John
I can produce this bug on build libvirt-0.10.2-44.el6.x86_64

Verify it on build with following result, could you please help us confirm whether they are worked as expected now?

libvirt-0.10.2-49.el6.x86_64

1.Prepare a nfs server with root_squash
# cat /etc/exports
/var/tmp/nfs *(rw,root_squash)

# ll /var/tmp/nfs/ -d
drwxrwxrwx. 2 root root 4096 Feb 27 18:05 /var/tmp/nfs/

2. Pre-create the guest's save file and prepare a guest img on the nfs server

3. touch two pre-create save file from nfs server, the default user and group permission are root:root, chown one of them to root:qemu

# touch server1.save
# touch server2.save
# ll
total 3739908
-rwxrwxrwx. 1 root qemu 3829661696 Feb 27 18:05 rhel6.img
-rw-r--r--. 1 root root          0 Feb 27 18:14 server1.save
-rw-r--r--. 1 root qemu          0 Feb 27 18:14 server2.save

4. touch one pre-create save file from nfs client, the default user and group permission are nfsnobody:nfsnobody

# touch client.save
[root@client pl]# ll
total 3739908
-rw-r--r--. 1 nfsnobody nfsnobody          0 Feb 27 05:15 client.save
-rwxrwxrwx. 1 root      qemu      3829661696 Feb 27 05:05 rhel6.img
-rw-r--r--. 1 root      root               0 Feb 27 05:14 server1.save
-rw-r--r--. 1 root      qemu               0 Feb 27 05:14 server2.save

5. keep the qemu.conf default configuration in nfs client
#vim /etc/libvirt/qemu.conf
user = "qemu"
group= "qemu"
dynamic_ownership = 1

6.create a nfs pool on the nfs client, and start the pool
#cat nfs.xml
<pool type='netfs'>
  <name>nfs</name>
  <uuid>08380649-cf25-4af7-b816-6f8494003f69</uuid>
  <capacity unit='bytes'>206423719936</capacity>
  <allocation unit='bytes'>121639010304</allocation>
  <available unit='bytes'>84784709632</available>
  <source>
    <host name='$nfs_server_addr'/>
    <dir path='/var/tmp/nfs'/>
    <format type='nfs'/>
  </source>
  <target>
    <path>/tmp/pl</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

#mkdir /tmp/pl
#virsh pool-define nfs.xml
#virsh pool-start nfs

7.Start a normal guest which guest's img locate in the nfs server

# virsh list
 Id    Name                           State
----------------------------------------------------
 6     test1                          running

8. try to save guest to the nfs server using pre-create save file, all failed

# virsh save test /tmp/pl/server1.save 
error: Failed to save domain test to /tmp/pl/server1.save
error: Error from child process creating '/tmp/pl/server1.save': Permission denied

[root@client pl]# virsh save test /tmp/pl/server2.save 
error: Failed to save domain test to /tmp/pl/server2.save
error: Error from child process creating '/tmp/pl/server2.save': Permission denied

# virsh save test /tmp/pl/client.save 
error: Failed to save domain test to /tmp/pl/client.save
error: Error from child process creating '/tmp/pl/client.save': Operation not permitted

9. check the pre-create save file still existing, libvirt does not delete them even though save guest failed with above error

# ll
total 3739908
-rw-r--r--. 1 nfsnobody nfsnobody          0 Feb 27 18:15 client.save
-rwxrwxrwx. 1 root      qemu      3829661696 Feb 27 18:20 rhel6.img
-rw-r--r--. 1 root      root               0 Feb 27 18:14 server1.save
-rw-r--r--. 1 root      qemu               0 Feb 27 18:14 server2.save

10. try to save guest to nfs server directory with a non-existing file, it is successful

# virsh save test1 /tmp/pl/new.save

Domain test saved to /tmp/pl/new.save

the save file generated success on nfs server and with qemu:qemu permission
]# ll
total 4143012
-rw-r--r--. 1 nfsnobody nfsnobody          0 Feb 27 18:15 client.save
-rw-r-----. 1 qemu      qemu       412776683 Feb 27 18:24 new.save
-rwxrwxrwx. 1 root      qemu      3829661696 Feb 27 18:24 rhel6.img
-rw-r--r--. 1 root      root               0 Feb 27 18:14 server1.save
-rw-r--r--. 1 root      qemu               0 Feb 27 18:14 server2.save


11. restore guest with this save file, also successful, guest works well
# virsh restore /tmp/pl/new.save 
Domain restored from /tmp/pl/new.save

are the above steps enough to verify this bug?

Comment 6 John Ferlan 2015-02-27 11:28:27 UTC
Looks fine to me

Comment 8 errata-xmlrpc 2015-07-22 05:48:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1252.html