Bug 1282288 - Unable to set permission when a volume is created in root squash netfs pool
Unable to set permission when a volume is created in root squash netfs pool
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.2
x86_64 Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: John Ferlan
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-15 22:09 EST by yangyang
Modified: 2016-11-03 14:30 EDT (History)
6 users (show)

See Also:
Fixed In Version: libvirt-1.3.1-1.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-03 14:30:48 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
sample code which should show issue (3.55 KB, text/plain)
2015-11-16 16:23 EST, John Ferlan
no flags Details

  None (edit)
Description yangyang 2015-11-15 22:09:29 EST
Description of problem:
Given a root squash netfs pool, if volume target mode is changed to 0600, raw volume cannot be created successfully with "permission denied" error. But changed mode to 0604, it can be created successfully. But for qcow2 volume, no matter what mode provided, volume are always not permitted to create.

Version-Release number of selected component (if applicable):
libvirt-1.2.17-13.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Setup a nfs server with root_squash
# cat /etc/exports
/var/lib/libvirt/images/nfs *(rw,sync,root_squash)
Change the nfs dir owner to qemu, and chmod to 755.
# chown qemu.qemu /var/lib/libvirt/images/nfs \
&& chmod 755 /var/lib/libvirt/images/nfs


2. Create nfs pool with qemu user and group.
# cat nfs.xml
<pool type='netfs'>
<name>nfs</name>
<source>
<host name='10.66.x.x'/>
<dir path='/var/lib/libvirt/images/nfs'/>
<format type='nfs'/>
</source>
<target>
<path>/mnt/nfs</path>
<permissions>
<mode>0755</mode>
<owner>107</owner>
<group>107</group>
</permissions>
</target>

# virsh pool-define nfs.xml&& virsh pool-build nfs && virsh pool-start nfs
Pool nfs defined from nfs.xml

Pool nfs built

Pool nfs started

3. create raw volume with 644 mode successfully
# cat nfs.vol.xml 
<volume type='file'>
  <name>test.raw</name>
  <capacity unit='bytes'>1048576000</capacity>
  <target>
    <format type='raw'/>
<permissions>
<mode>0644</mode>
<owner>107</owner>
<group>107</group>
</permissions>
  </target>
</volume>

# virsh vol-create nfs nfs.vol.xml 
Vol test.raw created from nfs.vol.xml

# virsh vol-list nfs | grep test.raw
 test.raw             /mnt/nfs/test.raw

4. change mode to 600, raw volume failed to create
# cat nfs.vol.xml 
<volume type='file'>
  <name>test.raw</name>
  <capacity unit='bytes'>1048576000</capacity>
  <target>
    <format type='raw'/>
<permissions>
<mode>0600</mode>
<owner>107</owner>
<group>107</group>
</permissions>
  </target>
</volume>

# virsh vol-create nfs nfs.vol.xml 
error: Failed to create vol from nfs.vol.xml
error: cannot open volume '/mnt/nfs/test.raw': Permission denied

[root@rhel7_test yy]# virsh vol-list nfs | grep test.raw

[root@rhel7_test yy]# ll /mnt/nfs/test.raw
ls: cannot access /mnt/nfs/test.raw: No such file or directory

5. create qcow2 volume with 644 mode, failed
# cat nfs.vol.xml 
<volume type='file'>
  <name>test.qcow2</name>
  <capacity unit='bytes'>1048576000</capacity>
  <target>
    <format type='qcow2'/>
<permissions>
<mode>0644</mode>
<owner>107</owner>
<group>107</group>
</permissions>
  </target>
</volume>

# virsh vol-create nfs nfs.vol.xml 
error: Failed to create vol from nfs.vol.xml
error: cannot set mode of '/mnt/nfs/test.qcow2' to 0644: Operation not permitted

6. change mode to 666, failed to create volume
# cat nfs.vol.xml 
<volume type='file'>
  <name>test.qcow2</name>
  <capacity unit='bytes'>1048576000</capacity>
  <target>
    <format type='qcow2'/>
<permissions>
<mode>0666</mode>
<owner>107</owner>
<group>107</group>
</permissions>
  </target>
</volume>

# virsh vol-create nfs nfs.vol.xml 
error: Failed to create vol from nfs.vol.xml
error: cannot set mode of '/mnt/nfs/test.qcow2' to 0666: Operation not permitted

Actual results:


Expected results:
Raw volume with 600 mode should be created without error
Qcow2 volume should be created without error

Additional info:
Comment 1 John Ferlan 2015-11-16 11:28:01 EST
A bz with multiple things going on - need to separatef raw and qcow2 issues!

First, not sure I'd expect setting the 'raw' image to 0600 to succeed because of the root-squash environment there's no way for "root" to *read* information about the raw volume. Setting it to 666 does work though.  Remember in the root squash environment, the file would be created under qemu:qemu with prots 0600, then when root then goes to read the file it cannot do so. It's a very restrictive environment.  The code to get the created files' information doesn't do so under the uid/gid of what's defined in the pool.  I'm not convinced it should, but if it were to do so would require some effort to do so.

With respect to the qcow2 file creation using 0644, this is something that I found in the process of working on bz 1233003 and should be resolved with commit id '77346f27'.

With respect to the qcow2 file creating using 0666, something is amiss here, but I'm not quite sure what yet. It's a different code path through libvirt which uses umask in order to create the file using specific permissions. For some reason in that root squash environment the umask was ignored when running the execve command which does the qemu-img command to create the file under the uid/gid of qemu:qemu (the default of 0644 was used - hence why the other issue would be 'solved'). With the patch that fixes the creation issue in the root squash env, if I check the permissions of the created file I see they are 0644, but then the code tries as root to set to 0666 which would fail. Not sure why the creation through execve isn't getting the right umask.  Will continue to investigate as the primary focus of this bug.
Comment 2 John Ferlan 2015-11-16 16:23 EST
Created attachment 1095093 [details]
sample code which should show issue
Comment 3 John Ferlan 2015-11-16 17:08:39 EST
Seems as though qemu-img doesn't honor the umask - in fact it seems the various file open/create options use the 0644 mode/protection value. I would think for the most part that would be "intentional" and having to teach qemu about root squash nfs "oddities" is not something I'd expect to happen. So I'll have to make some more adjustments in libvirt.

I dug into the libvirt code and found the place where the execv() is called to run qemu-img in order to create the image using "/usr/bin/qemu-img create -f qcow2 -o compat=0.10 /tmp/netfs-pool/qemu-img-test.img 3953688K" and changed it on the fly to be just "touch /tmp/netfs-pool/qemu-img-test.img".  Doing that resulted in the file be created with the 0666 permissions/mode as desired.  For the qemu-img, only permissions/mode of 0644 (e.g. the default) would work properly.

The code I attached provides an example of the calls made... Comments in the code show how to set up the environment...
Comment 5 John Ferlan 2015-11-19 11:53:24 EST
I posted some patches upstream to address both the setting mode to 0600 (e.g. what turns out to be the default protections) and failure for execv'd process to honor the umask.  Let's see what the upstream community says, see:

http://www.redhat.com/archives/libvir-list/2015-November/msg00593.html
Comment 6 John Ferlan 2015-11-20 17:27:51 EST
Patches pushed upstream

5e3ad0b77598063ff041018c895aa1c3d86bee07
v1.2.21-111-g5e3ad0b

and 

v1.2.21-110-gce6506b
Comment 8 yisun 2016-02-26 03:16:22 EST
Hi John, 
Just wanna confirm with you about the fix.
My understanding is that:

raw volume:
nothing changed. mode=666/644 works, and mode=600 not working as expected. 


qcow2 volume:
1. mode=600 not working as expected. 
2. mode=644 will cause a "touch /tmp/netfs-pool/qemu-img-test.img" failed, so it's still not working.
3. mod=666 works fine now. 


Did I understand this correctly?
Comment 9 John Ferlan 2016-03-16 07:40:24 EDT
My apologies - I forget this needinfo. I tagged it as a todo item, but just never got around to it. To be honest, I've been working in other areas of the code and this root squash stuff isn't fresh in my mind. It's been a few months since I worked on it. The essential piece to understand is that when a mount point is configured with root squash (e.g. via /etc/exports and listing "root_squash" in the attributes, such as my entry "/home/bzs/rootsquash/nfs *(rw,sync,root_squash)"). Additionally the private bz1253609 has more details - it's private for a reason.

There's a lot of elements in play when it comes to NFS root squash and permissions play a large role in whether the NFS Server allows root to access the directory. Most importantly is the pool owner/group and mode bits, but secondarily permissions are set in the volume.xml (owner/group and mode). IIRC what I found was that when libvirt used qemu-img in order to create the volume, the mode bits weren't set correctly (they used the qemu defaults), so the referenced patch 'adjusted' those bits appropriately and of course made sure to open the volume correctly in a root squash environment

It's probably easier to show you what I used to test...

My example root_squash nfs pool has the following setup:

# cat nfs.xml
<pool type='netfs'>
  <name>rootsquash</name>
  <uuid>2b406837-034f-48bb-86b5-59acb458cd13</uuid>
  <capacity unit='bytes'>10435428352</capacity>
  <allocation unit='bytes'>32505856</allocation>
  <available unit='bytes'>10402922496</available>
  <source>
    <host name='localhost'/>
    <dir path='/home/bzs/rootsquash/nfs'/>
    <format type='nfs'/>
  </source>
  <target>
    <path>/tmp/netfs-rootsquash-pool</path>
    <permissions>
      <mode>0755</mode>
      <owner>107</owner>
      <group>107</group>
      <label>unconfined_u:object_r:home_root_t:s0</label>
    </permissions>
  </target>
</pool>

# ls -ald /home/bzs/rootsquash/nfs
drwxr-xr-x. 2 qemu qemu 4096 Mar 16 07:12 /home/bzs/rootsquash/nfs

# exportfs -v
...
/home/bzs/rootsquash/nfs
		<world>(rw,wdelay,root_squash,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
...

# virsh pool-undefine bz1253609
Pool bz1253609 has been undefined

# virsh pool-define nfs.xml
Pool rootsquash defined from nfs.xml

# virsh pool-build rootsquash
Pool rootsquash built

# virsh pool-start rootsquash
Pool rootsquash started

 cat vol.xml
<volume>
    <name>test.img</name>
    <source>
    </source>
    <capacity>4048576000</capacity>
    <allocation>0</allocation>
    <target>
        <format type='raw'/>
        <permissions>
            <mode>0644</mode>
            <owner>107</owner>
            <group>107</group>
            <label>system_u:object_r:nfs_t:s0</label>
        </permissions>
    </target>
</volume>

# virsh vol-create rootsquash vol.xml
Vol test.img created from vol.xml

# virsh vol-list rootsquash
 Name                 Path                                    
------------------------------------------------------------------------------
 test.img             /tmp/netfs-rootsquash-pool/test.img     

# ls -al /tmp/netfs-rootsquash-pool/test.img
-rw-r--r--. 1 qemu qemu 4048576000 Mar 16 07:21 /tmp/netfs-rootsquash-pool/test.img

# cat vol-0600.xml
<volume>
    <name>test-0600.img</name>
    <source>
    </source>
    <capacity>1048576000</capacity>
    <allocation>0</allocation>
    <target>
        <format type='raw'/>
        <permissions>
            <mode>0600</mode>
            <owner>107</owner>
            <group>107</group>
            <label>system_u:object_r:nfs_t:s0</label>
        </permissions>
    </target>
</volume>

# virsh vol-create rootsquash vol-0600.xml
Vol test-0600.img created from vol-0600.xml

# ls -al /tmp/netfs-rootsquash-pool/test-0600.img
-rw-------. 1 qemu qemu 1048576000 Mar 16 07:20 /tmp/netfs-rootsquash-pool/test-0600.img

# virsh vol-list rootsquash
 Name                 Path                                    
------------------------------------------------------------------------------
 test-0600.img        /tmp/netfs-rootsquash-pool/test-0600.img
 test.img             /tmp/netfs-rootsquash-pool/test.img     

# cat vol-0666.xml
<volume>
    <name>test-0666.img</name>
    <source>
    </source>
    <capacity>1048576000</capacity>
    <allocation>0</allocation>
    <target>
        <format type='raw'/>
        <permissions>
            <mode>0666</mode>
            <owner>107</owner>
            <group>107</group>
            <label>system_u:object_r:nfs_t:s0</label>
        </permissions>
    </target>
</volume>

# virsh vol-create rootsquash vol-0666.xml
Vol test-0666.img created from vol-0666.xml

# ls -al /tmp/netfs-rootsquash-pool/test-0666.img
-rw-rw-rw-. 1 qemu qemu 1048576000 Mar 16 07:23 /tmp/netfs-rootsquash-pool/test-0666.img

# virsh vol-list rootsquash
 Name                 Path                                    
------------------------------------------------------------------------------
 test-0600.img        /tmp/netfs-rootsquash-pool/test-0600.img
 test-0666.img        /tmp/netfs-rootsquash-pool/test-0666.img
 test.img             /tmp/netfs-rootsquash-pool/test.img     

# cat qemu-img.xml
<volume>
    <name>qemu-img-test.img</name>
    <source>
    </source>
    <capacity>4048576000</capacity>
    <allocation>0</allocation>
    <target>
        <format type='qcow2'/>
        <permissions>
            <mode>0644</mode>
            <owner>107</owner>
            <group>107</group>
            <label>system_u:object_r:nfs_t:s0</label>
        </permissions>
    </target>
</volume>

# virsh vol-create rootsquash qemu-img.xml
Vol qemu-img-test.img created from qemu-img.xml

# ls -al /tmp/netfs-rootsquash-pool/qemu-img-test.img
-rw-r--r--. 1 qemu qemu 197120 Mar 16 07:26 /tmp/netfs-rootsquash-pool/qemu-img-test.img

# virsh vol-list rootsquash
 Name                 Path                                    
------------------------------------------------------------------------------
 qemu-img-test.img    /tmp/netfs-rootsquash-pool/qemu-img-test.img
 test-0600.img        /tmp/netfs-rootsquash-pool/test-0600.img
 test-0666.img        /tmp/netfs-rootsquash-pool/test-0666.img
 test.img             /tmp/netfs-rootsquash-pool/test.img     

# cat qemu-img-600.xml
<volume>
    <name>qemu-img-test-0600.img</name>
    <source>
    </source>
    <capacity>4048576000</capacity>
    <allocation>0</allocation>
    <target>
        <format type='qcow2'/>
        <permissions>
            <mode>0600</mode>
            <owner>107</owner>
            <group>107</group>
            <label>system_u:object_r:nfs_t:s0</label>
        </permissions>
    </target>
</volume>

# virsh vol-create rootsquash qemu-img-600.xml
Vol qemu-img-test-0600.img created from qemu-img-600.xml

# ls -al /tmp/netfs-rootsquash-pool/qemu-img-test-0600.img
-rw-------. 1 qemu qemu 197120 Mar 16 07:28 /tmp/netfs-rootsquash-pool/qemu-img-test-0600.img

# virsh vol-list rootsquash
 Name                 Path                                    
------------------------------------------------------------------------------
 qemu-img-test-0600.img /tmp/netfs-rootsquash-pool/qemu-img-test-0600.img
 qemu-img-test.img    /tmp/netfs-rootsquash-pool/qemu-img-test.img
 test-0600.img        /tmp/netfs-rootsquash-pool/test-0600.img
 test-0666.img        /tmp/netfs-rootsquash-pool/test-0666.img
 test.img             /tmp/netfs-rootsquash-pool/test.img     

# cat qemu-img-666.xml
<volume>
    <name>qemu-img-test-0666.img</name>
    <source>
    </source>
    <capacity>4048576000</capacity>
    <allocation>0</allocation>
    <target>
        <format type='qcow2'/>
        <permissions>
            <mode>0666</mode>
            <owner>107</owner>
            <group>107</group>
            <label>system_u:object_r:nfs_t:s0</label>
        </permissions>
    </target>
</volume>

# virsh vol-create rootsquash qemu-img-666.xml
Vol qemu-img-test-0666.img created from qemu-img-666.xml

# ls -al /tmp/netfs-rootsquash-pool/qemu-img-test-0666.img
-rw-rw-rw-. 1 qemu qemu 197120 Mar 16 07:28 /tmp/netfs-rootsquash-pool/qemu-img-test-0666.img

# virsh vol-list rootsquash
 Name                 Path                                    
------------------------------------------------------------------------------
 qemu-img-test-0600.img /tmp/netfs-rootsquash-pool/qemu-img-test-0600.img
 qemu-img-test-0666.img /tmp/netfs-rootsquash-pool/qemu-img-test-0666.img
 qemu-img-test.img    /tmp/netfs-rootsquash-pool/qemu-img-test.img
 test-0600.img        /tmp/netfs-rootsquash-pool/test-0600.img
 test-0666.img        /tmp/netfs-rootsquash-pool/test-0666.img
 test.img             /tmp/netfs-rootsquash-pool/test.img     

# ls -al /tmp/netfs-rootsquash-pool/
total 592
drwxr-xr-x.  2 qemu qemu       4096 Mar 16 07:28 .
drwxrwxrwt. 25 root root        540 Mar 16 07:37 ..
-rw-------.  1 qemu qemu     197120 Mar 16 07:28 qemu-img-test-0600.img
-rw-rw-rw-.  1 qemu qemu     197120 Mar 16 07:28 qemu-img-test-0666.img
-rw-r--r--.  1 qemu qemu     197120 Mar 16 07:26 qemu-img-test.img
-rw-------.  1 qemu qemu 1048576000 Mar 16 07:20 test-0600.img
-rw-rw-rw-.  1 qemu qemu 1048576000 Mar 16 07:23 test-0666.img
-rw-r--r--.  1 qemu qemu 4048576000 Mar 16 07:21 test.img
Comment 10 yisun 2016-03-17 06:53:40 EDT
Thx John for the info. I asked about this bug in comment 8 because I used a rhel6.7 to build nfs server and the behaviour was strange. 
Today I used rhel7.3 to build nfs server and everything works fine.

PASSED

Verified on:
Libvirt host:
    libvirt-1.3.2-1.el7.x86_64
    kernel-3.10.0-365.el7.x86_64
NFS server:
    kernel-3.10.0-363.el7.x86_64



Steps as follow:
On NFS Server:
# uname -r
3.10.0-363.el7.x86_64

# cat /etc/exports
/var/lib/libvirt/images/nfs *(rw,sync,root_squash)

# service nfs restart
Redirecting to /bin/systemctl restart  nfs.service




On libvirt host:
#cat nfs.xml
<pool type='netfs'>
<name>nfs</name>
<source>
<host name='NFS_Server_IP'/>
<dir path='/var/lib/libvirt/images/nfs'/>
<format type='nfs'/>
</source>
<target>
<path>/mnt/nfs</path>
<permissions>
<mode>0755</mode>
<owner>107</owner>
<group>107</group>
</permissions>
</target>
</pool>


# virsh pool-define nfs.xml&& virsh pool-build nfs && virsh pool-start nfs
Pool nfs defined from nfs.xml
Pool nfs built
Pool nfs started


# cat 600.raw.vol 
<volume type='file'>
  <name>raw-600</name>
  <capacity unit='bytes'>104857600</capacity>
  <target>
    <format type='raw'/>
<permissions>
<mode>0600</mode>
<owner>107</owner>
<group>107</group>
</permissions>
  </target>
</volume>


# virsh vol-create nfs 600.raw.vol 
Vol raw-600 created from 600.raw.vol


# cat 644.raw.vol 
<volume type='file'>
  <name>raw-644</name>
  <capacity unit='bytes'>104857600</capacity>
  <target>
    <format type='raw'/>
<permissions>
<mode>0644</mode>
<owner>107</owner>
<group>107</group>
</permissions>
  </target>
</volume>


# virsh vol-create nfs 644.raw.vol 
Vol raw-644 created from 644.raw.vol


# cat 666.raw.vol 
<volume type='file'>
  <name>raw-666</name>
  <capacity unit='bytes'>104857600</capacity>
  <target>
    <format type='raw'/>
<permissions>
<mode>0666</mode>
<owner>107</owner>
<group>107</group>
</permissions>
  </target>
</volume>


# virsh vol-create nfs 666.raw.vol 
Vol raw-666 created from 666.raw.vol


# cat 600.qcow2.vol 
<volume type='file'>
  <name>qcow2-600</name>
  <capacity unit='bytes'>104857600</capacity>
  <target>
    <format type='qcow2'/>
<permissions>
<mode>0600</mode>
<owner>107</owner>
<group>107</group>
</permissions>
  </target>
</volume>

# virsh vol-create nfs 600.qcow2.vol 
Vol qcow2-600 created from 600.qcow2.vol


# cat 644.qcow2.vol 
<volume type='file'>
  <name>qcow2-644</name>
  <capacity unit='bytes'>104857600</capacity>
  <target>
    <format type='qcow2'/>
<permissions>
<mode>0644</mode>
<owner>107</owner>
<group>107</group>
</permissions>
  </target>
</volume>


# virsh vol-create nfs 644.qcow2.vol 
Vol qcow2-644 created from 644.qcow2.vol


# cat 666.qcow2.vol 
<volume type='file'>
  <name>qcow2-666</name>
  <capacity unit='bytes'>104857600</capacity>
  <target>
    <format type='qcow2'/>
<permissions>
<mode>0666</mode>
<owner>107</owner>
<group>107</group>
</permissions>
  </target>
</volume>


# virsh vol-create nfs 666.qcow2.vol 
Vol qcow2-666 created from 666.qcow2.vol


# virsh vol-list nfs --details
 Name       Path                Type    Capacity  Allocation
-------------------------------------------------------------
 qcow2-600  /mnt/nfs/qcow2-600  file  100.00 MiB  196.00 KiB
 qcow2-644  /mnt/nfs/qcow2-644  file  100.00 MiB  196.00 KiB
 qcow2-666  /mnt/nfs/qcow2-666  file  100.00 MiB  196.00 KiB
 raw-600    /mnt/nfs/raw-600    file  100.00 MiB  404.00 KiB
 raw-644    /mnt/nfs/raw-644    file  100.00 MiB  404.00 KiB
 raw-666    /mnt/nfs/raw-666    file  100.00 MiB  404.00 KiB


on NFS Server:

# ls -al /var/lib/libvirt/images/nfs/
total 1800
drwxr-xr-x. 2 qemu qemu        96 Mar 17 06:36 .
drwx--x--x. 3 root root        30 Mar  1 07:10 ..
-rw-------. 1 qemu qemu    197120 Mar 17 06:36 qcow2-600
-rw-r--r--. 1 qemu qemu    197120 Mar 17 06:36 qcow2-644
-rw-rw-rw-. 1 qemu qemu    197120 Mar 17 06:36 qcow2-666
-rw-------. 1 qemu qemu 104857600 Mar 17 06:33 raw-600
-rw-r--r--. 1 qemu qemu 104857600 Mar 17 06:34 raw-644
-rw-rw-rw-. 1 qemu qemu 104857600 Mar 17 06:34 raw-666
Comment 12 errata-xmlrpc 2016-11-03 14:30:48 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html

Note You need to log in before you can comment on or make changes to this bug.