Bug 1282288
| Summary: | Unable to set permission when a volume is created in root squash netfs pool | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Yang Yang <yanyang> | ||||
| Component: | libvirt | Assignee: | John Ferlan <jferlan> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.2 | CC: | dyuan, jferlan, mzhan, rbalakri, xuzhang, yisun | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | libvirt-1.3.1-1.el7 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-11-03 18:30:48 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
A bz with multiple things going on - need to separatef raw and qcow2 issues! First, not sure I'd expect setting the 'raw' image to 0600 to succeed because of the root-squash environment there's no way for "root" to *read* information about the raw volume. Setting it to 666 does work though. Remember in the root squash environment, the file would be created under qemu:qemu with prots 0600, then when root then goes to read the file it cannot do so. It's a very restrictive environment. The code to get the created files' information doesn't do so under the uid/gid of what's defined in the pool. I'm not convinced it should, but if it were to do so would require some effort to do so. With respect to the qcow2 file creation using 0644, this is something that I found in the process of working on bz 1233003 and should be resolved with commit id '77346f27'. With respect to the qcow2 file creating using 0666, something is amiss here, but I'm not quite sure what yet. It's a different code path through libvirt which uses umask in order to create the file using specific permissions. For some reason in that root squash environment the umask was ignored when running the execve command which does the qemu-img command to create the file under the uid/gid of qemu:qemu (the default of 0644 was used - hence why the other issue would be 'solved'). With the patch that fixes the creation issue in the root squash env, if I check the permissions of the created file I see they are 0644, but then the code tries as root to set to 0666 which would fail. Not sure why the creation through execve isn't getting the right umask. Will continue to investigate as the primary focus of this bug. Created attachment 1095093 [details]
sample code which should show issue
Seems as though qemu-img doesn't honor the umask - in fact it seems the various file open/create options use the 0644 mode/protection value. I would think for the most part that would be "intentional" and having to teach qemu about root squash nfs "oddities" is not something I'd expect to happen. So I'll have to make some more adjustments in libvirt. I dug into the libvirt code and found the place where the execv() is called to run qemu-img in order to create the image using "/usr/bin/qemu-img create -f qcow2 -o compat=0.10 /tmp/netfs-pool/qemu-img-test.img 3953688K" and changed it on the fly to be just "touch /tmp/netfs-pool/qemu-img-test.img". Doing that resulted in the file be created with the 0666 permissions/mode as desired. For the qemu-img, only permissions/mode of 0644 (e.g. the default) would work properly. The code I attached provides an example of the calls made... Comments in the code show how to set up the environment... I posted some patches upstream to address both the setting mode to 0600 (e.g. what turns out to be the default protections) and failure for execv'd process to honor the umask. Let's see what the upstream community says, see: http://www.redhat.com/archives/libvir-list/2015-November/msg00593.html Patches pushed upstream 5e3ad0b77598063ff041018c895aa1c3d86bee07 v1.2.21-111-g5e3ad0b and v1.2.21-110-gce6506b Hi John, Just wanna confirm with you about the fix. My understanding is that: raw volume: nothing changed. mode=666/644 works, and mode=600 not working as expected. qcow2 volume: 1. mode=600 not working as expected. 2. mode=644 will cause a "touch /tmp/netfs-pool/qemu-img-test.img" failed, so it's still not working. 3. mod=666 works fine now. Did I understand this correctly? My apologies - I forget this needinfo. I tagged it as a todo item, but just never got around to it. To be honest, I've been working in other areas of the code and this root squash stuff isn't fresh in my mind. It's been a few months since I worked on it. The essential piece to understand is that when a mount point is configured with root squash (e.g. via /etc/exports and listing "root_squash" in the attributes, such as my entry "/home/bzs/rootsquash/nfs *(rw,sync,root_squash)"). Additionally the private bz1253609 has more details - it's private for a reason. There's a lot of elements in play when it comes to NFS root squash and permissions play a large role in whether the NFS Server allows root to access the directory. Most importantly is the pool owner/group and mode bits, but secondarily permissions are set in the volume.xml (owner/group and mode). IIRC what I found was that when libvirt used qemu-img in order to create the volume, the mode bits weren't set correctly (they used the qemu defaults), so the referenced patch 'adjusted' those bits appropriately and of course made sure to open the volume correctly in a root squash environment It's probably easier to show you what I used to test... My example root_squash nfs pool has the following setup: # cat nfs.xml <pool type='netfs'> <name>rootsquash</name> <uuid>2b406837-034f-48bb-86b5-59acb458cd13</uuid> <capacity unit='bytes'>10435428352</capacity> <allocation unit='bytes'>32505856</allocation> <available unit='bytes'>10402922496</available> <source> <host name='localhost'/> <dir path='/home/bzs/rootsquash/nfs'/> <format type='nfs'/> </source> <target> <path>/tmp/netfs-rootsquash-pool</path> <permissions> <mode>0755</mode> <owner>107</owner> <group>107</group> <label>unconfined_u:object_r:home_root_t:s0</label> </permissions> </target> </pool> # ls -ald /home/bzs/rootsquash/nfs drwxr-xr-x. 2 qemu qemu 4096 Mar 16 07:12 /home/bzs/rootsquash/nfs # exportfs -v ... /home/bzs/rootsquash/nfs <world>(rw,wdelay,root_squash,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash) ... # virsh pool-undefine bz1253609 Pool bz1253609 has been undefined # virsh pool-define nfs.xml Pool rootsquash defined from nfs.xml # virsh pool-build rootsquash Pool rootsquash built # virsh pool-start rootsquash Pool rootsquash started cat vol.xml <volume> <name>test.img</name> <source> </source> <capacity>4048576000</capacity> <allocation>0</allocation> <target> <format type='raw'/> <permissions> <mode>0644</mode> <owner>107</owner> <group>107</group> <label>system_u:object_r:nfs_t:s0</label> </permissions> </target> </volume> # virsh vol-create rootsquash vol.xml Vol test.img created from vol.xml # virsh vol-list rootsquash Name Path ------------------------------------------------------------------------------ test.img /tmp/netfs-rootsquash-pool/test.img # ls -al /tmp/netfs-rootsquash-pool/test.img -rw-r--r--. 1 qemu qemu 4048576000 Mar 16 07:21 /tmp/netfs-rootsquash-pool/test.img # cat vol-0600.xml <volume> <name>test-0600.img</name> <source> </source> <capacity>1048576000</capacity> <allocation>0</allocation> <target> <format type='raw'/> <permissions> <mode>0600</mode> <owner>107</owner> <group>107</group> <label>system_u:object_r:nfs_t:s0</label> </permissions> </target> </volume> # virsh vol-create rootsquash vol-0600.xml Vol test-0600.img created from vol-0600.xml # ls -al /tmp/netfs-rootsquash-pool/test-0600.img -rw-------. 1 qemu qemu 1048576000 Mar 16 07:20 /tmp/netfs-rootsquash-pool/test-0600.img # virsh vol-list rootsquash Name Path ------------------------------------------------------------------------------ test-0600.img /tmp/netfs-rootsquash-pool/test-0600.img test.img /tmp/netfs-rootsquash-pool/test.img # cat vol-0666.xml <volume> <name>test-0666.img</name> <source> </source> <capacity>1048576000</capacity> <allocation>0</allocation> <target> <format type='raw'/> <permissions> <mode>0666</mode> <owner>107</owner> <group>107</group> <label>system_u:object_r:nfs_t:s0</label> </permissions> </target> </volume> # virsh vol-create rootsquash vol-0666.xml Vol test-0666.img created from vol-0666.xml # ls -al /tmp/netfs-rootsquash-pool/test-0666.img -rw-rw-rw-. 1 qemu qemu 1048576000 Mar 16 07:23 /tmp/netfs-rootsquash-pool/test-0666.img # virsh vol-list rootsquash Name Path ------------------------------------------------------------------------------ test-0600.img /tmp/netfs-rootsquash-pool/test-0600.img test-0666.img /tmp/netfs-rootsquash-pool/test-0666.img test.img /tmp/netfs-rootsquash-pool/test.img # cat qemu-img.xml <volume> <name>qemu-img-test.img</name> <source> </source> <capacity>4048576000</capacity> <allocation>0</allocation> <target> <format type='qcow2'/> <permissions> <mode>0644</mode> <owner>107</owner> <group>107</group> <label>system_u:object_r:nfs_t:s0</label> </permissions> </target> </volume> # virsh vol-create rootsquash qemu-img.xml Vol qemu-img-test.img created from qemu-img.xml # ls -al /tmp/netfs-rootsquash-pool/qemu-img-test.img -rw-r--r--. 1 qemu qemu 197120 Mar 16 07:26 /tmp/netfs-rootsquash-pool/qemu-img-test.img # virsh vol-list rootsquash Name Path ------------------------------------------------------------------------------ qemu-img-test.img /tmp/netfs-rootsquash-pool/qemu-img-test.img test-0600.img /tmp/netfs-rootsquash-pool/test-0600.img test-0666.img /tmp/netfs-rootsquash-pool/test-0666.img test.img /tmp/netfs-rootsquash-pool/test.img # cat qemu-img-600.xml <volume> <name>qemu-img-test-0600.img</name> <source> </source> <capacity>4048576000</capacity> <allocation>0</allocation> <target> <format type='qcow2'/> <permissions> <mode>0600</mode> <owner>107</owner> <group>107</group> <label>system_u:object_r:nfs_t:s0</label> </permissions> </target> </volume> # virsh vol-create rootsquash qemu-img-600.xml Vol qemu-img-test-0600.img created from qemu-img-600.xml # ls -al /tmp/netfs-rootsquash-pool/qemu-img-test-0600.img -rw-------. 1 qemu qemu 197120 Mar 16 07:28 /tmp/netfs-rootsquash-pool/qemu-img-test-0600.img # virsh vol-list rootsquash Name Path ------------------------------------------------------------------------------ qemu-img-test-0600.img /tmp/netfs-rootsquash-pool/qemu-img-test-0600.img qemu-img-test.img /tmp/netfs-rootsquash-pool/qemu-img-test.img test-0600.img /tmp/netfs-rootsquash-pool/test-0600.img test-0666.img /tmp/netfs-rootsquash-pool/test-0666.img test.img /tmp/netfs-rootsquash-pool/test.img # cat qemu-img-666.xml <volume> <name>qemu-img-test-0666.img</name> <source> </source> <capacity>4048576000</capacity> <allocation>0</allocation> <target> <format type='qcow2'/> <permissions> <mode>0666</mode> <owner>107</owner> <group>107</group> <label>system_u:object_r:nfs_t:s0</label> </permissions> </target> </volume> # virsh vol-create rootsquash qemu-img-666.xml Vol qemu-img-test-0666.img created from qemu-img-666.xml # ls -al /tmp/netfs-rootsquash-pool/qemu-img-test-0666.img -rw-rw-rw-. 1 qemu qemu 197120 Mar 16 07:28 /tmp/netfs-rootsquash-pool/qemu-img-test-0666.img # virsh vol-list rootsquash Name Path ------------------------------------------------------------------------------ qemu-img-test-0600.img /tmp/netfs-rootsquash-pool/qemu-img-test-0600.img qemu-img-test-0666.img /tmp/netfs-rootsquash-pool/qemu-img-test-0666.img qemu-img-test.img /tmp/netfs-rootsquash-pool/qemu-img-test.img test-0600.img /tmp/netfs-rootsquash-pool/test-0600.img test-0666.img /tmp/netfs-rootsquash-pool/test-0666.img test.img /tmp/netfs-rootsquash-pool/test.img # ls -al /tmp/netfs-rootsquash-pool/ total 592 drwxr-xr-x. 2 qemu qemu 4096 Mar 16 07:28 . drwxrwxrwt. 25 root root 540 Mar 16 07:37 .. -rw-------. 1 qemu qemu 197120 Mar 16 07:28 qemu-img-test-0600.img -rw-rw-rw-. 1 qemu qemu 197120 Mar 16 07:28 qemu-img-test-0666.img -rw-r--r--. 1 qemu qemu 197120 Mar 16 07:26 qemu-img-test.img -rw-------. 1 qemu qemu 1048576000 Mar 16 07:20 test-0600.img -rw-rw-rw-. 1 qemu qemu 1048576000 Mar 16 07:23 test-0666.img -rw-r--r--. 1 qemu qemu 4048576000 Mar 16 07:21 test.img Thx John for the info. I asked about this bug in comment 8 because I used a rhel6.7 to build nfs server and the behaviour was strange. Today I used rhel7.3 to build nfs server and everything works fine. PASSED Verified on: Libvirt host: libvirt-1.3.2-1.el7.x86_64 kernel-3.10.0-365.el7.x86_64 NFS server: kernel-3.10.0-363.el7.x86_64 Steps as follow: On NFS Server: # uname -r 3.10.0-363.el7.x86_64 # cat /etc/exports /var/lib/libvirt/images/nfs *(rw,sync,root_squash) # service nfs restart Redirecting to /bin/systemctl restart nfs.service On libvirt host: #cat nfs.xml <pool type='netfs'> <name>nfs</name> <source> <host name='NFS_Server_IP'/> <dir path='/var/lib/libvirt/images/nfs'/> <format type='nfs'/> </source> <target> <path>/mnt/nfs</path> <permissions> <mode>0755</mode> <owner>107</owner> <group>107</group> </permissions> </target> </pool> # virsh pool-define nfs.xml&& virsh pool-build nfs && virsh pool-start nfs Pool nfs defined from nfs.xml Pool nfs built Pool nfs started # cat 600.raw.vol <volume type='file'> <name>raw-600</name> <capacity unit='bytes'>104857600</capacity> <target> <format type='raw'/> <permissions> <mode>0600</mode> <owner>107</owner> <group>107</group> </permissions> </target> </volume> # virsh vol-create nfs 600.raw.vol Vol raw-600 created from 600.raw.vol # cat 644.raw.vol <volume type='file'> <name>raw-644</name> <capacity unit='bytes'>104857600</capacity> <target> <format type='raw'/> <permissions> <mode>0644</mode> <owner>107</owner> <group>107</group> </permissions> </target> </volume> # virsh vol-create nfs 644.raw.vol Vol raw-644 created from 644.raw.vol # cat 666.raw.vol <volume type='file'> <name>raw-666</name> <capacity unit='bytes'>104857600</capacity> <target> <format type='raw'/> <permissions> <mode>0666</mode> <owner>107</owner> <group>107</group> </permissions> </target> </volume> # virsh vol-create nfs 666.raw.vol Vol raw-666 created from 666.raw.vol # cat 600.qcow2.vol <volume type='file'> <name>qcow2-600</name> <capacity unit='bytes'>104857600</capacity> <target> <format type='qcow2'/> <permissions> <mode>0600</mode> <owner>107</owner> <group>107</group> </permissions> </target> </volume> # virsh vol-create nfs 600.qcow2.vol Vol qcow2-600 created from 600.qcow2.vol # cat 644.qcow2.vol <volume type='file'> <name>qcow2-644</name> <capacity unit='bytes'>104857600</capacity> <target> <format type='qcow2'/> <permissions> <mode>0644</mode> <owner>107</owner> <group>107</group> </permissions> </target> </volume> # virsh vol-create nfs 644.qcow2.vol Vol qcow2-644 created from 644.qcow2.vol # cat 666.qcow2.vol <volume type='file'> <name>qcow2-666</name> <capacity unit='bytes'>104857600</capacity> <target> <format type='qcow2'/> <permissions> <mode>0666</mode> <owner>107</owner> <group>107</group> </permissions> </target> </volume> # virsh vol-create nfs 666.qcow2.vol Vol qcow2-666 created from 666.qcow2.vol # virsh vol-list nfs --details Name Path Type Capacity Allocation ------------------------------------------------------------- qcow2-600 /mnt/nfs/qcow2-600 file 100.00 MiB 196.00 KiB qcow2-644 /mnt/nfs/qcow2-644 file 100.00 MiB 196.00 KiB qcow2-666 /mnt/nfs/qcow2-666 file 100.00 MiB 196.00 KiB raw-600 /mnt/nfs/raw-600 file 100.00 MiB 404.00 KiB raw-644 /mnt/nfs/raw-644 file 100.00 MiB 404.00 KiB raw-666 /mnt/nfs/raw-666 file 100.00 MiB 404.00 KiB on NFS Server: # ls -al /var/lib/libvirt/images/nfs/ total 1800 drwxr-xr-x. 2 qemu qemu 96 Mar 17 06:36 . drwx--x--x. 3 root root 30 Mar 1 07:10 .. -rw-------. 1 qemu qemu 197120 Mar 17 06:36 qcow2-600 -rw-r--r--. 1 qemu qemu 197120 Mar 17 06:36 qcow2-644 -rw-rw-rw-. 1 qemu qemu 197120 Mar 17 06:36 qcow2-666 -rw-------. 1 qemu qemu 104857600 Mar 17 06:33 raw-600 -rw-r--r--. 1 qemu qemu 104857600 Mar 17 06:34 raw-644 -rw-rw-rw-. 1 qemu qemu 104857600 Mar 17 06:34 raw-666 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-2577.html |
Description of problem: Given a root squash netfs pool, if volume target mode is changed to 0600, raw volume cannot be created successfully with "permission denied" error. But changed mode to 0604, it can be created successfully. But for qcow2 volume, no matter what mode provided, volume are always not permitted to create. Version-Release number of selected component (if applicable): libvirt-1.2.17-13.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.Setup a nfs server with root_squash # cat /etc/exports /var/lib/libvirt/images/nfs *(rw,sync,root_squash) Change the nfs dir owner to qemu, and chmod to 755. # chown qemu.qemu /var/lib/libvirt/images/nfs \ && chmod 755 /var/lib/libvirt/images/nfs 2. Create nfs pool with qemu user and group. # cat nfs.xml <pool type='netfs'> <name>nfs</name> <source> <host name='10.66.x.x'/> <dir path='/var/lib/libvirt/images/nfs'/> <format type='nfs'/> </source> <target> <path>/mnt/nfs</path> <permissions> <mode>0755</mode> <owner>107</owner> <group>107</group> </permissions> </target> # virsh pool-define nfs.xml&& virsh pool-build nfs && virsh pool-start nfs Pool nfs defined from nfs.xml Pool nfs built Pool nfs started 3. create raw volume with 644 mode successfully # cat nfs.vol.xml <volume type='file'> <name>test.raw</name> <capacity unit='bytes'>1048576000</capacity> <target> <format type='raw'/> <permissions> <mode>0644</mode> <owner>107</owner> <group>107</group> </permissions> </target> </volume> # virsh vol-create nfs nfs.vol.xml Vol test.raw created from nfs.vol.xml # virsh vol-list nfs | grep test.raw test.raw /mnt/nfs/test.raw 4. change mode to 600, raw volume failed to create # cat nfs.vol.xml <volume type='file'> <name>test.raw</name> <capacity unit='bytes'>1048576000</capacity> <target> <format type='raw'/> <permissions> <mode>0600</mode> <owner>107</owner> <group>107</group> </permissions> </target> </volume> # virsh vol-create nfs nfs.vol.xml error: Failed to create vol from nfs.vol.xml error: cannot open volume '/mnt/nfs/test.raw': Permission denied [root@rhel7_test yy]# virsh vol-list nfs | grep test.raw [root@rhel7_test yy]# ll /mnt/nfs/test.raw ls: cannot access /mnt/nfs/test.raw: No such file or directory 5. create qcow2 volume with 644 mode, failed # cat nfs.vol.xml <volume type='file'> <name>test.qcow2</name> <capacity unit='bytes'>1048576000</capacity> <target> <format type='qcow2'/> <permissions> <mode>0644</mode> <owner>107</owner> <group>107</group> </permissions> </target> </volume> # virsh vol-create nfs nfs.vol.xml error: Failed to create vol from nfs.vol.xml error: cannot set mode of '/mnt/nfs/test.qcow2' to 0644: Operation not permitted 6. change mode to 666, failed to create volume # cat nfs.vol.xml <volume type='file'> <name>test.qcow2</name> <capacity unit='bytes'>1048576000</capacity> <target> <format type='qcow2'/> <permissions> <mode>0666</mode> <owner>107</owner> <group>107</group> </permissions> </target> </volume> # virsh vol-create nfs nfs.vol.xml error: Failed to create vol from nfs.vol.xml error: cannot set mode of '/mnt/nfs/test.qcow2' to 0666: Operation not permitted Actual results: Expected results: Raw volume with 600 mode should be created without error Qcow2 volume should be created without error Additional info: