Created attachment 1702183 [details] guest xml Description of problem: Permission denied when using 'virt-qemu-run -r' to start guest with luks disk: Unable to read /tmp/test1/lib/qemu/domain-1-vm1/master-key.aes: Failed to open file “/tmp/test1/lib/qemu/domain-1-vm1/master-key.aes”: Permission denied Version-Release number of selected component (if applicable): libvirt-6.5.0-1.el8.x86_64 qemu-kvm-5.0.0-2.module+el8.3.0+7379+0505d6ca.x86_64 How reproducible: 100% Steps to Reproduce: 1. prepare a qcow2 luks image: #qemu-img create --object secret,id=sec0,data=123456,for‐ -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 /var/lib/libvirt/images/luks.qcow2 1G 2.Check the image: # qemu-img check --object secret,id=sec0,data=123456 --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=/var/lib/libvirt/images/luks.qcow2 No errors were found on the image. Image end offset: 2359296 3.Prepare secret xml and secret value file: #cat /xml/secret.xml <secret ephemeral='no' private='yes'> <description>LUKS Sample Secret</description> <uuid>f981dd17-143f-45bc-88e6-ed1fe20ce9da</uuid> <usage type='volume'> <volume>/var/lib/libvirt/images/luks.img</volume> </usage> </secret> #cat /xml/secret-value 123456 4.Prepare a guest xml with the luks image(the same uuid with the secret xml): #cat /tmp/vm1.xml ... <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='threads' copy_on_read='off'/> <source file='/var/lib/libvirt/images/luks.qcow2' index='1'> <encryption format='luks'> <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/> </encryption> </source> ... </disk> ... 5.Start guest with "virt-qemu-run -r": # virt-qemu-run -s /xml/secret.xml,/xml/value -d -v -r /tmp/test1 /tmp/vm1.xml virt-qemu-run: 84: initializing libvirt virt-qemu-run: 1157: initializing signal handlers virt-qemu-run: 1174: preparing event loop thread virt-qemu-run: 1223: opening secret:///embed?root=/tmp/test1 virt-qemu-run: 2126: loading secret /xml/secret.xml and /xml/value virt-qemu-run: 3405: opening qemu:///embed?root=/tmp/test1 virt-qemu-run: 51628: starting guest /tmp/vm1.xml virt-qemu-run: cannot start VM: internal error: process exited while connecting to monitor: 2020-07-23T08:22:08.722251Z qemu-kvm: -object secret,id=masterKey0,format=raw,file=/tmp/test1/lib/qemu/domain-1-vm1/master-key.aes: Unable to read /tmp/test1/lib/qemu/domain-1-vm1/master-key.aes: Failed to open file “/tmp/test1/lib/qemu/domain-1-vm1/master-key.aes”: Permission denied virt-qemu-run: 1088346: cleaned up, exiting Actual results: Permission denied to read master-key.aes when using 'virt-qemu-run -r' to start guest with luks disk. Expected results: Can start guest successfully. Additonal info: 1.Can start guest successfully if without specifying root dir; 2.If not reproduce, just reproduce step5 with different directory, such as: # virt-qemu-run -s /xml/secret.xml,/xml/value -d -v -r /tmp/test1 /tmp/vm2.xml
Do you have SELinux enabled, and does the problem go away if set to permissive mode instead of enforcing.bb
(In reply to Daniel Berrangé from comment #1) > Do you have SELinux enabled, and does the problem go away if set to > permissive mode instead of enforcing.bb Yes, I set selinux to permissive mode. Sorry for forgetting to add it in the comment 0. It's not related with selinux. Can always start guest successfully if not specifying root dir.
/tmp is slightly special as a filesystem, could you retry with some other dir like /srv/embed or $HOME/embed
(In reply to Daniel Berrangé from comment #4) > /tmp is slightly special as a filesystem, could you retry with some other > dir like /srv/embed or $HOME/embed The issue also happens with dir under 'srv': #setenforce 0 # virt-qemu-run -s /xml/secret.xml,/xml/value -d -v /scripts/fs-1.xml -r /srv/embed virt-qemu-run: 77: initializing libvirt virt-qemu-run: 1710: initializing signal handlers virt-qemu-run: 1719: preparing event loop thread virt-qemu-run: 1782: opening secret:///embed?root=/srv/embed virt-qemu-run: 2801: loading secret /xml/secret.xml and /xml/value virt-qemu-run: 3429: opening qemu:///embed?root=/srv/embed virt-qemu-run: 48360: starting guest /scripts/fs-1.xml virt-qemu-run: cannot start VM: Cannot open log file: '/srv/embed/log/qemu/fs-fs0-virtiofsd.log': Device or resource busy virt-qemu-run: 213619: cleaned up, exiting [root@dell-per440-14 ~]# pkill -9 virtiofsd [root@dell-per440-14 ~]# virt-qemu-run -s /xml/secret.xml,/xml/value -d -v /scripts/fs-1.xml -r /srv/embed virt-qemu-run: 38: initializing libvirt virt-qemu-run: 1730: initializing signal handlers virt-qemu-run: 1740: preparing event loop thread virt-qemu-run: 1803: opening secret:///embed?root=/srv/embed virt-qemu-run: 2767: loading secret /xml/secret.xml and /xml/value virt-qemu-run: 3626: opening qemu:///embed?root=/srv/embed virt-qemu-run: 55429: starting guest /scripts/fs-1.xml 2020-08-10 09:10:07.882+0000: 96389: info : libvirt version: 6.6.0, package: 2.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-08-05-16:52:08, ) 2020-08-10 09:10:07.882+0000: 96389: info : hostname: dell-per440-14.lab.eng.pek2.redhat.com 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope//emulator (16) 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope/ (16) 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope//emulator (16) 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope/ (16) 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpuset/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope//emulator (16) 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpuset/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope/ (16) virt-qemu-run: cannot start VM: internal error: process exited while connecting to monitor: 2020-08-10T09:10:07.681158Z qemu-kvm: -object secret,id=masterKey0,format=raw,file=/srv/embed/lib/qemu/domain-1-fs/master-key.aes: Unable to read /srv/embed/lib/qemu/domain-1-fs/master-key.aes: Failed to open file “/srv/embed/lib/qemu/domain-1-fs/master-key.aes”: Permission denied virt-qemu-run: 812829: cleaned up, exiting
I've debugged this problem and found that the root cause is that if root dir does not exist, then the secret driver creates it (indirectly, because it's trying to mkdir() its own configDir and it constructs all parent dirs if missing), but it creates it with mode=S_IRWXU (0700) which means nobody else can access anything in the directory even if they know the path. And that's exactly what qemu is trying to do. Patch coming up shortly.
Patches posted upstream: https://listman.redhat.com/archives/libvir-list/2021-March/msg00010.html
Fixed upstream as: 76f3b2988b qemu_shim: Always pre-create root dir v7.1.0-196-g76f3b2988b
Work for me on libvirt v7.1.0-291-g9eb7e9e817 qemu-5.1.0-9.fc33.x86_64: 1. Set selinux to permissive # setenforce 0 2. Make sure virtlogd is running # systemctl status virtlogd ● virtlogd.service - Virtual machine log manager Loaded: loaded (/usr/lib/systemd/system/virtlogd.service; indirect; vendor preset: disabled) Active: active (running) since Tue 2021-03-23 08:26:46 UTC; 9min ago 3. Test as comment0 ➜ ~ virt-qemu-run -s /tmp/secret.xml,/tmp/value -d -v -r /tmp/test1 /tmp/hhan.xml virt-qemu-run: 71: initializing libvirt 131326 virt-qemu-run: 1433: initializing signal handlers virt-qemu-run: 1559: preparing event loop thread virt-qemu-run: 1649: opening secret:///embed?root=%2Ftmp%2Ftest1 virt-qemu-run: 2804: loading secret /tmp/secret.xml and /tmp/value virt-qemu-run: 2991: opening qemu:///embed?root=%2Ftmp%2Ftest1 virt-qemu-run: 14591: fetching guest config /tmp/hhan.xml virt-qemu-run: 14623: starting guest /tmp/hhan.xml 2021-03-23 08:30:28.434+0000: 131326: info : libvirt version: 7.2.0, package: 1.fc33 (Unknown, 2021-03-23-07:32:16, hhan-fedora) 2021-03-23 08:30:28.434+0000: 131326: info : hostname: hhan-fedora 2021-03-23 08:30:28.434+0000: 131326: warning : qemuSetupDevicesCgroup:760 : Group devices ACL is not accessible, disabling filtering virt-qemu-run: 3271144: guest running, Ctrl-C to stop now ... <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/hhan.qcow2' index='2'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='threads' copy_on_read='off'/> <source file='/var/lib/libvirt/images/luks.qcow2' index='1'> <encryption format='luks'> <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/> </encryption> </source> <backingStore/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> ...
BTW, test with a missing root dir
Test as comment10 on libvirt-7.3.0-1.module+el8.5.0+11004+f4810536.x86_64 qemu-kvm-6.0.0-16.module+el8.5.0+10848+2dccc46d.x86_64. PASS
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4684