Bug 1859873
| Summary: | Permission denied to read master-key.aes when using 'virt-qemu-run -r ' with a non-exist root dir | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | yafu <yafu> | ||||
| Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Han Han <hhan> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 8.3 | CC: | berrange, chhu, dyuan, jdenemar, jsuchane, lmen, mprivozn, virt-maint, xuzhang | ||||
| Target Milestone: | rc | Keywords: | Triaged, Upstream | ||||
| Target Release: | 8.3 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | libvirt-7.3.0-1.el8 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-11-16 07:50:50 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | 7.2.0 | ||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Do you have SELinux enabled, and does the problem go away if set to permissive mode instead of enforcing.bb (In reply to Daniel Berrangé from comment #1) > Do you have SELinux enabled, and does the problem go away if set to > permissive mode instead of enforcing.bb Yes, I set selinux to permissive mode. Sorry for forgetting to add it in the comment 0. It's not related with selinux. Can always start guest successfully if not specifying root dir. /tmp is slightly special as a filesystem, could you retry with some other dir like /srv/embed or $HOME/embed (In reply to Daniel Berrangé from comment #4) > /tmp is slightly special as a filesystem, could you retry with some other > dir like /srv/embed or $HOME/embed The issue also happens with dir under 'srv': #setenforce 0 # virt-qemu-run -s /xml/secret.xml,/xml/value -d -v /scripts/fs-1.xml -r /srv/embed virt-qemu-run: 77: initializing libvirt virt-qemu-run: 1710: initializing signal handlers virt-qemu-run: 1719: preparing event loop thread virt-qemu-run: 1782: opening secret:///embed?root=/srv/embed virt-qemu-run: 2801: loading secret /xml/secret.xml and /xml/value virt-qemu-run: 3429: opening qemu:///embed?root=/srv/embed virt-qemu-run: 48360: starting guest /scripts/fs-1.xml virt-qemu-run: cannot start VM: Cannot open log file: '/srv/embed/log/qemu/fs-fs0-virtiofsd.log': Device or resource busy virt-qemu-run: 213619: cleaned up, exiting [root@dell-per440-14 ~]# pkill -9 virtiofsd [root@dell-per440-14 ~]# virt-qemu-run -s /xml/secret.xml,/xml/value -d -v /scripts/fs-1.xml -r /srv/embed virt-qemu-run: 38: initializing libvirt virt-qemu-run: 1730: initializing signal handlers virt-qemu-run: 1740: preparing event loop thread virt-qemu-run: 1803: opening secret:///embed?root=/srv/embed virt-qemu-run: 2767: loading secret /xml/secret.xml and /xml/value virt-qemu-run: 3626: opening qemu:///embed?root=/srv/embed virt-qemu-run: 55429: starting guest /scripts/fs-1.xml 2020-08-10 09:10:07.882+0000: 96389: info : libvirt version: 6.6.0, package: 2.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-08-05-16:52:08, ) 2020-08-10 09:10:07.882+0000: 96389: info : hostname: dell-per440-14.lab.eng.pek2.redhat.com 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope//emulator (16) 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope/ (16) 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope//emulator (16) 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope/ (16) 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpuset/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope//emulator (16) 2020-08-10 09:10:07.882+0000: 96389: error : virCgroupRemoveRecursively:2360 : Unable to remove /sys/fs/cgroup/cpuset/machine.slice/machine-test.slice/machine-qemu\x2dembed\x2d0ae5c50b\x2d1\x2dfs.scope/ (16) virt-qemu-run: cannot start VM: internal error: process exited while connecting to monitor: 2020-08-10T09:10:07.681158Z qemu-kvm: -object secret,id=masterKey0,format=raw,file=/srv/embed/lib/qemu/domain-1-fs/master-key.aes: Unable to read /srv/embed/lib/qemu/domain-1-fs/master-key.aes: Failed to open file “/srv/embed/lib/qemu/domain-1-fs/master-key.aes”: Permission denied virt-qemu-run: 812829: cleaned up, exiting I've debugged this problem and found that the root cause is that if root dir does not exist, then the secret driver creates it (indirectly, because it's trying to mkdir() its own configDir and it constructs all parent dirs if missing), but it creates it with mode=S_IRWXU (0700) which means nobody else can access anything in the directory even if they know the path. And that's exactly what qemu is trying to do. Patch coming up shortly. Patches posted upstream: https://listman.redhat.com/archives/libvir-list/2021-March/msg00010.html Fixed upstream as: 76f3b2988b qemu_shim: Always pre-create root dir v7.1.0-196-g76f3b2988b Work for me on libvirt v7.1.0-291-g9eb7e9e817 qemu-5.1.0-9.fc33.x86_64:
1. Set selinux to permissive
# setenforce 0
2. Make sure virtlogd is running
# systemctl status virtlogd
● virtlogd.service - Virtual machine log manager
Loaded: loaded (/usr/lib/systemd/system/virtlogd.service; indirect; vendor preset: disabled)
Active: active (running) since Tue 2021-03-23 08:26:46 UTC; 9min ago
3. Test as comment0
➜ ~ virt-qemu-run -s /tmp/secret.xml,/tmp/value -d -v -r /tmp/test1 /tmp/hhan.xml
virt-qemu-run: 71: initializing libvirt 131326
virt-qemu-run: 1433: initializing signal handlers
virt-qemu-run: 1559: preparing event loop thread
virt-qemu-run: 1649: opening secret:///embed?root=%2Ftmp%2Ftest1
virt-qemu-run: 2804: loading secret /tmp/secret.xml and /tmp/value
virt-qemu-run: 2991: opening qemu:///embed?root=%2Ftmp%2Ftest1
virt-qemu-run: 14591: fetching guest config /tmp/hhan.xml
virt-qemu-run: 14623: starting guest /tmp/hhan.xml
2021-03-23 08:30:28.434+0000: 131326: info : libvirt version: 7.2.0, package: 1.fc33 (Unknown, 2021-03-23-07:32:16, hhan-fedora)
2021-03-23 08:30:28.434+0000: 131326: info : hostname: hhan-fedora
2021-03-23 08:30:28.434+0000: 131326: warning : qemuSetupDevicesCgroup:760 : Group devices ACL is not accessible, disabling filtering
virt-qemu-run: 3271144: guest running, Ctrl-C to stop now
...
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/hhan.qcow2' index='2'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none' io='threads' copy_on_read='off'/>
<source file='/var/lib/libvirt/images/luks.qcow2' index='1'>
<encryption format='luks'>
<secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/>
</encryption>
</source>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk>
...
BTW, test with a missing root dir Test as comment10 on libvirt-7.3.0-1.module+el8.5.0+11004+f4810536.x86_64 qemu-kvm-6.0.0-16.module+el8.5.0+10848+2dccc46d.x86_64. PASS Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4684 |
Created attachment 1702183 [details] guest xml Description of problem: Permission denied when using 'virt-qemu-run -r' to start guest with luks disk: Unable to read /tmp/test1/lib/qemu/domain-1-vm1/master-key.aes: Failed to open file “/tmp/test1/lib/qemu/domain-1-vm1/master-key.aes”: Permission denied Version-Release number of selected component (if applicable): libvirt-6.5.0-1.el8.x86_64 qemu-kvm-5.0.0-2.module+el8.3.0+7379+0505d6ca.x86_64 How reproducible: 100% Steps to Reproduce: 1. prepare a qcow2 luks image: #qemu-img create --object secret,id=sec0,data=123456,for‐ -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 /var/lib/libvirt/images/luks.qcow2 1G 2.Check the image: # qemu-img check --object secret,id=sec0,data=123456 --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=/var/lib/libvirt/images/luks.qcow2 No errors were found on the image. Image end offset: 2359296 3.Prepare secret xml and secret value file: #cat /xml/secret.xml <secret ephemeral='no' private='yes'> <description>LUKS Sample Secret</description> <uuid>f981dd17-143f-45bc-88e6-ed1fe20ce9da</uuid> <usage type='volume'> <volume>/var/lib/libvirt/images/luks.img</volume> </usage> </secret> #cat /xml/secret-value 123456 4.Prepare a guest xml with the luks image(the same uuid with the secret xml): #cat /tmp/vm1.xml ... <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='threads' copy_on_read='off'/> <source file='/var/lib/libvirt/images/luks.qcow2' index='1'> <encryption format='luks'> <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/> </encryption> </source> ... </disk> ... 5.Start guest with "virt-qemu-run -r": # virt-qemu-run -s /xml/secret.xml,/xml/value -d -v -r /tmp/test1 /tmp/vm1.xml virt-qemu-run: 84: initializing libvirt virt-qemu-run: 1157: initializing signal handlers virt-qemu-run: 1174: preparing event loop thread virt-qemu-run: 1223: opening secret:///embed?root=/tmp/test1 virt-qemu-run: 2126: loading secret /xml/secret.xml and /xml/value virt-qemu-run: 3405: opening qemu:///embed?root=/tmp/test1 virt-qemu-run: 51628: starting guest /tmp/vm1.xml virt-qemu-run: cannot start VM: internal error: process exited while connecting to monitor: 2020-07-23T08:22:08.722251Z qemu-kvm: -object secret,id=masterKey0,format=raw,file=/tmp/test1/lib/qemu/domain-1-vm1/master-key.aes: Unable to read /tmp/test1/lib/qemu/domain-1-vm1/master-key.aes: Failed to open file “/tmp/test1/lib/qemu/domain-1-vm1/master-key.aes”: Permission denied virt-qemu-run: 1088346: cleaned up, exiting Actual results: Permission denied to read master-key.aes when using 'virt-qemu-run -r' to start guest with luks disk. Expected results: Can start guest successfully. Additonal info: 1.Can start guest successfully if without specifying root dir; 2.If not reproduce, just reproduce step5 with different directory, such as: # virt-qemu-run -s /xml/secret.xml,/xml/value -d -v -r /tmp/test1 /tmp/vm2.xml