Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
It's better to improve the error message while failing hot-plug pci due to permission denied reason.
From the current error message and log, we can't find the root reason.
Version-Release number of selected component (if applicable):
libvirt-0.10.2-36.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.427.el6.x86_64
kernel-2.6.32-468.el6.x86_64
How reproducible:
100%
Steps to Reproduce:
1. edit file "/etc/libvirt/qemu.conf", change the following setting value to "0".
dynamic_ownership = 0
2. check the ownership of the guest image, it's "root:root" as default.
# ll /var/lib/libvirt/images/
-rw-------. 1 root root 8589934592 May 26 01:45 rhel6.6.img
3. fail to start the guest, due to the ownership permission.
# virsh start rhel6.6
error: Failed to start domain rhel6.6
error: internal error Process exited while reading console log output: char device redirected to /dev/pts/5
qemu-kvm: -drive file=/var/lib/libvirt/images/rhel6.6.img,if=none,id=drive-ide0-0-0,format=raw,cache=none: could not open disk image /var/lib/libvirt/images/rhel6.6.img: Permission denied
4. change the image ownership permission to "qemu:qemu".
# chown qemu:qemu /var/lib/libvirt/images/rhel6.6.img
# ll /var/lib/libvirt/images/
-rw-------. 1 qemu qemu 8589934592 May 26 01:45 rhel6.6.img
5. start the guest again, it will be successfully.
# virsh start rhel6.6
Domain rhel6.6 started
6. check the ownership of one pci device, it's "root:root" as default also.
# ll /sys/bus/pci/devices/0000\:00\:19.0/
......
-rw-r--r--. 1 root root 256 May 25 23:40 config
......
--w-------. 1 root root 4096 May 26 01:14 reset
-r--r--r--. 1 root root 4096 May 26 01:14 resource
-rw-------. 1 root root 131072 May 26 01:14 resource0
-rw-------. 1 root root 4096 May 26 01:14 resource1
-rw-------. 1 root root 32 May 26 01:14 resource2
......
7. fail to hot-plug the pci device to the guest, but the error message is clear for debugging.
# virsh attach-device rhel6.6 pci.xml
error: Failed to attach device from pci.xml
error: internal error unable to execute QEMU command 'device_add': Device 'pci-assign' could not be initialized
8. check the libvirtd log, it's also hard for debugging the fail reason.
2014-05-26 08:07:32.582+0000: 7425: error : qemuMonitorJSONCheckError:357 : internal error unable to execute QEMU command 'device_add': Device 'pci-assign' could not be initialized
2014-05-26 08:07:32.584+0000: 7425: error : qemuMonitorJSONCheckError:357 : internal error unable to execute QEMU command 'closefd': File descriptor named 'fd-hostdev0' not found
2014-05-26 08:07:32.584+0000: 7425: warning : qemuMonitorAddDeviceWithFd:2707 : failed to close device handle 'fd-hostdev0'
9. change the ownership of the pci device to "qemu:qemu".
# chown qemu:qemu /sys/bus/pci/devices/0000\:00\:19.0/*
# ll /sys/bus/pci/devices/0000\:00\:19.0/
......
-rw-r--r--. 1 qemu qemu 256 May 25 23:40 config
......
--w-------. 1 qemu qemu 4096 May 26 01:14 reset
-r--r--r--. 1 qemu qemu 4096 May 26 01:14 resource
-rw-------. 1 qemu qemu 131072 May 26 01:14 resource0
-rw-------. 1 qemu qemu 4096 May 26 01:14 resource1
-rw-------. 1 qemu qemu 32 May 26 01:14 resource2
......
10. hot-plug the pci device to guest, it is successful after change the ownership of the pci device.
# virsh attach-device rhel6.6 pci.xml
Device attached successfully
Actual results:
As step 7 and 8, can't analyse the root failed reason from the error message and libvirtd log.
Expected results:
It's better to improve the error message like step 3, contains some key words like "permission denied".
This is QEMU reporting just a pretty general error "Device 'pci-assign' could not be initialized" without providing any hints about the reason.
Comment 3Markus Armbruster
2015-01-15 09:30:13 UTC
Steps 1 to 5 show how image permission errors during startup are
reported. This is additional information not really pertinent to the
bug.
Steps 6 to 8 show how PCI device permission errors during hot plug are
reported. This is what the bug report is about.
Steps 9 to 10 demonstrate that the error is really a permission error,
by showing that fixing permissions make it go away.
Please provide libvirt logs for steps 6 to 8.
Alternatively, provide a reproducer for qemu-kvm showing the unhelpful error message.
(In reply to Markus Armbruster from comment #3)
> Steps 1 to 5 show how image permission errors during startup are
> reported. This is additional information not really pertinent to the
> bug.
As for the steps 1 to 5, I listed them out for your reference the error message, while the disk image ownership is not correct.
>
> Steps 6 to 8 show how PCI device permission errors during hot plug are
> reported. This is what the bug report is about.
>
> Steps 9 to 10 demonstrate that the error is really a permission error,
> by showing that fixing permissions make it go away.
>
> Please provide libvirt logs for steps 6 to 8.
Add the libvirtd log in comment 4 for your reference.
>
> Alternatively, provide a reproducer for qemu-kvm showing the unhelpful error
> message.
Comment 6Markus Armbruster
2015-01-27 15:57:58 UTC
The libvirt log sure looks like the qemu-kvm is from RHEL-7! RHEL-6 has no vfio-pci device. Your original report quotes the error message as "Device 'pci-assign' could not be initialized", which RHEL-6's qemu-kvm can actually produce. Are you sure you attached the right log file?
Comment 10Markus Armbruster
2015-01-28 07:56:42 UTC
Note: there's already RHEL-7 bug 616415. This bug is a special case
of the same issue in RHEL-6. Therefore, no need to clone this bug to
RHEL-7.
Comment 12Markus Armbruster
2015-01-28 08:16:27 UTC
Error reporting for device_add in QMP is generally poor. We're
working on improving it upstream, and hope the improvements can make
it into RHEL-7 (bug 616415). Unfortunately, the improvements involve
infrastructure not available in RHEL-6, and backporting it would be
far too intrusive. This makes the general problem intractable in
RHEL-6.
Instead, we can only add downstream-only hacks to improve specific
error cases. Example: bug 596279. Worthwhile only for sufficiently
common errors.
We already closed bug 678963 "QMP: does not emit the real reasons of
vfs assignment failure" because its error is insufficiently common:
limit of 8 assigned devices exceeded.
This one is the same problem (unspecific error in QMP) reached on a
different route: device node permission denied. I'm closing it as a
duplicate of bug 678963, on the assumption that it's not sufficiently
common. Please reopen if there's evidence that the bad error
reporting is affecting customers.
*** This bug has been marked as a duplicate of bug 678963 ***