| Summary: | libvirt is unable to access EMC PowerPath devices | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Robert Stroetgen <stroetgen> | ||||
| Component: | libvirt | Assignee: | Osier Yang <jyang> | ||||
| Status: | CLOSED CANTFIX | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 6.1 | CC: | acathrow, ajia, armbru, berthiaume_wayne, crobinso, dallan, dyuan, jyang, madhu.tarikere, michen, mkenneth, mzhan, psubrama, rwheeler, rwu, salmy, sluo, tburke, virt-maint, weizhan, xen-maint, ydu | ||||
| Target Milestone: | rc | ||||||
| Target Release: | 6.3 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2011-10-14 19:54:58 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Bug Depends On: | |||||||
| Bug Blocks: | 786481, 806717 | ||||||
| Attachments: |
|
||||||
Some more findings trying to start kvm with the libvirt parameters (from /var/log/libvirt/qemu/webapps-test.log): [root@vmhost3 ~]# LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name webapps-test -uuid 495e052d-259a-d31f-31c5-ba7622563bc0 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/webapps-test.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/dev/emcpowerr,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=20,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:21:ae:b1,bus=pci.0,addr=0x3 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k de -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -> does not boot (blank screen) Removing the -netdev parameter does not help. Removing the -netdev parameter and the -S parameter makes the vm boot. [root@vmhost3 ~]# su qemu -m -c "LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -M rhel6.0.0 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name webapps-test -uuid 495e052d-259a-d31f-31c5-ba7622563bc0 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/webapps-test.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/dev/emcpowerr,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k de -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5" Doing this als qemu user makes the vm boot, too: [root@vmhost3 ~]# su qemu -m -c "LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -M rhel6.0.0 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name webapps-test -uuid 495e052d-259a-d31f-31c5-ba7622563bc0 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/webapps-test.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/dev/emcpowerr,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k de -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -netdev tap,fd=20,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:21:ae:b1,bus=pci.0,addr=0x3" Can I prevent the -S parameter of kvm in the libvirt configuration to test if the vm boots without this parameter from virsh? Migth there be a relationship between the -S parameter and the EMC problem? Hi, Robert, libvirt always use "-S" of qemu-kvm at startup, and start the CPUs of guest later with sending command "cont" to qemu monitor. So unfortunately, there is no configuration to do it. I don't think it relates with "-netdev", as you don't specify it as a bootable device, my first guessing is it perhaps is problem of seabois, because as you saw in the VNC console, it can't recongnize the device as bootable disk. Hi Osier Yang,
thank you for your hint. I do not believe that it is a bios problem.
I configured the vm to boot from CDROM (super-grub-disk):
<os>
<type arch='x86_64' machine='rhel6.0.0'>hvm</type>
<boot dev='cdrom'/>
</os>
...
<disk type='block' device='disk'>
<driver name='qemu' type='raw'/>
<source dev='/dev/emcpowerc'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/vms/super_grub_disk_hybrid-1.98s1.iso'/>
<target dev='hda' bus='ide'/>
<readonly/>
<shareable/>
<address type='drive' controller='0' bus='0' unit='0'/>
</disk>
The vm boots from the CD-ROM. But the EMC device is not visible. Selecting "List devices/partitions" I only see the CD-ROM partition. "Detect any OS" does not find anything.
Another idea was disabling selinux - without any difference.
But I noticed that libvirt does not load the CD-ROM image from a NFS share.
qemu: could not open disk image /vms/shared/iso/gparted-live-0.8.0-5.iso: Permission denied
But qemu user is able to see it:
[root@vmhost3 vms]# su qemu -m -c "ls -al /vms/shared/iso/super_grub_disk_hybrid-1.98s1.iso"
-rw-rw-rw-. 1 root root 1443840 3. Nov 2010 /vms/shared/iso/super_grub_disk_hybrid-1.98s1.iso
Why is there any difference in permissions? Might this be the same problem with /dev/emcpowerc?
[snip] But I noticed that libvirt does not load the CD-ROM image from a NFS share. qemu: could not open disk image /vms/shared/iso/gparted-live-0.8.0-5.iso: Permission denied [/snip] You need to turn sebool "virt_use_nfs" on. # setsebool virt_use_nfs=on AFAIK, if it's permission problem, you should get error like "Permission denied" when trying to boot from /dev/emcpowerc. I have no EMC storage around, so have no way to investigate, but could you try to hotplug /dev/emcpowerc to guest in qemu directly to see what will happen? Because as you booted from the CDROM with /dev/emcpowerc as an aditional disk, another way is to check dmesg, I guess there might be something useful. (In reply to comment #4) > Because as you booted from the CDROM with /dev/emcpowerc as an aditional disk, > another way > is to check dmesg, I guess there might be something useful. Nothing usefull on dmesg: tun: Universal TUN/TAP device driver, 1.6 tun: (C) 1999-2004 Max Krasnyansky <maxk> device kvmnet-weba-t entered promiscuous mode br0: port 3(kvmnet-weba-t) entering forwarding state (In reply to comment #4) > I have no EMC storage around, so have no way to investigate, but could you try > to hotplug /dev/emcpowerc to guest in qemu directly to see what will happen? I tried to hotplug the device with [root@vmhost3 ~]# virsh attach-disk webapps /dev/emcpowerc vdb I see the device, but no partitions: root@webapps:/mnt# ls -al /dev/vd* brw-rw---- 1 root disk 252, 0 2011-05-10 14:05 /dev/vda brw-rw---- 1 root disk 252, 1 2011-05-10 14:06 /dev/vda1 brw-rw---- 1 root disk 252, 2 2011-05-10 14:05 /dev/vda2 brw-rw---- 1 root disk 252, 5 2011-05-10 14:05 /dev/vda5 brw-rw---- 1 root disk 252, 16 2011-05-10 17:06 /dev/vdb root@webapps:/mnt# fdisk /dev/vdb Unable to read /dev/vdb No problem for iSCSI devices. Nothing at all in the log files. (In reply to comment #6) Increasing libvirtd debug level does not show any difference between emc and iSCSI devices: [root@vmhost3 etc]# virsh attach-disk webapps /dev/emcpowerc vdb May 11 09:40:04 vmhost3 libvirtd: 09:40:04.635: info : qemudDispatchSignalEvent:397 : Received unexpected signal 17 May 11 09:40:04 vmhost3 libvirtd: 09:40:04.766: info : qemudDispatchSignalEvent:397 : Received unexpected signal 17 May 11 09:40:04 vmhost3 libvirtd: 09:40:04.769: info : qemuSecurityDACSetOwnership:40 : Setting DAC user and group on '/dev/emcpowerc' to '107:107' May 11 09:40:04 vmhost3 libvirtd: 09:40:04.769: info : SELinuxSetFilecon:323 : Setting SELinux context on '/dev/emcpowerc' to 'system_u:object_r:svirt_image_t:s0:c179,c601' [root@vmhost3 etc]# virsh attach-disk webapps /dev/disk/by-path/ip-172.17.0.103\:3260-iscsi-iqn.2010-11.de.gei\:RAID4VM.iscsi12.vg0.service-lun-13 vdc May 11 09:42:14 vmhost3 libvirtd: 09:42:14.669: info : qemuSecurityDACSetOwnership:40 : Setting DAC user and group on '/dev/disk/by-path/ip-172.17.0.103:3260-iscsi-iqn.2010-11.de.gei:RAID4VM.iscsi12.vg0.service-lun-13' to '107:107' May 11 09:42:14 vmhost3 libvirtd: 09:42:14.670: info : SELinuxSetFilecon:323 : Setting SELinux context on '/dev/disk/by-path/ip-172.17.0.103:3260-iscsi-iqn.2010-11.de.gei:RAID4VM.iscsi12.vg0.service-lun-13' to 'system_u:object_r:svirt_image_t:s0:c179,c601' May 11 09:42:17 vmhost3 libvirtd: 09:42:17.216: info : qemudDispatchServer:1398 : Turn off polkit auth for privileged client 24906 I don't think there will be difference, as the device is attached to guest successfully, IMO, this is not libvirt problem, more like qemu or kernel problem, but I need to conform it before could reassign it to qemu-kvm, unfortunately I can't do anything without a EMC powerpath device. Thank you for your patience. It seem rather difficult to debug. But why does qemu-kvm work (just removing the -S and the -netdev parameters) without any problem? [root@vmhost3 temp]# su qemu -m -c "LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -M rhel6.0.0 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name webapps-test -uuid 495e052d-259a-d31f-31c5-ba7622563bc0 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/webapps-test.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/dev/emcpowerc,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k de -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5" What is the difference between executing qemu-kvm directly and executing it via libvirt? I'm not quite farmiliar with qemu-kvm, but don't think it relates with "-netdev", as I said in comments 2. I guess you removed "-netdev" command line just because of you saw something like "tap device doesn't exist", that's because of the tap device is created by libvirt automatically, if you execute the command line directly, you need to create it mannually first. The difference between executing qemu-kvm directly and executing it via libvirt, and which may cause the problem of accessing the disk is libvirt set svirt labels on the used devices, but as comments 4 said, if there is any permission problem, you should got the error early before qemu-kvm trying to bootup the guest. After updating (kernel-2.6.32-131.0.15.el6.x86_64, libvirt-0.8.7-18.el6.x86_64, qemu-kvm-0.12.1.2-2.160.el6.x86_64) it still does not work, but I get some more debug output: /var/log/messages May 23 14:37:19 vmhost3 libvirtd: 14:37:19.170: 22308: info : qemudDispatchServer:1433 : Turn off polkit auth for privileged client pid 22364 from 127.0.0.1;0 May 23 14:37:19 vmhost3 libvirtd: 14:37:19.189: 22312: info : qemuSecurityDACSetOwnership:40 : Setting DAC user and group on '/dev/emcpowerc' to '107:107' May 23 14:37:19 vmhost3 libvirtd: 14:37:19.190: 22312: info : SELinuxSetFilecon:326 : Setting SELinux context on '/dev/emcpowerc' to 'system_u:object_r:svirt_image_t:s0:c659,c725' May 23 14:37:19 vmhost3 libvirtd: 14:37:19.341: 22312: info : brProbeVnetHdr:451 : IFF_VNET_HDR wird aktiviert May 23 14:37:19 vmhost3 kernel: device kvmnet-weba-t entered promiscuous mode May 23 14:37:19 vmhost3 kernel: br0: port 2(kvmnet-weba-t) entering forwarding state May 23 14:37:19 vmhost3 libvirtd: 14:37:19.372: 22312: warning : qemudStartVMDaemon:3336 : Executing /usr/libexec/qemu-kvm May 23 14:37:19 vmhost3 libvirtd: 14:37:19.377: 22312: warning : qemudStartVMDaemon:3346 : Executing done /usr/libexec/qemu-kvm May 23 14:37:19 vmhost3 qemu-kvm: Could not find keytab file: /etc/qemu/krb5.tab: No such file or directory May 23 14:37:22 vmhost3 ntpd[8259]: Listening on interface #15 kvmnet-weba-t, fe80::fc54:ff:fe21:aeb1#123 Enabled May 23 14:37:39 vmhost3 libvirtd: 14:37:39.561: 22308: info : qemudDispatchServer:1433 : Turn off polkit auth for privileged client pid 22464 from 127.0.0.1;0 May 23 14:37:39 vmhost3 libvirtd: 14:37:39.595: 22309: info : qemuSecurityDACRestoreSecurityFileLabel:80 : Restoring DAC user and group on '/dev/emcpowerc' May 23 14:37:39 vmhost3 libvirtd: 14:37:39.596: 22309: info : qemuSecurityDACSetOwnership:40 : Setting DAC user and group on '/dev/emcpowerc' to '0:0' May 23 14:37:39 vmhost3 libvirtd: 14:37:39.596: 22309: info : SELinuxRestoreSecurityFileLabel:430 : Restoring SELinux context on '/dev/emcpowerc' May 23 14:37:39 vmhost3 kernel: br0: port 2(kvmnet-weba-t) entering disabled state May 23 14:37:39 vmhost3 kernel: device kvmnet-weba-t left promiscuous mode May 23 14:37:39 vmhost3 kernel: br0: port 2(kvmnet-weba-t) entering disabled state May 23 14:37:39 vmhost3 libvirtd: 14:37:39.703: 22309: info : SELinuxSetFilecon:326 : Setting SELinux context on '/dev/emcpowerc' to 'system_u:object_r:device_t:s0' May 23 14:37:39 vmhost3 libvirtd: 14:37:39.723: 22309: error : virCgroupRemoveRecursively:679 : Unable to remove /cgroup/cpu/libvirt/qemu/webapps-test/ (16) May 23 14:37:39 vmhost3 libvirtd: 14:37:39.723: 22309: error : virCgroupRemoveRecursively:679 : Unable to remove /cgroup/cpuacct/libvirt/qemu/webapps-test/ (16) May 23 14:37:39 vmhost3 libvirtd: 14:37:39.723: 22309: error : virCgroupRemoveRecursively:679 : Unable to remove /cgroup/cpuset/libvirt/qemu/webapps-test/ (16) May 23 14:37:39 vmhost3 libvirtd: 14:37:39.723: 22309: error : virCgroupRemoveRecursively:679 : Unable to remove /cgroup/memory/libvirt/qemu/webapps-test/ (16) May 23 14:37:39 vmhost3 libvirtd: 14:37:39.723: 22309: error : virCgroupRemoveRecursively:679 : Unable to remove /cgroup/devices/libvirt/qemu/webapps-test/ (16) May 23 14:37:39 vmhost3 libvirtd: 14:37:39.723: 22309: error : virCgroupRemoveRecursively:679 : Unable to remove /cgroup/freezer/libvirt/qemu/webapps-test/ (16) May 23 14:37:39 vmhost3 libvirtd: 14:37:39.723: 22309: error : virCgroupRemoveRecursively:679 : Unable to remove /cgroup/blkio/libvirt/qemu/webapps-test/ (16) May 23 14:37:41 vmhost3 ntpd[8259]: Deleting interface #15 kvmnet-weba-t, fe80::fc54:ff:fe21:aeb1#123, interface stats: received=0, sent=0, dropped=0, active_time=19 secs May 23 14:38:01 vmhost3 libvirtd: 14:38:01.831: 22308: info : qemudDispatchServer:1433 : Turn off polkit auth for privileged client pid 22561 from 127.0.0.1;0 /var/log/libvirt/qemu/webapps-test.log 2011-05-23 14:37:19.351: starting up LC_ALL=C LD_LIBRARY_PATH=/etc/emc/rsa/cst/lib PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name webapps-test -uuid 495e052d-259a-d31f-31c5-ba7622563bc0 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/webapps-test.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/dev/emcpowerc,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:21:ae:b1,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -k de -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 14:37:19.376: 22371: info : libvirt version: 0.8.7, package: 18.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2011-04-18-10:28:30, x86-008.build.bos.redhat.com) 14:37:19.376: 22371: debug : virCgroupNew:566 : New group /libvirt/qemu/webapps-test 14:37:19.376: 22371: debug : virCgroupDetect:245 : Detected mount/mapping 0:cpu at /cgroup/cpu in 14:37:19.376: 22371: debug : virCgroupDetect:245 : Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in 14:37:19.376: 22371: debug : virCgroupDetect:245 : Detected mount/mapping 2:cpuset at /cgroup/cpuset in 14:37:19.376: 22371: debug : virCgroupDetect:245 : Detected mount/mapping 3:memory at /cgroup/memory in 14:37:19.376: 22371: debug : virCgroupDetect:245 : Detected mount/mapping 4:devices at /cgroup/devices in 14:37:19.376: 22371: debug : virCgroupDetect:245 : Detected mount/mapping 5:freezer at /cgroup/freezer in 14:37:19.376: 22371: debug : virCgroupDetect:245 : Detected mount/mapping 6:blkio at /cgroup/blkio in 14:37:19.376: 22371: debug : virCgroupMakeGroup:497 : Make group /libvirt/qemu/webapps-test 14:37:19.376: 22371: debug : virCgroupMakeGroup:509 : Make controller /cgroup/cpu/libvirt/qemu/webapps-test/ 14:37:19.377: 22371: debug : virCgroupMakeGroup:509 : Make controller /cgroup/cpuacct/libvirt/qemu/webapps-test/ 14:37:19.377: 22371: debug : virCgroupMakeGroup:509 : Make controller /cgroup/cpuset/libvirt/qemu/webapps-test/ 14:37:19.377: 22371: debug : virCgroupMakeGroup:509 : Make controller /cgroup/memory/libvirt/qemu/webapps-test/ 14:37:19.377: 22371: debug : virCgroupMakeGroup:509 : Make controller /cgroup/devices/libvirt/qemu/webapps-test/ 14:37:19.377: 22371: debug : virCgroupMakeGroup:509 : Make controller /cgroup/freezer/libvirt/qemu/webapps-test/ 14:37:19.377: 22371: debug : virCgroupMakeGroup:509 : Make controller /cgroup/blkio/libvirt/qemu/webapps-test/ 14:37:19.377: 22371: debug : virCgroupSetValueStr:290 : Set value '/cgroup/cpu/libvirt/qemu/webapps-test/tasks' to '22371' 14:37:19.389: 22371: debug : virCgroupSetValueStr:290 : Set value '/cgroup/cpuacct/libvirt/qemu/webapps-test/tasks' to '22371' 14:37:19.396: 22371: debug : virCgroupSetValueStr:290 : Set value '/cgroup/cpuset/libvirt/qemu/webapps-test/tasks' to '22371' 14:37:19.407: 22371: debug : virCgroupSetValueStr:290 : Set value '/cgroup/memory/libvirt/qemu/webapps-test/tasks' to '22371' 14:37:19.413: 22371: debug : virCgroupSetValueStr:290 : Set value '/cgroup/devices/libvirt/qemu/webapps-test/tasks' to '22371' 14:37:19.422: 22371: debug : virCgroupSetValueStr:290 : Set value '/cgroup/freezer/libvirt/qemu/webapps-test/tasks' to '22371' 14:37:19.427: 22371: debug : virCgroupSetValueStr:290 : Set value '/cgroup/blkio/libvirt/qemu/webapps-test/tasks' to '22371' 14:37:19.438: 22371: debug : qemudInitCpuAffinity:2383 : Setting CPU affinity 14:37:19.439: 22371: debug : qemuSecurityDACSetProcessLabel:547 : Dropping privileges of VM to 107:107 char device redirected to /dev/pts/1 Using CPU model "cpu64-rhel6" Using CPU model "cpu64-rhel6" block I/O error in device 'drive-virtio-disk0': Input/output error (5) Configuring the device as ide drive, I find the following error: block I/O error in device 'drive-ide0-0-1': Operation not permitted (1) (In reply to comment #12) > Configuring the device as ide drive, I find the following error: > > block I/O error in device 'drive-ide0-0-1': Operation not permitted (1) Can you please provide qemu strace output of the above? It looks like some issue with permission to this device. (In reply to comment #17) > Can you please provide qemu strace output of the above? > It looks like some issue with permission to this device. Sorry, we completely uninstalled the EMC PowerPath software and use dm-multipath now. I am not able to create a test environment, because we use the machines productively. It's caused (let's say it's the direct cause, not root cause) by cgroup devices controller indeed. After disabled "devices" controller in /etc/libvirt/qemu.conf, the guest can read and write the disk successfully. How to disable "devices" controller: 1) disable "devices" controller in /etc/libvirt/qemu.conf cgroup_controllers = [ "cpu", "memory" ] Just as danpb said, the root cause should be "the closed source EMC powerpath module might not integrate itself correctly with the cgroups devices controller". Because even if the "devices" controller is enabled. The powperpath disk is allowed by devices controller too, see the devices whitelist when "devices" controller is enabled: # cat /cgroup/devices/libvirt/qemu/RHEL4.8/devices.list b 120:1 rw <<<<< /dev/emcpowera1, major == 120, minor == 1 c 136:* rw c 1:3 rw c 1:7 rw c 1:5 rw c 1:8 rw c 1:9 rw c 5:2 rw c 10:232 rw c 254:0 rw c 10:228 rw # ls -l /dev/emcpowera1 brw-rw---- 1 qemu qemu 120, 1 9月 29 04:17 /dev/emcpowera1 After "devices" controller is disabled: # cat /cgroup/devices/libvirt/qemu/RHEL4.8/devices.list a *:* rwm Did more tracking, it looks strange to me that qemu-kvm tries to query the CDROM status (ioctl(11, CDROM_DRIVE_STATUS, ...); see below for more details) on powerpath device if the cgroups devices controller is enabled. And the error from qemu's stderr "block I/O error in device 'drive-virtio-disk1': Input/output error (5)" actually is caued by the ioctl call. If disabled devices controller, there is no ioctl(11, CDROM_DRIVE_STATUS, ...), and also no error in the stderr anymore, and can see and operate on the disk quite well in guest. qemu command lines: ==== LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name RHEL4.8 -uuid 104d08e9-5044-741a-24a5-35a62a10519f -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/RHEL4.8.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/var/lib/libvirt/images/RHEL4.8.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=threads -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/dev/emcpowera1,if=none,id=drive-virtio-disk1,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:ed:b5,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 Trace qemu process ==== # virsh start RHEL4.8; strace -o trace.2 -p $(pidof qemu-kvm) [ ------------ devices controller enabled ------------------- ] # ls -l /proc/$(pidof qemu-kvm)/fd/ 总用量 0 lrwx------ 1 qemu qemu 64 9月 30 03:12 0 -> /dev/null l-wx------ 1 qemu qemu 64 9月 30 03:12 1 -> /var/log/libvirt/qemu/RHEL4.8.log lrwx------ 1 qemu qemu 64 9月 30 03:12 10 -> anon_inode:[signalfd] lrwx------ 1 qemu qemu 64 9月 30 03:12 11 -> /dev/emcpowera1 lrwx------ 1 qemu qemu 64 9月 30 03:12 12 -> anon_inode:kvm-vcpu lrwx------ 1 qemu qemu 64 9月 30 03:12 13 -> socket:[167704] lrwx------ 1 qemu qemu 64 9月 30 03:12 14 -> socket:[167702] lrwx------ 1 qemu qemu 64 9月 30 03:12 15 -> anon_inode:[eventfd] lrwx------ 1 qemu qemu 64 9月 30 03:12 16 -> anon_inode:[eventfd] lrwx------ 1 qemu qemu 64 9月 30 03:12 17 -> anon_inode:[signalfd] lrwx------ 1 qemu qemu 64 9月 30 03:12 18 -> socket:[167705] lrwx------ 1 qemu qemu 64 9月 30 03:12 19 -> anon_inode:[eventfd] l-wx------ 1 qemu qemu 64 9月 30 03:12 2 -> /var/log/libvirt/qemu/RHEL4.8.log lrwx------ 1 qemu qemu 64 9月 30 03:12 20 -> anon_inode:[eventfd] lrwx------ 1 qemu qemu 64 9月 30 03:12 22 -> /dev/net/tun lrwx------ 1 qemu qemu 64 9月 30 03:12 23 -> /dev/vhost-net lrwx------ 1 qemu qemu 64 9月 30 03:12 3 -> socket:[167692] lrwx------ 1 qemu qemu 64 9月 30 03:12 4 -> /dev/ptmx lrwx------ 1 qemu qemu 64 9月 30 03:12 5 -> /dev/kvm lrwx------ 1 qemu qemu 64 9月 30 03:12 6 -> anon_inode:kvm-vm lr-x------ 1 qemu qemu 64 9月 30 03:12 7 -> pipe:[167698] l-wx------ 1 qemu qemu 64 9月 30 03:12 8 -> pipe:[167698] lrwx------ 1 qemu qemu 64 9月 30 03:12 9 -> /var/lib/libvirt/images/RHEL4.8.img ==== Many ioctls with CDROM_DRIVE_STATUS command in strace log ioctl(11, CDROM_DRIVE_STATUS, 0x7fffffff) = 1 write(2, "block I/O error in device 'drive"..., 71) = 71 ==== From stderr of qemu-kvm block I/O error in device 'drive-virtio-disk1': Input/output error (5) block I/O error in device 'drive-virtio-disk1': Input/output error (5) block I/O error in device 'drive-virtio-disk1': Input/output error (5) block I/O error in device 'drive-virtio-disk1': Input/output error (5) [ ------------ devices controller disabled ------------------- ] # ls /proc/$(pidof qemu-kvm)/fd -l 总用量 0 lrwx------ 1 qemu qemu 64 9月 30 03:29 0 -> /dev/null l-wx------ 1 qemu qemu 64 9月 30 03:29 1 -> /var/log/libvirt/qemu/RHEL4.8.log lrwx------ 1 qemu qemu 64 9月 30 03:29 10 -> anon_inode:[signalfd] lrwx------ 1 qemu qemu 64 9月 30 03:29 11 -> /dev/emcpowera1 lrwx------ 1 qemu qemu 64 9月 30 03:29 12 -> anon_inode:kvm-vcpu lrwx------ 1 qemu qemu 64 9月 30 03:29 13 -> socket:[169181] lrwx------ 1 qemu qemu 64 9月 30 03:29 14 -> socket:[169179] lrwx------ 1 qemu qemu 64 9月 30 03:29 15 -> anon_inode:[eventfd] lrwx------ 1 qemu qemu 64 9月 30 03:29 16 -> anon_inode:[eventfd] lrwx------ 1 qemu qemu 64 9月 30 03:29 17 -> anon_inode:[signalfd] lrwx------ 1 qemu qemu 64 9月 30 03:29 18 -> socket:[169182] lrwx------ 1 qemu qemu 64 9月 30 03:29 19 -> anon_inode:[eventfd] l-wx------ 1 qemu qemu 64 9月 30 03:29 2 -> /var/log/libvirt/qemu/RHEL4.8.log lrwx------ 1 qemu qemu 64 9月 30 03:29 20 -> anon_inode:[eventfd] lrwx------ 1 qemu qemu 64 9月 30 03:29 22 -> /dev/net/tun lrwx------ 1 qemu qemu 64 9月 30 03:29 23 -> /dev/vhost-net lrwx------ 1 qemu qemu 64 9月 30 03:29 3 -> socket:[169169] lrwx------ 1 qemu qemu 64 9月 30 03:29 4 -> /dev/ptmx lrwx------ 1 qemu qemu 64 9月 30 03:29 5 -> /dev/kvm lrwx------ 1 qemu qemu 64 9月 30 03:29 6 -> anon_inode:kvm-vm lr-x------ 1 qemu qemu 64 9月 30 03:29 7 -> pipe:[169176] l-wx------ 1 qemu qemu 64 9月 30 03:29 8 -> pipe:[169176] lrwx------ 1 qemu qemu 64 9月 30 03:29 9 -> /var/lib/libvirt/images/RHEL4.8.img ==== No ioctl(11, CDROM_DRIVE_STATUS, ...) statements in the strace log ==== No erros in qemu-kvm stderr ==== Can see and operate on the disk in guest successfully, E.g. [root@localhost ~]# !fdisk fdisk /dev/vdb The number of cylinders for this disk is set to 145620. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (1-145620, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-145620, default 145620): Using default value 145620 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@localhost ~]# !cat cat /proc/partitions major minor #blocks name 253 0 8388608 vda 253 1 104391 vda1 253 2 5116702 vda2 253 3 1052257 vda3 253 16 73392921 vdb 253 18 73392448 vdb2 traced qemu with gdb more,
1) Start guest with "-S"
# virsh start RHEL4.8 --paused
2) Attach guest process to gdb
# gdb -p $(pidof qemu-kvm) -se /usr/lib/debug/usr/libexec/qemu-kvm.debug
3) Send "cont" to qemu monitor
# virsh qemu-monitor-command RHEL4.8 --cmd "cont" --hmp
================
(gdb) break cdrom_is_inserted
Breakpoint 1 at 0x499100: file block/raw-posix.c, line 1159.
(gdb) thread apply bt all
(gdb) c
Continuing.
[New Thread 0x7ff66ae68700 (LWP 28995)]
Breakpoint 1, cdrom_is_inserted (bs=0x1a25980) at block/raw-posix.c:1159
1159 {
(gdb) list
1154 out:
1155 return prio;
1156 }
1157
1158 static int cdrom_is_inserted(BlockDriverState *bs)
1159 {
1160 BDRVRawState *s = bs->opaque;
1161 int ret;
1162
1163 ret = ioctl(s->fd, CDROM_DRIVE_STATUS, CDSL_CURRENT);
(gdb) list
1164 if (ret == CDS_DISC_OK)
1165 return 1;
1166 return 0;
1167 }
1168
1169 static int cdrom_eject(BlockDriverState *bs, int eject_flag)
1170 {
1171 BDRVRawState *s = bs->opaque;
1172
1173 if (eject_flag) {
(gdb) bt
#0 cdrom_is_inserted (bs=0x1a25980) at block/raw-posix.c:1159
#1 0x000000000047dba1 in bdrv_check_byte_request (bs=0x1a25020, sector_num=<value optimized out>, nb_sectors=<value optimized out>)
at block.c:865
#2 bdrv_check_request (bs=0x1a25020, sector_num=<value optimized out>, nb_sectors=<value optimized out>) at block.c:885
#3 0x0000000001a25020 in ?? ()
#4 0x00000000008d9300 in ?? ()
#5 0x0000000000000000 in ?? ()
The "ioctl" in "cdrom_is_inserted" is exact what give the error "block I/O error in device 'drive-virtio-disk1': Input/output error (5)". But I get no luck to get more backtrace yet. And I still don't get a clear clue about "devices controller -> the error -> the problem".
cdrom_inserted() is a method of block driver "host_cdrom". Looks like we somehow ended up with the "host_cdrom" driver, which is very wrong. My theory should be easy to verify in the debugger. If it's true, the next question is how that driver got selected. Perhaps the code that picks the appropriate driver gets confused by something. Again, should be easy to figure out in the debugger. Osier, since you can reproduce in the debugger already, I suggest you grab me on IRC at the first convenient time, so we can analyse this together. (In reply to comment #28) > cdrom_inserted() is a method of block driver "host_cdrom". Looks like we > somehow ended up with the "host_cdrom" driver, which is very wrong. True, I doubted the error is by qemu tries to call .probe_device method of the block drivers in the beginning, before I run gdb on it. If it is, that's more sensiable than cdrom_inserted(), :-) > > My theory should be easy to verify in the debugger. If it's true, the next > question is how that driver got selected. Perhaps the code that picks the > appropriate driver gets confused by something. Again, should be easy to figure > out in the debugger. I can't get the whole backtrace by # gdb -p $(pidof qemu-kvm) -se /usr/lib/debug/usr/libexec/qemu-kvm.debug it seems to me many symbols can't be found. /usr/lib/debug/usr/libexec/qemu-kvm.debug is from qemu-kvm-debuginfo I installed on the testing machine. > > Osier, since you can reproduce in the debugger already, I suggest you grab me > on IRC at the first convenient time, so we can analyse this together. Great. looks like you are 6 hours ahead of me, I will choose a proper time to grab you. I verified in the debugger that the qemu-kvm running under libvirt picks block driver raw over host_cdrom for the drive with file=/dev/emcpowera1. This is because ioctl(fd, CDROM_DRIVE_STATUS, CDSL_CURRENT) succeeds for it, which makes cdrom_probe_device() return 100. This is a kernel bug, almost certainly in the device driver.
If I start qemu-kvm outside libvirt (with a suitably modified command line), then the same ioctl() fails, as it should.
To simplify testing I created a little program to examine the ioctl's behavior. When I run it in a root shell, it shows the ioctl fails. When I run it as a child of qemu-kvm under libvirt (p system("...") in gdb, find output in the log), the ioctl succeeds.
Obviously, libvirt changes the environment in a way that triggers the kernel device driver bug.
SELinux is disabled on the test machine, so we can rule that one out.
Comment#24 fingers cgroups. I tried to verify, no luck. I figure I screwed up the test, need to try again.
Created attachment 527891 [details] Test program mentioned in comment#31 Outside libvirt: # /home/armbru/ioctl-cdsl /dev/sr0 /dev/tty /dev/emcpowera1 /dev/sr0 1 /dev/tty -1 /dev/emcpowera1 -1 Under libvirt, namely gdb attached to qemu-kvm running under libvirt, then (gdb) p system("/home/armbru/ioctl-cdsl /dev/emcpowera1") Detaching after fork from child process 31169. $7 = 0 Check the log: # tail -n 1 RHEL4.8.log /dev/emcpowera1 1 Re comment#25: "it looks strange to me that qemu-kvm tries to query the CDROM status [...] on powerpath device if the cgroups devices controller is enabled". It queries CD-ROM status first in cdrom_probe_device(), to figure out whether this is a CD-ROM device. If it is, it queries frequently in cdrom_is_inserted(). You saw the second kind of query (comment#27) The powerpath device isn't a CD-ROM, and its ioctl must not succeed. It does, in certain circumstances, and once the probing ioctl succeeds, all is lost. The "strange" queries you observed are just another symptom of being lost. Re comment#9 "why does qemu-kvm work (just removing the -S and the -netdev parameters) without any problem?". Libvirt sets up the network backend by passing an file descriptor: "-netdev tap,fd=20,id=hostnet0". Simply copying that qemu-kvm invocation into a shell can't work, because you don't pass the fd. Deleting -netdev "fixes" the problem by getting rid of the broken network backend configuration. Libvirt starts qemu-kvm stopped (-S), then continues it with a monitor command. You need to do the same, or delete -S. Network backends and stop/continue are completely unrelated to the powerpath bug. Something libvirt does triggers the bug, most likely cgroups. If we manage to do the same on the command line, we'll trigger it there, too. I figure the needinfos are obsolete and can be cleaned out now. (In reply to comment #31) > I verified in the debugger that the qemu-kvm running under libvirt picks block > driver raw over host_cdrom for the drive with file=/dev/emcpowera1. This is > because ioctl(fd, CDROM_DRIVE_STATUS, CDSL_CURRENT) succeeds for it, which > makes cdrom_probe_device() return 100. This is a kernel bug, almost certainly > in the device driver. > > If I start qemu-kvm outside libvirt (with a suitably modified command line), > then the same ioctl() fails, as it should. > > To simplify testing I created a little program to examine the ioctl's behavior. > When I run it in a root shell, it shows the ioctl fails. When I run it as a > child of qemu-kvm under libvirt (p system("...") in gdb, find output in the > log), the ioctl succeeds. > > Obviously, libvirt changes the environment in a way that triggers the kernel > device driver bug. > > SELinux is disabled on the test machine, so we can rule that one out. > > Comment#24 fingers cgroups. I tried to verify, no luck. I figure I screwed up > the test, need to try again. I tried to reproduce the problem with Markus's testing program in a lazy manner. Sleeps several secs in ioctl-cdsl before the ioctl, execute ioctl-cdsl, and start following script quickly so that the cgroup could be setup before ioctl happens. #! /bin/sh while [ -z $pid ]; do pid=$(pidof ioctl-cdsl) done [ -d /cgroup/devices/ioctl-cdsl ] && mkdir /cgroup/devices/ioctl-cdsl echo $pid > /cgroup/devices/ioctl-cdsl/tasks echo a > /cgroup/devices/ioctl-cdsl/devices.deny # "120:1" is "major:minor" of /dev/emcpowera1 echo "b 120:1 rw" > /cgroup/devices/ioctl-cdsl/devices.all # /home/armbru/ioctl-cdsl /dev/emcpowera1 /dev/emcpowera1 1 So, it's clear the problem is not of either libvirt or qemu-kvm, but of kernel. Given the results of the investigation, it appears that this is a problem with the PowerPath driver, so I'm going to close this BZ as CANTFIX. However, we would be happy to work with EMC if our input would be helpful. Turning off the cgroups devices controller as noted in comment 24 will work around the problem. I append this information to our service request at EMC. We ware quite satisfied using dm-multipath, now. Thanks for all the help and investigation! *** Bug 825920 has been marked as a duplicate of this bug. *** *** Bug 825920 has been marked as a duplicate of this bug. *** |
Description of problem: After moving from iSCSI to EMC PowerPath libvirt is unable to access the devices. Version-Release number of selected component (if applicable): kernel-2.6.32-71.el6.x86_64 kernel-2.6.32-71.18.2.el6.x86_64 kernel-2.6.32-71.24.1.el6.x86_64 libvirt-0.8.1-27.el6_0.6.x86_64 qemu-kvm-0.12.1.2-2.113.el6_0.8.x86_64 EMCpower.LINUX-5.6.0.00.00-143.x86_64 How reproducible: Try to use EMC PowerPath devices with libvirt. Happens on two different machines. Steps to Reproduce: 1. 2. 3. Actual results: Virtual machine is started, but EMC PowerPath device not readable. Virtual machine is unable to boot. Expected results: Virtual machine is started and EMC PowerPath device is readable. Virtual machine is able to boot. Additional info: [root@vmhost3 ~]# ls -al /dev/emcpowerr brw-rw----. 1 root disk 120, 272 9. Mai 10:14 /dev/emcpowerr The disk is accessible, we moved the data successfully with dd. Old disk configuration: <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/disk/by-path/ip-172.17.0.103:3260-iscsi-iqn.2010-11.de.gei:RAID4VM.iscsi10.vg0.webapps-lun-11'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> New disk configuration: <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/emcpowerr'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> We can access the disk with virt-rescue: [root@vmhost3 ~]# virt-rescue /dev/emcpowerr ... ><rescue> fdisk -l /dev/vda Disk /dev/vda: 107.4 GB, 107374182400 bytes 255 heads, 63 sectors/track, 13054 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000ee0d2 Device Boot Start End Blocks Id System /dev/vda1 * 1 6345 50966181 83 Linux /dev/vda2 6346 6527 1461915 5 Extended /dev/vda5 6346 6527 1461883+ 82 Linux swap / Solaris We can successfully boot with kvm: [root@vmhost3 ~]# /usr/libexec/qemu-kvm -hda /dev/emcpowerr -boot c -vnc :0 We can successfully boot with kvm as libvirt user qemu after adding this user to the group disk, too: [root@vmhost3 ~]# su qemu -m -c "/usr/libexec/qemu-kvm -hda /dev/emcpowerr -boot c -vnc :0" We can not boot with virsh: [root@vmhost3 ~]# virsh start webapps-test Domain webapps-test gestartet Error in VNC console: Booting from Hard drive... Boot failed: not a bootable disk No bootable devices The device is correctly mapped to qemu by libvirt: [root@vmhost3 ~]# ls -al /dev/emcpowerr brw-rw----. 1 qemu qemu 120, 272 9. Mai 10:21 /dev/emcpowerr /var/log/libvirt/qemu/webapps-test.log: LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name webapps-test -uuid 495e052d-259a-d31f-31c5-ba7622563bc0 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/webapps-test.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/dev/emcpowerr,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=20,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:21:ae:b1,bus=pci.0,addr=0x3 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k de -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 char device redirected to /dev/pts/1 Read failed Setting loglevel to debug does not produce more helpfull output. Nothing helpfull in /var/log/messages. Changing the libvirt user to root/root in /etc/libvirt/qemu.conf does not make any difference.