Bug 599122 - Unable to launch QEMU with a guest disk filename containing a ':'
Unable to launch QEMU with a guest disk filename containing a ':'
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.0
All Linux
low Severity medium
: rc
: ---
Assigned To: chellwig@redhat.com
Virtualization Bugs
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-06-02 13:42 EDT by Daniel Berrange
Modified: 2013-01-09 17:39 EST (History)
8 users (show)

See Also:
Fixed In Version: qemu-kvm-0.12.1.2-2.84.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-07-02 01:02:03 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
"fix" host device detection (1.42 KB, patch)
2010-06-03 07:59 EDT, chellwig@redhat.com
no flags Details | Diff

  None (edit)
Description Daniel Berrange 2010-06-02 13:42:28 EDT
Description of problem:
Any attempt to launch QEMU with a guest disk whose filename contains a ':' character results in an failure with 'Invalid argument' error.

This makes it impossible to use stable device names from the /dev/disk/by-path/ directory, which are critical for using iSCSI / SCSI HBAs.

This worked in earlier RHEL6 spins, so I believe some recent patch has broken this capability.

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.69.fc12.x86_64

How reproducible:
Always

Steps to Reproduce:
1. # /usr/libexec/qemu-kvm -drive file=/dev/disk/by-path/ip-192.168.254.185\:3260-iscsi-iqn.2004-04.com.qnap\:ts-439proii\:iscsi.berrangehome.bf6d84-lun-0
qemu: could not open disk image /dev/disk/by-path/ip-192.168.254.185:3260-iscsi-iqn.2004-04.com.qnap:ts-439proii:iscsi.berrangehome.bf6d84-lun-0: Invalid argument

2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Daniel Berrange 2010-06-02 13:49:52 EDT
NB. mistake in the version in the initial report - ignore the 'fc12' tag in the NEVR, this is in fact the 'el6' RPM.

Downgrading to qemu-kvm-0.12.1.2-2.55.el6 makes it work again.


Upstream suffers the same problem, and it appears to have been introduced with this change, though I'm not able to see what the flaw in that commit is. Reverting that commit makes it work again for me

  commit 84a12e6648444f517055138a7d7f25a22d7e1029
  Author: Christoph Hellwig <hch@lst.de>
  Date:   Wed Apr 7 22:30:24 2010 +0200

    block: separate raw images from the file protocol
Comment 2 chellwig@redhat.com 2010-06-03 03:56:03 EDT
This didn't work even before. 

Qemu 0.12.4:
# touch foo:0
# qemu-io foo:0
qemu-io> write 0 512
write failed: No medium found

Qemu HEAD:
# touch foo:0
# /opt/qemu/bin/qemu-io foo:0 
qemu-io: can't open device foo:0
qemu-io>

So the only difference is slightly better error handling.  The issues has been around upstream for a long time, with various attempts to fix it, none successful.  Search the qemu-devel archives for "support colon in filenames" in the subject line.
Comment 3 Daniel Berrange 2010-06-03 06:02:02 EDT
I don't know why your qemu 0.12.4 is giving you a write error, but empirically it works just fine for me. All my KVM guests are installed onto iSCSI devices using the paths that I illustrate above, so it clearly must have worked in the past or I would not have been able to install / run these guests. 

The full command line for one of my guests is thus:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 800 -smp 1,sockets=1,cores=1,threads=1 -name rhel6x86_64 -uuid ad8961e9-156f-746f-5a4e-f220bfafd08d -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/rhel6x86_64.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/dev/disk/by-path/ip-192.168.254.185:3260-iscsi-iqn.2004-04.com.qnap:ts-439proii:iscsi.berrangehome.bf6d84-lun-1,if=none,id=drive-ide0-0-0,boot=on,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/dev/disk/by-path/ip-192.168.254.185:3260-iscsi-iqn.2004-04.com.qnap:ts-439proii:iscsi.berrangehome.bf6d84-lun-0,if=none,id=drive-ide0-0-1,format=raw,cache=off -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=20,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:0a:ca:84,bus=pci.0,addr=0x4 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 

This boots successfully with qemu-kvm-0.12.1.2-2.55.el6 and fails with qemu-kvm-0.12.1.2-2.69.el6
Comment 4 chellwig@redhat.com 2010-06-03 06:43:51 EDT
As per my testcase and the mailinglist discussion it didn't work.  Please send me a simple testcase using qemu-io where it worked for you.
Comment 5 Daniel Berrange 2010-06-03 06:52:25 EDT
I really don't see why you have to continue to deny that it used to work, when I have shown that I have many installed & running iSCSI based guests all using storage whose files contain a ':'. 

But anyway, yes, the qemu-io used to work in qemu-kvm-0.12.1.2-2.55.el6

# /usr/bin/qemu-io /dev/disk/by-path/ip-192.168.254.185\:3260-iscsi-iqn.2004-04.com.qnap\:ts-439proii\:iscsi.berrangehome.bf6d84-lun-0 
qemu-io> read 0 512
read 512/512 bytes at offset 0
512.000000 bytes, 1 ops; 0.0000 sec (55.414 KiB/sec and 110.8279 ops/sec)



And in -2.69.el6 now fails

# /usr/bin/qemu-io /dev/disk/by-path/ip-192.168.254.185\:3260-iscsi-iqn.2004-04.com.qnap\:ts-439proii\:iscsi.berrangehome.bf6d84-lun-0 
qemu-io: can't open device /dev/disk/by-path/ip-192.168.254.185:3260-iscsi-iqn.2004-04.com.qnap:ts-439proii:iscsi.berrangehome.bf6d84-lun-0
qemu-io>
Comment 6 Daniel Berrange 2010-06-03 07:19:13 EDT
FYI the reason your test case in comment #2 didn't work is because your 'foo:0' name was pointing to a plain file. If it points to a block device it will work.

eg

# dd if=/dev/zero of=foo count=0 seek=100 bs=1M
# losetup -f foo
# ln -s /dev/loop0 foo\:0
# qemu-io foo\:0 
qemu-io> read 0 512
read 512/512 bytes at offset 0
512.000000 bytes, 1 ops; 0.0000 sec (3.590 MiB/sec and 7352.9412 ops/sec)
Comment 7 chellwig@redhat.com 2010-06-03 07:59:03 EDT
Created attachment 419342 [details]
"fix" host device detection

Indeed, I can reproduce it with a block device.

The reason is that historically we did the host device probing before parsing the protocol specifier.  This was wrong and removed with the file/raw split, but allowing host devices with colons in them to work was a side-effect of this.  The following patch restores that behaviour, although I'm not quite happy with it.
Comment 8 Daniel Berrange 2010-06-03 08:45:18 EDT
Perhaps instead of that hack, another idea would be to skip the parsing of the protocol specifier if the path starts with a '/' ? Block driver protocol names shouldn't ever contain a '/', so a leading '/' on the path is a good indicator that there is no protocol specifier.

Agree with your upstream comments that we should fix this properly in Markus' new '-blockdev' arg to have an explicit parameter instead of encoding in the filename.
Comment 9 Kevin Wolf 2010-06-04 08:04:38 EDT
You can specify the protocol explicitly with host_device:/dev/disk/by-path/ip-192.168.254.185\:3260-iscsi-iqn.2004-04.com.qnap\:ts-439proii\:iscsi.berrangehome.bf6d84-lun-0, it shouldn't try to interpret the colons in your path then.

Christoph's hack  possibly introduces a bug in other places - when a protocol driver accepts something that looks like a host device but in fact isn't. You wouldn't be able to use that driver with this option. I don't think it's useful right now (blkdebug needs another parameter, otherwise it would be the one), but not allowing it feels wrong.

(In reply to comment #8)
> Perhaps instead of that hack, another idea would be to skip the parsing of the
> protocol specifier if the path starts with a '/' ? Block driver protocol names
> shouldn't ever contain a '/', so a leading '/' on the path is a good indicator
> that there is no protocol specifier.

We've had that discussion before. Introducing more magic is not the right solution.

> Agree with your upstream comments that we should fix this properly in Markus'
> new '-blockdev' arg to have an explicit parameter instead of encoding in the
> filename.    

Not sure if it's possible to maintain compatibility and get a fix for this, especially considering that protocols can be stacked. But it's definitely something we should have a look at, at least. Adding Markus to CC.
Comment 10 Daniel Berrange 2010-06-04 08:49:07 EDT
> You can specify the protocol explicitly with
> host_device:/dev/disk/by-path/ip-192.168.254.185\:3260-iscsi-
> iqn.2004-04.com.qnap\:ts-439proii\:iscsi.berrangehome.bf6d84-lun-0,
> it shouldn't try to interpret the colons in your path then.

Which version is this supposed to work in ?  All versions I've tried fail with QEMU not stripping the protocol before trying to open the file.

Latest RHEL6:

# strace -e trace=open,stat /usr/libexec/qemu-kvm -drive file=host_device:/dev/disk/by-path/ip-192.168.254.185\:3260-iscsi-iqn.2004-04.com.qnap\:ts-439proii\:iscsi.berrangehome.bf6d84-lun-0 

open("host_device:/dev/disk/by-path/ip-192.168.254.185:3260-iscsi-iqn.2004-04.com.qnap:ts-439proii:iscsi.berrangehome.bf6d84-lun-0", O_RDONLY|O_SYNC|O_CLOEXEC) = -1 ENOENT (No such file or directory)
qemu: could not open disk image host_device:/dev/disk/by-path/ip-192.168.254.185:3260-iscsi-iqn.2004-04.com.qnap:ts-439proii:iscsi.berrangehome.bf6d84-lun-0: No such file or directory


Older RHEL6 pre the change that caused this BZ


# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.50.el6.x86_64

# strace -e trace=stat,open /usr/libexec/qemu-kvm -drive file=host_device:/root/foo\:bar 
stat("host_device:/root/foo:bar", 0x7fff1b996d40) = -1 ENOENT (No such file or directory)
qemu: could not open disk image host_device:/root/foo:bar: No such file or directory


Even older Fedora 12 qemu:

# rpm -q qemu-kvm
qemu-kvm-0.11.0-13.fc12.x86_64
# strace -e trace=open,stat qemu -drive file=host_device:/root/foo\:wibble  -vnc :2
stat("host_device:/root/foo:wibble", 0x7fff112113e0) = -1 ENOENT (No such file or directory)
Comment 11 Kevin Wolf 2010-06-04 09:18:01 EDT
Sorry, I seem to have confused my todo list with what's actually implemented... Anyway, would it help/be enough if this actually worked?
Comment 12 Daniel Berrange 2010-06-04 09:33:23 EDT
Well there's two issues at play where

 - Existing deployed libvirt installs working with new QEMU. This BZ is about the regression in block device handling for those.

 - Future libvirt with new QEMU. This will use Markus' -blockdev arg, which hopefully won't have this crazy encoding scheme for protocols embedded in filenames. 

So I don't think making 'host_device:' prefixes work is any real use/benefit. We need to fix the regression for the existing -drive syntax, and implement a saner scheme for future -blockdev syntax.
Comment 13 Markus Armbruster 2010-06-11 04:11:12 EDT
-blockdev definitely needs to support arbitrary filenames as cleanly as possible.  Crazy "when filename contains ':', it's really an encoding of a protocol" schemes are out of the question.
Comment 14 chellwig@redhat.com 2010-06-23 04:45:11 EDT
So what's our plan for this?  My patch while hacky reverts us to the pre raw/file split semantics for -drive and even if it's not perfect is pretty much required _iff_ we want to keep the -drive syntax compatible.  Or should we just ignore the -drive issue and wait for -blockdev?

Even if device names with a colon are quite stupid udev tends to create them and we'll have to support them for RHEL6.
Comment 15 Kevin Wolf 2010-06-23 05:21:00 EDT
Let's go with your hack. It feels wrong, but this is what compatibility means.
Comment 16 Markus Armbruster 2010-06-23 07:54:29 EDT
I agree.  We'll get our chance to do it right with -blockdev.
Comment 21 lihuang 2010-07-02 01:01:29 EDT
Verified on qemu-kvm-0.12.1.2-2.90.el6 (RHEL6.0-20100701.0)

/usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name xp-ide -uuid 7112a225-77f6-5970-7d3c-d39658af4c2f -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/xp-ide.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=localtime -no-reboot -boot d -drive file=/dev/disk/by-path/ip-10.66.91.141:3260-iscsi-iqn.2001-04.com.example:storage-lun-1,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/data/nfs-win/xp/en_windows_xp_professional_with_service_pack_3_x86_cd_x14-80428.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,cache=none -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=20,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:cd:86:67,bus=pci.0,addr=0x4 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -k en-us -vga std -device AC97,id=sound0,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3

virsh # list 
 Id Name                 State
----------------------------------
  5 xp                   running
  8 xp-ide               running

virsh # pool-list
Name                 State      Autostart 
-----------------------------------------
default              active     yes       
images               active     yes       
iscsi                active     yes 

virsh # pool-info iscsi
Name:           iscsi
UUID:           f74d86f0-9f56-762f-19c7-dce25fc94c5d
State:          running
Persistent:     yes
Autostart:      yes
Capacity:       144.92 GB
Allocation:     144.92 GB
Available:      0.00 

virsh # vol-list iscsi
Name                 Path                                    
-----------------------------------------
6.0.0.1              /dev/disk/by-path/ip-10.66.91.141:3260-iscsi-iqn.2001-04.com.example:storage-lun-1

virsh # vol-info --pool iscsi 6.0.0.1
Name:           6.0.0.1
Type:           block
Capacity:       144.92 GB
Allocation:     144.92 GB

Note You need to log in before you can comment on or make changes to this bug.