Bug 184194

Summary: Device 51712 (vbd) could not be connected
Product: [Fedora] Fedora Reporter: Justin Conover <justin.conover>
Component: xenAssignee: Rik van Riel <riel>
Status: CLOSED NOTABUG QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: rawhideCC: andreas.bernstein, bstein, katzj, noa, sct
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2006-03-22 10:22:11 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
xend.log
none
xend-debug.log none

Description Justin Conover 2006-03-06 22:50:19 EST
Description of problem:
Initial xen install from xenguest-install.py errors out with:


Error: Device 51712 (vbd) could not be connected. Backend device not found.
If your install has exited, you can restart your guest by running
'xm create -c fc5-xen2'.
 xm create -c fc5-xen2
Using config file "/etc/xen/fc5-xen2".
Traceback (most recent call last):
  File "/usr/bin/pygrub", line 256, in ?
    cf = get_config(file, isconfig)
  File "/usr/bin/pygrub", line 110, in get_config
    if is_disk_image(fn):
  File "/usr/bin/pygrub", line 78, in is_disk_image
    buf = os.read(fd, 512)
OSError: [Errno 21] Is a directory
Error: Boot loader didn't return any data!




Version-Release number of selected component (if applicable):
kernel-xen-hypervisor-2.6.15-1.2009.4.2_FC5
xen-3.0.1-0.20060301.fc5.3

How reproducible:  100% so far

Steps to Reproduce:
1. /usr/sbin/xenguest-install.py
2. After you fill in the 4 questions it bombs out.

  # /usr/sbin/xenguest-install.py
What is the name of your virtual machine? fc5-xen2
 How much RAM should be allocated (in megabytes)? 256
 What would you like to use as the disk (path)? /xen
 What is the install location?
http://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/


Starting install...
Using config file "/etc/xen/fc5-xen2".
Error: Device 51712 (vbd) could not be connected. Backend device not found.
If your install has exited, you can restart your guest by running
'xm create -c fc5-xen2'.  Otherwise, you can reconnect to the console
by running 'xm console fc5-xen2'

# xm create -c fc5-xen2
Using config file "/etc/xen/fc5-xen2".
Traceback (most recent call last):
  File "/usr/bin/pygrub", line 256, in ?
    cf = get_config(file, isconfig)
  File "/usr/bin/pygrub", line 110, in get_config
    if is_disk_image(fn):
  File "/usr/bin/pygrub", line 78, in is_disk_image
    buf = os.read(fd, 512)
OSError: [Errno 21] Is a directory
Error: Boot loader didn't return any data!

# xm console fc5-xen2  (just hangs)

# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      489     2 r----- 14655.7
fc5-xen1                           1      256     1 --p---     0.0

Actual results:
Zero guest installs possible.

Expected results:
To install fc5 guest
Comment 1 Jeremy Katz 2006-03-06 22:58:37 EST
Did you have any loopback devs available?  Can you look and see what errors
there are in /var/log/xend.log and /var/log/xen-debug.log?

(Note that the xenguest-install script doesn't really have a good way of finding
out about failures of xm creating a domain, and so just always spews the "if
your install has exited..." bit -- in the future, I'm going to switch to using
libvirt to actually be able to have error handling :-)
Comment 2 Justin Conover 2006-03-07 09:41:06 EST
I'll check when I get home tonight, some reason I can't ssh to my box at home
the last two days, 
Comment 3 Justin Conover 2006-03-08 09:48:57 EST
Created attachment 125802 [details]
xend.log
Comment 4 Justin Conover 2006-03-08 09:50:12 EST
Created attachment 125803 [details]
xend-debug.log
Comment 5 Justin Conover 2006-03-08 13:05:15 EST
I didn't realize that the xen kernels had changed back to the old style naming
convention and yum wasn't updating them, so I just installed the latest from
work and will reboot and test when I get home to see if this is still a problem
on x86/smp box.

Thank you,
Comment 6 Justin Conover 2006-03-15 19:43:16 EST
# uname -r
2.6.15-1.2054_FC5xen0
[root@absolute ~]# /usr/sbin/xenguest-install.py
What is the name of your virtual machine? test1
 How much RAM should be allocated (in megabytes)? 256
 What would you like to use as the disk (path)? /xen
 What is the install location?
http://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/


Starting install...
Using config file "/etc/xen/test1".
Error: Device 51712 (vbd) could not be connected. Backend device not found.
If your install has exited, you can restart your guest by running
'xm create -c test1'.  Otherwise, you can reconnect to the console
by running 'xm console test1'
Comment 7 Noa Resare 2006-03-20 15:30:57 EST
I see this with fc5 final also. 

Clean system, kernel-xen0-2.6.15-1.2054_FC5, xen-3.0.1-4
Comment 8 Stephen Tweedie 2006-03-20 16:24:56 EST
Umm, looking at the initial comments:

  File "/usr/bin/pygrub", line 78, in is_disk_image
    buf = os.read(fd, 512)
  OSError: [Errno 21] Is a directory

Is "/xen" a directory?  You need to give it either a block device or the name of
a file (which need not already exist, it will create it if it needs to.)
Comment 9 Noa Resare 2006-03-20 16:57:24 EST
My traceback when trying to create a virtual machine looks like this:

[root@mandy noa]# /usr/sbin/xm create -c martin
Using config file "/etc/xen/martin".
Traceback (most recent call last):
  File "/usr/bin/pygrub", line 256, in ?
    cf = get_config(file, isconfig)
  File "/usr/bin/pygrub", line 139, in get_config
    raise RuntimeError, "Unable to read filesystem"
RuntimeError: Unable to read filesystem
Error: Boot loader didn't return any data!

[root@mandy noa]# ls -l /home/noa/martin.img
-rwxr-xr-x 1 root root 4294967297 20 mar 21.22 /home/noa/martin.img

/etc/xen/martin:

# Automatically generated xen config file
name = "martin"
memory = "500"
disk = [ 'file:/home/noa/martin.img,xvda,w' ]
vif = [ 'mac=00:16:3e:0f:37:9d' ]
bootloader="/usr/bin/pygrub"

on_reboot   = 'restart'
on_crash    = 'restart'
Comment 10 Justin Conover 2006-03-21 08:26:48 EST
(In reply to comment #8)
> Umm, looking at the initial comments:
> 
>   File "/usr/bin/pygrub", line 78, in is_disk_image
>     buf = os.read(fd, 512)
>   OSError: [Errno 21] Is a directory
> 
> Is "/xen" a directory?  You need to give it either a block device or the name of
> a file (which need not already exist, it will create it if it needs to.)
> 
$ ls -l /xen/
total 20
drwx------ 2 root root 16384 Mar  5 09:42 lost+found

On the example above, wouldn't this create "test1" under the /xen dir?
Comment 11 Stephen Tweedie 2006-03-21 09:52:00 EST
re comment #10:

No, there's no code in xenguest-install.py to automatically create a new subfile
under that directory: you need to give a full path to the final image file yourself.

re comment #9:

Does root definitely have access to that file?  I've had another report of this
problem recently which turned out to be because the file was mounted over NFS
from an export that did not have no_root_squash set, so root did not in fact
have write acces to it.
Comment 12 Justin Conover 2006-03-21 10:32:18 EST
Oops, apparently I didn't "read"

What would you like to use as the disk (path)? The local path and file name of
the file to serve as the disk image for the guest (eg, /home/joe/xenbox1). This
will be exported as a full disk to your guest.

http://www.fedoraproject.org/wiki/FedoraXenQuickstartFC5

Guess I assumed if 

What is the name of your virtual machine? This is the label used for display in
the various xm commands as well as being used as the name of the domain config
file in /etc/xen

That location would append the name to it, but again, my fault I didn't read it
completly.
Comment 13 Stephen Tweedie 2006-03-21 13:35:47 EST
OK, so Justin's report is a NOTABUG; I'll leave the bug in NEEDINFO while we
track down the problem in Noa Resare's case.
Comment 14 Noa Resare 2006-03-21 14:09:04 EST
Thanks. The file disk file is definitely readable, on a local ext3 filesystem.

Reading the source code I have reduced the failing case to this call

/usr/sbin/xm create -c kernel=/mnt/disc1/images/xen/vmlinuz
ramdisk=/mnt/disc1/images/xen/initrd.img extra=method=http://mandy/disc1
/etc/xen/test4

Result: 
Using config file "/etc/xen/test4".
Error: Device 51712 (vbd) could not be connected. Backend device not found.

[root@mandy /]# sha1sum /mnt/disc1/images/xen/[iv]*
a1bbe8a7cf2559326a18c7f6a010633d9e846e23  /mnt/disc1/images/xen/initrd.img
2a7cf0cb442c2ade89f1c695578027169a39a804  /mnt/disc1/images/xen/vmlinuz
[root@mandy noa]# python -c "import os; print os.access('/test4', os.W_OK)"
True
[root@mandy /]# cat /etc/xen/test4

# Automatically generated xen config file
name = "test4"
memory = "400"
disk = [ 'file:/test4,xvda,w' ]
vif = [ 'mac=00:16:3e:11:32:85' ]
Comment 15 Stephen Tweedie 2006-03-21 14:32:57 EST
The "Unable to read filesystem" error simply indicates that pygrub wasn't able
to open up the image file, locate a bootable partition in it, and find a valid
filesystem on that partition.  

Are you sure that a previous install has successfully completed on this partition?
Comment 16 Noa Resare 2006-03-21 16:23:26 EST
This problem is not related to pygrub, it shows up when creating the xen
instance with the installer kernel and initrd. At that stage the bootloader line
of the instance config file is commented out.

browsing around in the xend code is interesting (writing stuff in python surely
makes it more accesible) but I don't really get where the other end of the
xstransact communicatios channel in DevController resides. Once I can put some
more debugging statements at that place I think that finding out what goes wrong
should be pretty straight forward.
Comment 17 Noa Resare 2006-03-22 03:42:31 EST
For an unrelated reason I did a complete reinstall and now xen works as
expected. I have no idea what caused the problems unfortunately.

Please close this, and add an enhancement request for xen to have some more
detailed error reporting in the future :)

I'm sorry for the inconvenience.
Comment 18 Stephen Tweedie 2006-03-22 10:22:11 EST
OK, thanks!
Comment 19 Andreas Bernstein 2006-04-05 17:33:31 EDT
Problem: Stating XenU with file-based VBD ends in "Device <number> (vbd) could
not be connected. Backend device not found."

System: Linux SLES-TEST 2.6.12.6-xen3_12.1_sles9sp2 #1 SMP Tue Jan 31 14:32:26
PST 2006 i686 athlon i386 GNU/Linux

The error seems to be in the routine, which chooses/computes the name/number of
the next loop device to use for the file-based VBD.
With real partitions or LVM volumes everything is fine.

default number of loop devices: 8
Running a XenU with more than 2 file-based VBS raises the error.

entry in xen-hotplug.log:
/dev/loop10: No such device or address

Increasing the number of loops (to 32) solves the problem temporarily.
Interesting to call "losetup -a" while XenU is running:

Config file entry:
disk = ['file:/images/xen/xen-disk-1.img,hda1,w',
        'file:/images/xen/xen-disk-2.img,hda2,w',
        'file:/images/xen/xen-disk-3.img,hda3,w',
        'file:/images/xen/xen-disk-4.img,hda4,w',
        'file:/images/xen/xen-disk-5.img,hda5,w',
        'file:/images/xen/xen-disk-6.img,hda6,w']

SLES-TEST:/images/xen # losetup -a
/dev/loop0: [030a]:1308167 (/images/xen/xen-disk-4.img)
/dev/loop1: [030a]:1308166 (/images/xen/xen-disk-3.img)
/dev/loop10: [030a]:1308169 (/images/xen/xen-disk-6.img)
/dev/loop11: [030a]:1308165 (/images/xen/xen-disk-2.img)
/dev/loop12: [030a]:1308163 (/images/xen/xen-disk-1.img)
/dev/loop13: [030a]:1308168 (/images/xen/xen-disk-5.img)

Loops 2 ... 9 are not used (reproduced).

Tested on another machine (same linux version) - number of loop devices is now
128, 2 XenU running:

disk = [ 'file:/xen/maschine2/xen-disk-2.img,hda1,w',
         'file:/xen/maschine2/xen-disk-2-var.img,hda2,w',
         'file:/xen/maschine2/xen-disk-2-opt.img,hda3,w',
         'file:/xen/maschine2/xen-disk-2-swap.img,hda4,w',
         'file://xen/maschine2/td1.img,sda1,w',
         'file://xen/maschine2/td2.img,sda2,w',
         'file://xen/maschine2/td3.img,sda3,w',
         'file://xen/maschine2/td4.img,sda4,w' ]

XENTEST:/xen/maschine2 # losetup -a
/dev/loop0: [fe04]:10350400 (/xen/maschine2/xen-disk-2.img)
/dev/loop1: [fe04]:10350410 (/xen/maschine2/td3.img)
/dev/loop10: [fe04]:10350407 (/xen/maschine2/td2.img)
/dev/loop100: [fe04]:10350405 (/xen/maschine2/xen-disk-2-opt.img)
/dev/loop101: [fe04]:133 (/xen/maschine1/xen-disk-1.img)
/dev/loop102: [fe04]:10350411 (/xen/maschine2/td4.img)
/dev/loop103: [fe04]:10350404 (/xen/maschine2/xen-disk-2-var.img)
/dev/loop104: [fe04]:10350406 (/xen/maschine2/td1.img)

Xen does not use wide areas of /dev/loop[n] , then runs out of available device
numbers.

Hope, this helps to fix it...