Bug 716452

kudzu isn't really used by anything in RHEL6 - at least, nothing really important.  If any patches need to be made, it's likely to be in udev.

Please attach /tmp/storage.log after you get the error message to this bug report.  Thanks.

I'm afraid I have no /tmp/storage.log

I can give you the tail of the anaconda.log;

14:19:28,149 DEBUG   : waiting for hardware to initialize
14:19:28,436 INFO    : Trying to detect vendor driver discs
14:19:28,530 DEBUG   : probing buses
14:19:28,712 DEBUG   : waiting for hardware to initialize
14:19:29,784 ERROR   : got to setupCdrom without a CD device
14:19:32,099 DEBUG   : going to set language to en_US.UTF-8
14:19:32,099 INFO    : setting language to en_US.UTF-8
14:19:32,099 INFO    : starting STEP_METHOD

That's after it complained about no CD and then I dropped it into a shell.


More investigation shows that cdrom_id doesn't think xvdd is a CD drive either and that's a fairly short path through to xen_blockfront querying capabilities. It's possible this is a bug in the installer's kernel's driver which we've seen in Ubuntu builds which means DVD/CD capabilities don't report properly. However in that case, both devices were recognised as hard disks... so it's not exactly the same.

I've tried some additional experiments; it appears to be an issue with the kernel on the 6.1 install disk -- if I replace the kernel with the one from the 6.0 install disk, the rest of the 6.1 install disk initrd works OK.

We use the same kernel during installation as we do on the installed system.

That's a shame. That probably means that even if I let the installer run to completion, the installed system won't boot...

Hmm. Leaving it to run with the RH60 kernel and boot disk, the installer formats and partitions the xvda as expected, but then halts with a python exception saying;


anaconda 13.21.117 exception report ?
 ? Traceback (most recent call first):? ?
 ?   File "/usr/lib/anaconda/isys.py", line  ? ?
 ? 170, in umount? ?
 ?rc = _isys.umount(what)? ?
 ?   File "/usr/lib/anaconda/yuminstall.py", ? ?
 ? line 1029, in run? ?
 ?isys.umount("/mnt/stage2")? ?
 ?   File "/usr/lib/anaconda/yuminstall.py", ? ?
 ? line 1844, in doInstall? ?
 ?rc = self.ayum.run(self.instLog, cb,  ? ?
 ? anaconda.intf, anaconda.id) ?


Going to a console reveals that /mnt/stage2 really won't unmount (filesystem busy).

OK, I've had a play with loading modules and the kernel now successfully boots and runs the installer.

So the problem with the installer environment appears to be the xen_blkfront driver not detecting devices.

Is there any chance any patches might have gone missing? This bug was fixed in RHEL6.0 and this seems a regression from that.

Is this an HVM guest? I've never tried the install process you've outlined before, but the only difference I can think of between 6.0 and 6.1 that may have regressed this process is that for HVM guests we switched from pvdrivers being disabled by default to being enabled by default. To test that, you can add xen_emul_unplug=never to the kernel command line when booting it.

This is a pv guest.

Is the host running XenServer?  If so, can we get access to a reproduction environment?

Yes, the host is running XenServer.

This is reproducible with the free version of XenServer 5.6 -- the problem is that if you put a RHEL 6.1 install disk in and use the "Red Hat 6" VM template, then the kernel off the disk runs, but the installer won't detect the DVD drive (because it thinks /dev/xvdd is a hard disk not a removable read-only media drive because the device driver appears to not export the properties).

RHEL 6.0, by comparison, will boot and install.

I'm interested in this bug because apparently this workflow on the same host (XenServer 5.6) works for 6.0 guests, but not 6.1, which means a regression -- and we don't like regressions. OTOH, we don't support or test installing PV guests from CD/DVD, so it's actually surprising that it ever worked.

I decided I'd try to play with it, but the freely available XenServer 5.6 wants to delete my entire hard drive when I try to install it, so I'm not going to install it. Also, I don't know what this means

2. Boot from CD using RHEL template

is that something that becomes clear when using XenServer? Or can we get this template/installer?

Actually, even though this bug appears to be a RHEL kernel issue, I believe it would be better debugged by Citrix, since our host/tools don't support this workflow. I can make some kernels available to attempt to bisect the issue. We actually didn't touch much in blkfront wrt to PV guests between 6.0 and 6.1.

We think the problem may be a single line in the driver which seems to have fallen out of some people's kernel tree (Ubuntu being another distro which has lost it).

The patch is available here;

http://xen.1045712.n5.nabble.com/PATCH-blkfront-fix-data-size-for-xenbus-gather-in-blkfront-connect-td4364388.html


Is there any chance you could check this in the installer kernel source for 6.1 and see if that's wrong? If it is, and it's back in place in 6.2 then everything should be peachy.



"the freely available XenServer 5.6 wants
to delete my entire hard drive when I try to install it"

I can see how that would put you off...

"Boot from CD using RHEL template"

Yes, when you go to create a new VM, you get to pick from a list. One of which is for RHEL 6.x derived (32 or 62 bit).


"it's actually surprising that it ever worked."

We do go to quite some lengths to make it work :-)

Ah yes, we pulled in

7901d14 xen/blkfront: Use QUEUE_ORDERED_DRAIN for old backends

during 6.1 development, but never

4352b47 xen-blkfront: fix data size for xenbus_gather in blkfront_connect

so that is likely the culprit. Thanks for tracking it down for us. I'll create a kernel with this patch today and make it available for testing.

This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.

I've created a RHEL6 kernel rpm with the patch and put it here

http://people.redhat.com/drjones/kernel-2.6.32-170.el6716452.x86_64.rpm

sorry, only 64-bit, atm. I'll create a 32-bit, as well though. Thanks in advance for any testing done.

32-bit rpm: http://people.redhat.com/drjones/kernel-2.6.32-170.el6716452.i686.rpm

(In reply to comment #17)
> I've created a RHEL6 kernel rpm with the patch and put it here
> 
> http://people.redhat.com/drjones/kernel-2.6.32-170.el6716452.x86_64.rpm
> 
> sorry, only 64-bit, atm. I'll create a 32-bit, as well though. Thanks in
> advance for any testing done.

I can reproduce the issue with xenserver, but the fix doesn't work for me.
 
I'm not sure whether the test approach is correct to verify the fix:

[1] unpack the isolinux/initrd.img from the RHEL6.1 GA iso image.
[2] replace the xen-blkfront.ko.gz file extracted from initrd.img with the one which the fix has been applied
[3] re-pack the initrd.img and rebuild the iso image, boot up the vm (using RHEL6 template) via iso image, there is still no driver for block device, same output as RHEL6.1 GA during booting:
...
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/vbd/51760
XENBUS: Device with no driver: device/vif/0
...

I also tried to re-compile the xen-blkfront module after apply the patch to RHEL6.1 GA kernel, so the module has the same version with GA kernel, then the same steps as above, but still failed. So QE may need to try this fix after RHEL6.2 snapshot (with iso images composed) come out.

Patch(es) available on kernel-2.6.32-176.el6

Verified with RHEL6.2-20110823.1 (i386 and x86_64) iso images. kernel version is kernel-2.6.32-191.el6. XenServer is 5.6.100-47101p.

The vbd disk can be found during install, both the i386 and x86_64 guests can be installed and boot up after reboot successfully. So the fix works properly. Change this bug to VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2011-1530.html