Hide Forgot
Description of problem: Anaconda versions on CD/DVD images for RHEL6.1 fail to detect ISO images when presented as Xen virtual disks as CDROM. Also fails to see writeable block devices as hard disks. Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Insert ISO image as DVD block device 2. Boot from CD using RHEL template 3. Choose local DVD/CD install Actual results: Error message: "Unable to find any devices of the type needed for this installation type." Expected results: Mounting the CD for installation. 4. Choose "hard drive" install. Error message: "You don't seem to have any hard drives on your system!" Additional info: xen front device drivers have loaded, are functioning and have detected the block devices. Network interfaces are also detected correctly indicating that the VM domain is sensibly working. /dev/xvda (HD) and /dev/xvdd (DVD) are present as expected but appear not to be being detected as HD and DVD. Previous version (RHEL6.0) contained patches to correct this. Earlier versions in 5.x series did not. Citrix XenSource would like to put a fix inside XenServer to make this work, but we'd also like to ask if the 6.0 patches could be added into RHEL6.2 and also possibly if you could lend assistance in understanding how to fix 6.1 -- our changes which work in the 5.x series (patches to, and a rebuild of kudzu) don't seem applicable because the installer's structures seem to have changed.
kudzu isn't really used by anything in RHEL6 - at least, nothing really important. If any patches need to be made, it's likely to be in udev. Please attach /tmp/storage.log after you get the error message to this bug report. Thanks.
I'm afraid I have no /tmp/storage.log I can give you the tail of the anaconda.log; 14:19:28,149 DEBUG : waiting for hardware to initialize 14:19:28,436 INFO : Trying to detect vendor driver discs 14:19:28,530 DEBUG : probing buses 14:19:28,712 DEBUG : waiting for hardware to initialize 14:19:29,784 ERROR : got to setupCdrom without a CD device 14:19:32,099 DEBUG : going to set language to en_US.UTF-8 14:19:32,099 INFO : setting language to en_US.UTF-8 14:19:32,099 INFO : starting STEP_METHOD That's after it complained about no CD and then I dropped it into a shell. More investigation shows that cdrom_id doesn't think xvdd is a CD drive either and that's a fairly short path through to xen_blockfront querying capabilities. It's possible this is a bug in the installer's kernel's driver which we've seen in Ubuntu builds which means DVD/CD capabilities don't report properly. However in that case, both devices were recognised as hard disks... so it's not exactly the same.
I've tried some additional experiments; it appears to be an issue with the kernel on the 6.1 install disk -- if I replace the kernel with the one from the 6.0 install disk, the rest of the 6.1 install disk initrd works OK.
We use the same kernel during installation as we do on the installed system.
That's a shame. That probably means that even if I let the installer run to completion, the installed system won't boot...
Hmm. Leaving it to run with the RH60 kernel and boot disk, the installer formats and partitions the xvda as expected, but then halts with a python exception saying; anaconda 13.21.117 exception report ? ? Traceback (most recent call first):? ? ? File "/usr/lib/anaconda/isys.py", line ? ? ? 170, in umount? ? ?rc = _isys.umount(what)? ? ? File "/usr/lib/anaconda/yuminstall.py", ? ? ? line 1029, in run? ? ?isys.umount("/mnt/stage2")? ? ? File "/usr/lib/anaconda/yuminstall.py", ? ? ? line 1844, in doInstall? ? ?rc = self.ayum.run(self.instLog, cb, ? ? ? anaconda.intf, anaconda.id) ? Going to a console reveals that /mnt/stage2 really won't unmount (filesystem busy).
OK, I've had a play with loading modules and the kernel now successfully boots and runs the installer. So the problem with the installer environment appears to be the xen_blkfront driver not detecting devices. Is there any chance any patches might have gone missing? This bug was fixed in RHEL6.0 and this seems a regression from that.
Is this an HVM guest? I've never tried the install process you've outlined before, but the only difference I can think of between 6.0 and 6.1 that may have regressed this process is that for HVM guests we switched from pvdrivers being disabled by default to being enabled by default. To test that, you can add xen_emul_unplug=never to the kernel command line when booting it.
This is a pv guest.
Is the host running XenServer? If so, can we get access to a reproduction environment?
Yes, the host is running XenServer. This is reproducible with the free version of XenServer 5.6 -- the problem is that if you put a RHEL 6.1 install disk in and use the "Red Hat 6" VM template, then the kernel off the disk runs, but the installer won't detect the DVD drive (because it thinks /dev/xvdd is a hard disk not a removable read-only media drive because the device driver appears to not export the properties). RHEL 6.0, by comparison, will boot and install.
I'm interested in this bug because apparently this workflow on the same host (XenServer 5.6) works for 6.0 guests, but not 6.1, which means a regression -- and we don't like regressions. OTOH, we don't support or test installing PV guests from CD/DVD, so it's actually surprising that it ever worked. I decided I'd try to play with it, but the freely available XenServer 5.6 wants to delete my entire hard drive when I try to install it, so I'm not going to install it. Also, I don't know what this means 2. Boot from CD using RHEL template is that something that becomes clear when using XenServer? Or can we get this template/installer? Actually, even though this bug appears to be a RHEL kernel issue, I believe it would be better debugged by Citrix, since our host/tools don't support this workflow. I can make some kernels available to attempt to bisect the issue. We actually didn't touch much in blkfront wrt to PV guests between 6.0 and 6.1.
We think the problem may be a single line in the driver which seems to have fallen out of some people's kernel tree (Ubuntu being another distro which has lost it). The patch is available here; http://xen.1045712.n5.nabble.com/PATCH-blkfront-fix-data-size-for-xenbus-gather-in-blkfront-connect-td4364388.html Is there any chance you could check this in the installer kernel source for 6.1 and see if that's wrong? If it is, and it's back in place in 6.2 then everything should be peachy. "the freely available XenServer 5.6 wants to delete my entire hard drive when I try to install it" I can see how that would put you off... "Boot from CD using RHEL template" Yes, when you go to create a new VM, you get to pick from a list. One of which is for RHEL 6.x derived (32 or 62 bit). "it's actually surprising that it ever worked." We do go to quite some lengths to make it work :-)
Ah yes, we pulled in 7901d14 xen/blkfront: Use QUEUE_ORDERED_DRAIN for old backends during 6.1 development, but never 4352b47 xen-blkfront: fix data size for xenbus_gather in blkfront_connect so that is likely the culprit. Thanks for tracking it down for us. I'll create a kernel with this patch today and make it available for testing.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
I've created a RHEL6 kernel rpm with the patch and put it here http://people.redhat.com/drjones/kernel-2.6.32-170.el6716452.x86_64.rpm sorry, only 64-bit, atm. I'll create a 32-bit, as well though. Thanks in advance for any testing done.
32-bit rpm: http://people.redhat.com/drjones/kernel-2.6.32-170.el6716452.i686.rpm
(In reply to comment #17) > I've created a RHEL6 kernel rpm with the patch and put it here > > http://people.redhat.com/drjones/kernel-2.6.32-170.el6716452.x86_64.rpm > > sorry, only 64-bit, atm. I'll create a 32-bit, as well though. Thanks in > advance for any testing done. I can reproduce the issue with xenserver, but the fix doesn't work for me. I'm not sure whether the test approach is correct to verify the fix: [1] unpack the isolinux/initrd.img from the RHEL6.1 GA iso image. [2] replace the xen-blkfront.ko.gz file extracted from initrd.img with the one which the fix has been applied [3] re-pack the initrd.img and rebuild the iso image, boot up the vm (using RHEL6 template) via iso image, there is still no driver for block device, same output as RHEL6.1 GA during booting: ... XENBUS: Device with no driver: device/vbd/51712 XENBUS: Device with no driver: device/vbd/51760 XENBUS: Device with no driver: device/vif/0 ... I also tried to re-compile the xen-blkfront module after apply the patch to RHEL6.1 GA kernel, so the module has the same version with GA kernel, then the same steps as above, but still failed. So QE may need to try this fix after RHEL6.2 snapshot (with iso images composed) come out.
Patch(es) available on kernel-2.6.32-176.el6
Verified with RHEL6.2-20110823.1 (i386 and x86_64) iso images. kernel version is kernel-2.6.32-191.el6. XenServer is 5.6.100-47101p. The vbd disk can be found during install, both the i386 and x86_64 guests can be installed and boot up after reboot successfully. So the fix works properly. Change this bug to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2011-1530.html