Bug 613576 - Failed to install RHEL6.0 HVM guest from DVD ISO file
Failed to install RHEL6.0 HVM guest from DVD ISO file
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: udev (Show other bugs)
6.0
All Linux
high Severity high
: rc
: ---
Assigned To: Harald Hoyer
qe-baseos-daemons
: Regression, TestBlocker
Depends On:
Blocks: 464065 622763
  Show dependency treegraph
 
Reported: 2010-07-12 06:17 EDT by Lei Wang
Modified: 2010-11-10 16:51 EST (History)
10 users (show)

See Also:
Fixed In Version: udev-147-2.25.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 622763 (view as bug list)
Environment:
Last Closed: 2010-11-10 16:51:02 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
xm config file (722 bytes, text/plain)
2010-07-12 06:17 EDT, Lei Wang
no flags Details
error message box (91.42 KB, image/png)
2010-07-12 06:19 EDT, Lei Wang
no flags Details
more detailed info but not enough for we can't scroll up the screen (104.18 KB, image/png)
2010-07-12 06:26 EDT, Lei Wang
no flags Details
/tmp/anaconda.log (9.15 KB, text/plain)
2010-07-12 21:29 EDT, Lei Wang
no flags Details
/tmp/syslog (43.02 KB, text/plain)
2010-07-12 21:29 EDT, Lei Wang
no flags Details
/tmp/storage.log (121.31 KB, text/plain)
2010-07-12 21:30 EDT, Lei Wang
no flags Details
anaconda-613576.log (9.39 KB, text/plain)
2010-08-02 22:48 EDT, Lei Wang
no flags Details
all the log files under /tmp (124.32 KB, application/x-gzip)
2010-08-04 23:06 EDT, Lei Wang
no flags Details

  None (edit)
Description Lei Wang 2010-07-12 06:17:41 EDT
Created attachment 431129 [details]
xm config file

Description of problem:
Install RHEL6.0 HVM guest from DVD ISO file will fail with error "Unable to read group information from repositories.This is a problem with the generation of your install tree".

Version-Release number of selected component (if applicable):
RHEL6 Beta2 snapshot7 (RHEL6.0-20100707.4)
xen-3.0.3-113.el5
kernel-xen-2.6.18-204.el5

How reproducible:
Always

Steps to Reproduce:
1.xm create xm-test.conf (config file as attached)
2.virt-viewer vm1
3.installation started, "Next" all the way except "Skip" the media test
4.choose "Replace Existing Linux System(s)" as default, click "Next"
  
Actual results:
After format and create filesystem, error message box popped up:
"Unable to read group information from repositories.This is a problem with the generation of your install tree"

Expected results:
No error occurs, installation should finish.

Additional info:
1.Install RHEL6.0 HVM guest via boot.iso with http/ftp tree will success.
2.Use kvm to install RHEL6.0 from same version DVD ISO file will success.
3.The older version RHEL6.0-20100622.1 can be installed successfully to HVM guest.
Comment 1 Lei Wang 2010-07-12 06:19:11 EDT
Created attachment 431130 [details]
error message box
Comment 3 Lei Wang 2010-07-12 06:26:00 EDT
Created attachment 431133 [details]
more detailed info but not enough for we can't scroll up the screen
Comment 5 Chris Lumens 2010-07-12 09:55:51 EDT
Please attach /tmp/anaconda.log, /tmp/syslog, and /tmp/storage.log from your failed installation to this bug report.  Thanks.
Comment 6 Lei Wang 2010-07-12 21:29:15 EDT
Created attachment 431316 [details]
/tmp/anaconda.log
Comment 7 Lei Wang 2010-07-12 21:29:42 EDT
Created attachment 431317 [details]
/tmp/syslog
Comment 8 Lei Wang 2010-07-12 21:30:16 EDT
Created attachment 431318 [details]
/tmp/storage.log
Comment 9 Lei Wang 2010-07-12 21:33:31 EDT
Hi, Chris

Logs are attached as above, please check.
Comment 10 Chris Lumens 2010-07-13 13:49:33 EDT
This looks like some sort of kernel issue to me, judging by the error messages in your syslog.  Media check can catch these sorts of errors sometimes, but not always.

<7>ata2: drained 18 bytes to clear DRQ.
<3>ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
<3>ata2.00: BMDMA stat 0x5
<6>sr 1:0:0:0: [sr0] CDB: Inquiry: 12 00 00 00 fe 00
<3>ata2.00: cmd a0/01:00:00:fe:00/00:00:00:00:00/a0 tag 0 dma 16640 in
<3>         res 48/00:02:00:24:00/00:00:00:00:00/a0 Emask 0x2 (HSM violation)
<3>ata2.00: status: { DRDY DRQ }
<6>ata2: soft resetting link
<6>ata2.00: configured for MWDMA2
<6>ata2: EH complete
Comment 11 Lei Wang 2010-07-26 04:37:08 EDT
Retest with:

RHEL6 Beta2 snapshot8 (RHEL6.0-20100722.0)
xen-3.0.3-114.el5
kernel-xen-2.6.18-208.el5

The issue described in Bug Description still exists.
Comment 12 Lei Wang 2010-07-26 05:28:46 EDT
According to the regression bug 618144, install RHEL6.0 snapshot8 via boot.iso with http/ftp tree will fail now.
Comment 13 Andrew Jones 2010-07-27 12:39:44 EDT
I thought this would be a dup of bug 618172. The error messages indicate that
we can't get to a repo, which would make sense since the network interface
isn't working.  I reproduced it the problem and then jumped over to vterm 2 to
check ifconfig, there was no ethernet device configured, no routes. I 'modprobe
-r'ed the realtek driver and then manually set up the vnif with ifconfig/route
and a nameserver in resolv.conf. The jumped back to vterm 1 and tried to
proceed with the install, but got the same message.

It doesn't look like a kernel issue to me though (besides the ethernet stuff),
so possibly manually setting up the vnif isn't sufficient, but it's still a dup
of that.

Note, the most interesting message I see in the vterm logs is

ERROR : Error downloading None/.treeinfo: [Error 14] Could not open/read
file:///None/.treeinfo
Comment 14 Qixiang Wan 2010-07-27 23:33:49 EDT
(In reply to comment #13)
> I thought this would be a dup of bug 618172. The error messages indicate that
> we can't get to a repo, which would make sense since the network interface
> isn't working.  I reproduced it the problem and then jumped over to vterm 2 to
> check ifconfig, there was no ethernet device configured, no routes. I 'modprobe
> -r'ed the realtek driver and then manually set up the vnif with ifconfig/route
> and a nameserver in resolv.conf. The jumped back to vterm 1 and tried to
> proceed with the install, but got the same message.

Install from DVD should not require network interface to be up, the repo info should be retrieved from the DVD image. So it doesn't seem to be dup of bug 618172, but not sure whether it's a kernel issue.

> 
> It doesn't look like a kernel issue to me though (besides the ethernet stuff),
> so possibly manually setting up the vnif isn't sufficient, but it's still a dup
> of that.
> 
> Note, the most interesting message I see in the vterm logs is
> 
> ERROR : Error downloading None/.treeinfo: [Error 14] Could not open/read
> file:///None/.treeinfo    

we can see this error message while installing it on kvm hypervisor, but the installation can get pass on kvm.
Comment 17 Andrew Jones 2010-07-28 04:53:34 EDT
I was able to successfully 'yum update' my rhel6 hvm guest and boot/run fine. The error message pointed out in comment 10 has been around for all of fedora and rhel6 xendomu's life. It's verbose, looks disconcerting, but is harmless. We really need to debug this issue from the origin of the main problem, the halted install and error message from comment 1. I'm moving this back to anaconda as they have the tools to find the source of that error.
Comment 18 Brian Lane 2010-07-28 13:05:05 EDT
Have you run the media check on the iso image? 

syslog is showing errors when accessing the virtual cdrom device which is the likely cause of it not finding the install repository. This means that there may be a problem with the DVD image or the host system.

Have you tried a newer DVD iso?

Have you tried with the same image on different host hardware?

It may be possible that you are hitting bug 607650, depending on what the host system is.
Comment 19 Qixiang Wan 2010-07-28 23:11:58 EDT
(In reply to comment #18)
> Have you run the media check on the iso image? 
> 
> syslog is showing errors when accessing the virtual cdrom device which is the
> likely cause of it not finding the install repository. This means that there
> may be a problem with the DVD image or the host system.
> 
> Have you tried a newer DVD iso?
> 
> Have you tried with the same image on different host hardware?
> 
> It may be possible that you are hitting bug 607650, depending on what the host
> system is.    

No problem with the iso image, media check passed.

Install guest with the same iso on kvm hypervisor also passed.
Comment 20 Miroslav Rezanina 2010-07-29 02:59:51 EDT
Testing shows that media is ok and visible in guest. However, in compare with beta1 installation, there's no /mnt/source directory mounted. 

There's also strange message in anaconda.log - 

notifying kernel of 'change' event on device /sys/devices/pci0000:00/0000:00:01.1/host1/target1:0:0/1:0:0:0/block/sr0
Comment 21 Brian Lane 2010-08-02 17:12:03 EDT
We need some more detail about the error. Please re-run your install and pass:

updates=http://bcl.fedorapeople.org/updates/613576.img

to the kernel and then attach the anaconda.log file. This update should log more information as to why the exception is being triggered.
Comment 22 Lei Wang 2010-08-02 22:48:13 EDT
Created attachment 436174 [details]
anaconda-613576.log

Retest with:

updates=http://bcl.fedorapeople.org/updates/613576.img

passed to kernel.

Version-Release number of selected component (if applicable):
RHEL6 Beta2 snapshot8 (RHEL6.0-20100722.0)
xen-3.0.3-114.el5
kernel-xen-2.6.18-210.el5

According to 618172, add "type=netfront" to vif specification in xen guest config file or there'll be network error, see as below:
vif = ['type=netfront,mac=00:AE:77:13:4B:31,script=vif-bridge,bridge=xenbr0,model=rtl8139']

Others keep unchanged.

New anaconda.log as attached.
Comment 23 Brian Lane 2010-08-03 19:10:07 EDT
It looks like the install DVD is no longer connected, the last message in storage.log

21:25:26,182 DEBUG   : no type or existing type for sr0, bailing

means that it didn't find the install media, hence the error in anaconda.log

21:56:46,392 ERROR   : Error downloading None/.treeinfo: [Errno 14] Could not open/read file:///None/.treeinfo
21:56:46,801 DEBUG   : No Groups Available in any repository

I really don't think this is an anaconda problem, it only happens with your Xen setup, not KVM and we have had no other reports like this that I have seen.
Comment 24 Andrew Jones 2010-08-04 09:06:11 EDT
On the xen side we've already proven that the media is accessible
and working. You can access it directly mount/umount, etc. from vterm 2.
Furthermore, we've seen that Beta2-5.0 installs fine from DVD iso, but
snapshot 7 (and 8 as reported in this bug) don't, even though the Xen
host stays exactly the same, and the xen-specific changes for HVM RHEL6
guests in the kernel are turned off by default for snapshot 7 (you need
a command line option to enable them).

This problem may be specific to xen, but we still need to know why exactly,
i.e. why do we get "no type or existing type for sr0, bailing", when sr0
works from vterm 2, and why do we get the log Miroslav pointed out?
"notifying kernel of 'change' event on device". Does the change event come
from udev? Who caused it? What is the change? We need to dig deeper starting
from the top (anaconda) to find the source of these messages in order to
find the root cause. Or, possibly more simply, we need to bisect anaconda
keeping everything else constant to see when the install started failing.
Comment 25 Brian Lane 2010-08-04 12:42:43 EDT
ok, thanks for the additional background.

I notice we don't have the program.log file, that may be helpful. And while you are at it you may as well make a full traceback and attach that too. Go to tty2 after it fails to install and hit up arrow a couple of times until you see a kill -USR2 line and execute that. It will make an anaconda-tb-* file with the internal state dumped to it.

The change event is triggered by anaconda when it tries to read the format of /dev/sr0

iutil.notify_kernel("/sys%s" % device.sysfsPath)
udev_settle()

program.log will show us calling udevadm settle --timeout=300
Comment 26 Lei Wang 2010-08-04 23:06:06 EDT
Created attachment 436724 [details]
all the log files under /tmp

Retest with:

updates=http://bcl.fedorapeople.org/updates/613576.img

passed to kernel.

Version-Release number of selected component (if applicable):
RHEL6 Beta2 snapshot9 (RHEL6.0-20100730.5)
xen-3.0.3-114.el5
kernel-xen-2.6.18-210.el5

Attach all the log files under /tmp, including program.log and anaconda-tb-* file.
Comment 27 Brian Lane 2010-08-05 13:56:49 EDT
Could you change to tty2 and paste the output of:

blkid -p -o udev /dev/sr0

into this bug?
Comment 28 Brian Lane 2010-08-05 13:57:39 EDT
Sorry for the bugspam, and the output of:

udevadm info --query=all --name=sr0 | grep ID_FS_TYPE

Thanks!
Comment 29 Lei Wang 2010-08-05 22:13:45 EDT
Hi, Brian

Use same steps and component version with comment 26.

** Output of command "blkid -p -o udev /dev/sr0": **

ID_FS_LABEL=CDROM
ID_FS_LABEL_ENC=CDROM
ID_FS_TYPE=iso9660
ID_FS_USAGE=filesystem

** Output of command "udevadm info --query=all --name=sr0": **
(There's no output when grep ID_FS_TYPE, so put the output without grep here.)

P: /devices/pci0000:00/0000:00:01.1/host1/target1:0:0/1:0:0:0/block/sr0
N: sr0
S: block/11:0
S: scd0
S: disk/by-path/pci-0000:00:01.1-scsi-1:0:0:0
E: UDEV_LOG=3
E: DEVPATH=/devices/pci0000:00/0000:00:01.1/host1/target1:0:0/1:0:0:0/block/sr0
E: MAJOR=11
E: MINOR=0
E: DEVNAME=/dev/sr0
E: DEVTYPE=disk
E: SUBSYSTEM=block
E: MPATH_SBIN_PATH=/usr/sbin
E: ID_CDROM=1
E: ID_CDROM_MEDIA=1
E: ID_PATH=pci-0000:00:01.1-scsi-1:0:0:0
E: ACL_MANAGE=1
E: ANACBIN=/usr/sbin
E: DEVLINKS=/dev/block/11:0 /dev/scd0 /dev/disk/by-path/pci-0000:00:01.1-scsi-1:0:0:0
Comment 30 Brian Lane 2010-08-06 14:53:16 EDT
Thanks, this looks like a udev problem.
Comment 31 Harald Hoyer 2010-08-09 05:47:45 EDT
what is the output of:

/lib/udev/cdrom_id --debug /dev/sr0
Comment 32 Harald Hoyer 2010-08-09 05:52:16 EDT
ID_CDROM_MEDIA_TRACK_COUNT_DATA is missing, so I guess the virtual CDROM, does not implement all functions of a CDROM.
Comment 34 Lei Wang 2010-08-09 21:45:36 EDT
(In reply to comment #31)
> what is the output of:
> 
> /lib/udev/cdrom_id --debug /dev/sr0    

**Output of command "/lib/udev/cdrom_id --debug /dev/sr0"**

main: probing: '/dev/sr0'
cd_inquiry: INQUIRY: [QEMU    ][QEMU CD-ROM     ][0.8.]
cd_profiles: GET CONFIGURATION: number of profiles 4
cd_profiles: current profile 0x00
cd_profiles: no current profile, assuming no media
ID_CDROM=1
ID_CDROM_MEDIA=1
Comment 35 Harald Hoyer 2010-08-10 04:20:29 EDT
I can only work around the incomplete implementation of the virtual cdrom... I will have to add quirks with unknown side effects..
Comment 36 Andrew Jones 2010-08-10 04:29:15 EDT
We can still consider fixing this in Xen. If you help us by pinpointing exactly what's missing, then we can see how difficult it would be to patch ioemu.
Comment 37 Harald Hoyer 2010-08-10 04:45:09 EDT
cdrom_id wants the following scsi commands:

0x43 READ TOC
0x46 GET CONFIGURATION
0x51 READ DISC INFORMATION
Comment 38 Harald Hoyer 2010-08-10 04:48:18 EDT
oh, forgot:

0x12 INQUIRY

but that seems to be ok.
Comment 39 Harald Hoyer 2010-08-10 04:50:32 EDT
(In reply to comment #34)
> (In reply to comment #31)
> > what is the output of:
> > 
> > /lib/udev/cdrom_id --debug /dev/sr0    
> 
> **Output of command "/lib/udev/cdrom_id --debug /dev/sr0"**
> 
> main: probing: '/dev/sr0'
> cd_inquiry: INQUIRY: [QEMU    ][QEMU CD-ROM     ][0.8.]
> cd_profiles: GET CONFIGURATION: number of profiles 4
> cd_profiles: current profile 0x00
> cd_profiles: no current profile, assuming no media

It should at least return "current profile" == 0x08
Comment 40 Andrew Jones 2010-08-10 07:17:48 EDT
Hmm, actually it was kind of dumb of me to suggest we alone can fix this on the xen side. We probably should fix it on the xen side, but in order to preserve backwards compatibility (i.e. rhel6 guest iso installs on rhel 5.5 and down), we need to also get the quirks put in to work around this problem.  Let's use this bug for those quirks and I'll also clone it to xen userspace to consider the fix on the xen side for future xen releases.

Drew
Comment 41 Andrew Jones 2010-08-11 03:09:56 EDT
BTW, I'm still a bit confused as to how this issue got exposed now. The xen cdrom has been this way forever, so something in udev of recent rhel6 snapshots must have changed?
Comment 42 Harald Hoyer 2010-08-11 03:33:12 EDT
(In reply to comment #41)
> BTW, I'm still a bit confused as to how this issue got exposed now. The xen
> cdrom has been this way forever, so something in udev of recent rhel6 snapshots
> must have changed?    

* Tue Jun 29 2010 Harald Hoyer <harald@redhat.com> 147-2.19
- do not blkid blank or audio CDROMs 
Resolves: rhbz#606293
Comment 43 Harald Hoyer 2010-08-11 04:17:39 EDT
udev-147-2.23.el6
Comment 44 Harald Hoyer 2010-08-11 04:42:27 EDT
please provide the output of:

# /lib/udev/cdrom_id --debug /dev/sr0    

with udev-147-2.23.el6
Comment 45 Harald Hoyer 2010-08-11 05:31:07 EDT
udev-147-2.24.el6
Comment 46 Harald Hoyer 2010-08-11 06:10:34 EDT
udev-147-2.25.el6
Comment 47 Harald Hoyer 2010-08-11 06:13:44 EDT
# /lib/udev/cdrom_id --debug /dev/sr0
main: probing: '/dev/sr0'
cd_inquiry: INQUIRY: [QEMU    ][QEMU CD-ROM     ][0.8.]
cd_profiles: GET CONFIGURATION: number of profiles 4
cd_profiles: current profile 0x00
cd_profiles: no current profile, but CDS_DISC_OK, assuming incomplete implementation
cd_media_toc: READ TOC: len: 20
cd_media_toc: track=1 info=0x4(data) start_block=0
cd_media_toc: last track 1 starts at block 0
cd_media_info: disk type 00
ID_CDROM=1
ID_CDROM_MEDIA=1
ID_CDROM_MEDIA_CD=1
ID_CDROM_MEDIA_TRACK_COUNT_DATA=1

# echo change > /sys/block/sr0/uevent 
# udevadm info --query=all --name=sr0
P: /devices/pci0000:00/0000:00:01.1/host1/target1:0:0/1:0:0:0/block/sr0
N: sr0
S: block/11:0
S: scd0
S: disk/by-path/pci-0000:00:01.1-scsi-1:0:0:0
S: disk/by-label/RHEL_6.0\x20i386\x20Disc\x201
S: cdrom
E: UDEV_LOG=3
E: DEVPATH=/devices/pci0000:00/0000:00:01.1/host1/target1:0:0/1:0:0:0/block/sr0
E: MAJOR=11
E: MINOR=0
E: DEVNAME=/dev/sr0
E: DEVTYPE=disk
E: SUBSYSTEM=block
E: ID_CDROM=1
E: ID_CDROM_MEDIA=1
E: ID_CDROM_MEDIA_CD=1
E: ID_CDROM_MEDIA_TRACK_COUNT_DATA=1
E: ID_PATH=pci-0000:00:01.1-scsi-1:0:0:0
E: ID_FS_LABEL=RHEL_6.0_i386_Disc_1
E: ID_FS_LABEL_ENC=RHEL_6.0\x20i386\x20Disc\x201
E: ID_FS_TYPE=iso9660
E: ID_FS_USAGE=filesystem
E: ACL_MANAGE=1
E: GENERATED=1
E: DEVLINKS=/dev/block/11:0 /dev/scd0 /dev/disk/by-path/pci-0000:00:01.1-scsi-1:0:0:0 /dev/disk/by-label/RHEL_6.0\x20i386\x20Disc\x201 /dev/cdrom

# udevadm info --query=all --name=sr0 | grep ID_FS_TYPE
E: ID_FS_TYPE=iso9660


so, all should be fine now
Comment 49 Lei Wang 2010-08-16 03:51:23 EDT
Tested with:
RHEL6.0 guest:(both i386 and x86_64)
RHEL6.0 snapshot11(20100811.2)
udev-147-2.25.el6

RHEL5.5 host:(both i386 and x86_64)
xen-3.0.3-115.el5
kernel-xen-2.6.18-212.el5

The xen RHEL6 guest installation from DVD ISO completed successfully without "group information" error,also can boot up after installation, so verify this bug.
Comment 50 releng-rhel@redhat.com 2010-11-10 16:51:02 EST
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.