Bug 621043

Summary: DUD loaded from initramfs-based PXE/TFTP image failed
Product: Red Hat Enterprise Linux 6 Reporter: Chao Yang <chyang>
Component: anacondaAssignee: Martin Sivák <msivak>
Status: CLOSED CURRENTRELEASE QA Contact: Release Test Team <release-test-team-automation>
Severity: high Docs Contact:
Priority: high    
Version: 6.0CC: andriusb, bzeranski, chyang, coughlan, ddumas, hhoyer, jcm, msivak, qcai, syeghiay
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-11-11 15:25:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
VM define file.
none
/tmp/ directory of the VM
none
anaconda.log for load driver from harddisk.
none
syslog for load driver from harddisk.
none
/tmp/ dir for load driver image file from nfs server.
none
/tmp/ dir for load driver from http server.
none
load driver from http server screenshot.
none
load driver from http server /tmp/ folder, including anaconda.log, syslog etc.
none
load driver from http server VM config file.
none
screenshot of dud from TFTP.
none
syslog for dud from TFTP.
none
anaconda.log for dud from TFTP.
none
PXE/TFTP driver update after install check i386.
none
PXE/TFTP driver update after install check x86_64. none

Description Chao Yang 2010-08-04 05:58:25 UTC
Description of problem:
DUD loaded from FTP failed.

Version-Release number of selected component (if applicable):


How reproducible:
Always.

Steps to Reproduce:
1. Setup a ftp file server;

2. Make a driverdisk against 2.6.32-54 kernel;
###################################################
[root@ts1 ~]# cd /var/ftp/pub/
[root@ts1 pub]# ll
total 1972
-rw-r--r--. 1 root root 1021952 Aug  4 13:39 dd-54.iso
-rw-r--r--. 1 root root  995328 Aug  3 19:50 dd_sym53c8xx-rhel6.0-52.iso
[root@ts1 pub]# losetup -f dd-54.iso 
[root@ts1 pub]# losetup -a
/dev/loop0: [fd00]:1835172 (/var/ftp/pub/dd-54.iso)
[root@ts1 pub]# mount /dev/loop0 /mnt/
[root@ts1 pub]# rpm -qp --provides /mnt/rpms/x86_64/kmod-sym53c8xx-2.2.3.rhtest60b1-1.el6.x86_64.rpm 
kernel-modules = 2.6.32-54.el6.x86_64
sym53c8xx-kmod = 2.2.3.rhtest60b1-1.el6
kmod-sym53c8xx = 2.2.3.rhtest60b1-1.el6
kmod-sym53c8xx(x86-64) = 2.2.3.rhtest60b1-1.el6
[root@ts1 pub]# rpm -qp --provides /mnt/rpms/x86_64/kmod-sym53c8xx-debug-2.2.3.rhtest60b1-1.el6.x86_64.rpm 
kernel-modules = 2.6.32-54.el6.x86_64.debug
sym53c8xx-kmod = 2.2.3.rhtest60b1-1.el6
kmod-sym53c8xx-debug = 2.2.3.rhtest60b1-1.el6
kmod-sym53c8xx-debug(x86-64) = 2.2.3.rhtest60b1-1.el6
[root@ts1 pub]# 
###################################################

3. Create a VM in qemu with a SCSI disk;

4. Load DUD from ftp when boot.
###################################################
bash-4.1# busybox cat /sys/module/sym53c8xx/version
2.2.3
bash-4.1# busybox ls /tmp/DD/lib/modules/2.6.32-54.el6.x86_64/extra/sym53c8xx/sym53c8xx.ko
/tmp/DD/lib/modules/2.6.32-54.el6.x86_64/extra/sym53c8xx/sym53c8xx.ko
bash-4.1# rmmod sym53c8xx
bash-4.1# modprobe sym53c8xx
bash-4.1# busybox cat /sys/module/sym53c8xx/version
2.2.3rhtest60b1
bash-4.1# uname -r
2.6.32-54.el6.x86_64
###################################################

Actual results:
The driver have not been loaded.

Expected results:
It can be loaded from network location.

Additional info:
It seems like the file transfer from ftp is done, and the DUD got unpacked. But fail to load. I will try to perform the test on other driver.

Comment 1 Chao Yang 2010-08-04 06:00:37 UTC
Created attachment 436440 [details]
VM define file.

Comment 2 Chao Yang 2010-08-04 06:02:27 UTC
Created attachment 436441 [details]
/tmp/ directory of the VM

Comment 4 RHEL Program Management 2010-08-04 06:27:35 UTC
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **

Comment 5 Chao Yang 2010-08-04 08:00:10 UTC
Hi,

I also tried to load the driverdisk from harddisk, the same DUD and VM. The driver had been loaded as expected. 

*****************************
bash-4.1# busybox cat /sys/module/sym53c8xx/version
2.2.3rhtest60b1
bash-4.1#
*****************************

Comment 6 Chao Yang 2010-08-04 08:04:55 UTC
Created attachment 436463 [details]
anaconda.log for load driver from harddisk.

Comment 7 Chao Yang 2010-08-04 08:05:20 UTC
Created attachment 436464 [details]
syslog for load driver from harddisk.

Comment 8 Chao Yang 2010-08-10 07:20:39 UTC
Try NFS method, also failed to load. We can see the driver is download and unpacked from the nfs location, but didn't get loaded.

-----------------------------------------
bash-4.1# busybox cat /sys/module/sym53c8xx/version
2.2.3
bash-4.1# rmmod sym53c8xx
bash-4.1# modprobe sym53c8xx
bash-4.1# busybox cat /sys/module/sym53c8xx/version
2.2.3.rhtest60b1

Attached the /tmp/ dir for ref. Thanks.

Best,
Chao

Comment 9 Chao Yang 2010-08-10 07:22:15 UTC
Created attachment 437786 [details]
/tmp/ dir for load driver image file from nfs server.

Comment 10 Jon Masters 2010-08-10 08:38:22 UTC
Re-assign to Anaconda. I suspect it's very similar to some of the other issues. Note that this says FTP and not TFTP so it's perhaps not the lack of re-running depmod from within the initramfs, and besides, it looks like the driver is put in place but just not loaded. Weird. Anyway, it's an Anaconda issue I think :)

Comment 11 Chao Yang 2010-08-10 08:42:26 UTC
Created attachment 437810 [details]
/tmp/ dir for load driver from http server.

Comment 12 Chao Yang 2010-08-10 08:48:46 UTC
What I have observed here:

1. Failed to load driver disk from all three network locations(HTTP, FTP, NFS). 
2. Manually modprobe can load the update driver.

This is a basic function for DUD, give high Priority. If you need any further information please let me know. Thanks.

Best,
Chao

Comment 13 Martin Sivák 2010-08-10 09:06:21 UTC
If you check the /lib/modules/<kernel version>/modules.dep, can you see the module there with path in updates/DD dir? (should be)

And is there any other line with that driver not being in updates dir? (shouldn't be)

Also do you see the module being present in /tmp/DD? (should be)

If all my assertions are valid, then it doesn't seem to be Anaconda itself and I need log files, content of /tmp/DD and /tmp/DD-0 and detailed reproduction steps to investigate.

Comment 14 Chao Yang 2010-08-10 11:45:15 UTC
Hi Martin,

I Just checked the files, and here is answers.

(In reply to comment #13)
> If you check the /lib/modules/<kernel version>/modules.dep, can you see the
> module there with path in updates/DD dir? (should be)
> 

bash4-1# grep updates /lib/modules/2.6.32-59.1.el6.x86_64/modules.dep
updates/DD/2.6.32-59.1.el6.x86_64/extra/sym53c8xx/sym53c8xx.ko: kernel/drivers/scsi/scsi_transport_spi.ko.gz

> And is there any other line with that driver not being in updates dir?
> (shouldn't be)
> 

Nothing else.

> Also do you see the module being present in /tmp/DD? (should be)
> 

bash4-1# busybox ls /tmp/DD/lib/modules/2.6.32-59.1.el6.x86_64/extra/sym53c8xx/sym53c8xx.ko
/tmp/DD/lib/modules/2.6.32-59.1.el6.x86_64/extra/sym53c8xx/sym53c8xx.ko

> If all my assertions are valid, then it doesn't seem to be Anaconda itself and
> I need log files, content of /tmp/DD and /tmp/DD-0 and detailed reproduction
> steps to investigate.    

You can find log files, content of /tmp/DD and /tmp/DD-0 and in the attachment tarball files. 

How to reproduce:
1. config HTTP/FTP/NFS server with 2.6.32-59 version DUD.
2. modify virtual machine define xml file, add following lines:
<cmdline>linux dd=http://10.66.65.116/dd-59.iso</cmdline>    # Using http
<cmdline>linux dd=ftp://10.66.65.116/pub/dd-59.iso</cmdline> # Using ftp
<cmdline>linux dd=nfs:10.66.65.116:/nfs-repo/dd-59.iso</cmdline>  Using nfs
3. virsh define <vm xml file> to register vm to your virt-manager
4. start vm and the driver should be loaded automaticly.

Anything missing or wrong please let me know. Thanks a lot!

Best,
Chao

Comment 15 Chao Yang 2010-08-10 11:50:44 UTC
Hi Martin, 

Also you can get my xml define file in the attachment. 

Best,
Chao

Comment 17 Jon Masters 2010-08-10 22:38:59 UTC
There is another bug you should also be watching that is related, but not quite the same issue. That is that when building an initramfs, dracut does not include the "updates" directory. It does now include the "extra" and "weak-updates" directory as of today, and I am hoping the other directory for tomorrow.

https://bugzilla.redhat.com/show_bug.cgi?id=622641

Jon.

Comment 18 Jon Masters 2010-08-10 22:40:33 UTC
For clarity, comment 17 is just an FYI. It is not intended to address this.

Comment 20 David Cantrell 2010-08-17 00:11:42 UTC
Please retest using a current nightly tree containing anaconda-13.21.77-1 or later.

Comment 21 Jon Masters 2010-08-17 05:34:10 UTC
This will be done with the compose that completes on 20100817.

Jon.

Comment 22 Jon Masters 2010-08-17 07:07:46 UTC
Chao: can you test this please with the 2010817 when it shows up in a few hours?

Jon.

Comment 23 Jon Masters 2010-08-17 07:08:07 UTC
Thanks :)

Comment 24 Chao Yang 2010-08-17 07:31:51 UTC
Sure, when will the 201080817 tree available? 

Best,
Chao

Comment 25 Jon Masters 2010-08-17 07:57:09 UTC
TEST RESULTS
------------

The updates image supplied by Martin does get through the install, HOWEVER, the
kmod is not installed on the finish system and the updated driver is not used:

$ rpm -qa | grep kmod
<nothing>

This urgently requires fixing. Please investigate why the kmod RPM is not
targeted for installation, and perform any necessary testing.

Thanks for initial fix - we're half way there :)

Jon.

Comment 27 Chao Yang 2010-08-17 12:26:18 UTC
Just using the latest 20100817 tree (http://download.devel.redhat.com/nightly/RHEL6.0-20100817.n.0/6/Server/x86_64/os/) to reproduce. I found that driver still can't be loaded from http. But this time, instead of the error message, the installer prompt you to choose a language. And also try to load from CDROM. loaded correctly. 

Best,
Chao

Comment 28 Chao Yang 2010-08-17 12:27:02 UTC
Created attachment 439094 [details]
load driver from http server screenshot.

Comment 29 Chao Yang 2010-08-17 12:28:20 UTC
Created attachment 439095 [details]
load driver from http server /tmp/ folder, including anaconda.log, syslog etc.

Comment 30 Chao Yang 2010-08-17 12:29:01 UTC
Created attachment 439096 [details]
load driver from http server VM config file.

Comment 31 David Cantrell 2010-08-17 14:54:39 UTC
Workaround is to load driver disks from CD-ROM as stated in comment #27.  No data corruption occurs because of this bug, so given that we have that and a valid workaround, this doesn't meet remaining 6.0 blocker criteria.  Moving this one to 6.1.

Comment 32 Martin Sivák 2010-08-17 15:10:59 UTC
According to my testing, adding new drivers from network based driverdisks should work. Only updates are problematic, because we cannot refresh all loaded drivers due to NM being already started (it tracebacked when I tried).

Comment 34 Jon Masters 2010-08-17 17:41:20 UTC
Addendum to previous comment: It is ok to ship 6.0 without support for loading driver disks from FTP/HTTP, however we *must* verify that TFTP based (when contained within the initramfs) updates work at a minimum.

Jon.

Comment 35 Chao Yang 2010-08-18 04:20:19 UTC
I have just verified TFTP case, working fine. Here is my steps to verify:
1. Setup the PXE/TFTP server in the host.
2. Set a VM to boot from PXE, also change the network to a special pxe network.
3. Modify initrd image file, two things to do here:
    (1) Add "busybox" tool to bin;
    (2) Add dd.iso file;
   Following the instruction: "Anaconda Initrd Update" (http://intranet.corp.redhat.com/ic/intranet/AnacondaInitrdUpdate.html)
4. Boot the VM from PXE.
5. Driver updated correctlly.

Attach screenshot, syslog and anaconda.log.


Best,
Chao

Comment 36 Chao Yang 2010-08-18 04:20:52 UTC
Created attachment 439296 [details]
screenshot of dud from TFTP.

Comment 37 Chao Yang 2010-08-18 05:10:50 UTC
Created attachment 439299 [details]
syslog for dud from TFTP.

Comment 38 Chao Yang 2010-08-18 05:11:19 UTC
Created attachment 439300 [details]
anaconda.log for dud from TFTP.

Comment 39 Chao Yang 2010-08-18 06:11:34 UTC
Since the TFTP is working, remove the blocker ? flag. Thanks.

Best,
Chao

Comment 43 Jon Masters 2010-08-18 13:47:27 UTC
Chao: we need to test this with the 20100818 compose or a later version. That is currently being built, so in a few hours we can test it with the full nightly.

Jon.

Comment 48 Chao Yang 2010-08-19 13:35:10 UTC
Verified on i386 and x86_64 arch, driver update from PXE/TFTP works as expected.
So I suppose we are done here.

Best,
Chao

Comment 49 Chao Yang 2010-08-19 13:38:43 UTC
Sorry to forget the tree info: 20100818. Thanks.

Best,
Chao

Comment 50 Jon Masters 2010-08-19 20:08:00 UTC
Can you include a screenshot or some data from the install? Can you run:

$ rpm -qa | grep kmod

After the install and show that the module was loaded? Also:

$ modinfo <module name>

To show the right one is being picked up?

Thanks.

Comment 56 Jon Masters 2010-08-20 18:46:22 UTC
Thanks. I am willing to consider this VERIFIED now.

Comment 57 releng-rhel@redhat.com 2010-11-11 15:25:17 UTC
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.