Red Hat Bugzilla – Bug 951602
RHEV-H environment is missing kernel modules, breaking kdump over network
Last modified: 2014-01-21 14:31:41 EST
Description of problem: kdump via NFS fails with current RHEV-H on systems using a Intel Corporation I350 Gigabit Network Connection NIC as the driver for this NIC cannot be loaded in the kdump environment. Version-Release number of selected component (if applicable): rhev-hypervisor6-6.4-20130318.1.iso How reproducible: 100% Steps to Reproduce: 1. SSH into RHEV-H node 2. "modprobe igb" Actual results: [root@dhcp145-234 ~]# modprobe igb WARNING: Could not open '/lib/modules/2.6.32-358.2.1.el6.x86_64/kernel/drivers/pps/pps_core.ko': No such file or directory WARNING: Could not open '/lib/modules/2.6.32-358.2.1.el6.x86_64/kernel/drivers/ptp/ptp.ko': No such file or directory FATAL: Error inserting igb (/lib/modules/2.6.32-358.2.1.el6.x86_64/kernel/drivers/net/igb/igb.ko): Unknown symbol in module, or unknown parameter (see dmesg) [root@dhcp145-234 ~]# Expected results: Driver is loaded. Additional info: With the RHEL6.4 kernel, igb.ko depends on ptp.ko and pps_core.ko for some symbols. These modules are not included in the RHEV-H file system. When kdump is enabled, warning messages for these modules (and the "wmi" module) can be seen during the rebuild of the kdump initrd during RHEV-H startup. Under RHEL6.3-based RHEV-H, kdump over NFS has been reported to work on affected systems; flagging this as a regression accordingly. Presumably https://github.com/oVirt/Node/blob/master/recipe/common-minimizer.ks needs to be changed to get these modules included. It would probably be a good idea to have a general QA test for the RHEV-H image that ensures that set of kernel modules included is closed under symbol dereference, to prevent future issues of this kind.
I can repoduce this bug. reproduce NIC is Broadcom Corporation NetXtreme BCM5722 Gigabit Ethernet. Steps to Reproduce: 1. SSH into RHEV-H node 2. "modprobe igb" 4. insmod /lib/modules/2.6.32-358.2.1.el6.x86_64/kernel/drivers/net/igb/igb.ko 5.get the initrd-2.6.32-358.2.1.el6.x86_64kdump.img 6.zcat -f initrd-2.6.32-358.2.1.el6.x86_64kdump.img | cpio -it | grep -E 'igb|ppt|pps' 5.mkdir initrd && cd initrd && zcat -f ../initrd-2.6.32- 358.2.1.el6.x86_64kdump.img | cpio -id && nm --undefined-only lib/modules/2.6.32-358.2.1.el6.x86_64/igb.ko | grep ptp all the steps produce the same results as above.
testing with above build rhev-hypervisor6-6.4-20130409.0.auto586.el6_4.iso testing NIC is intel Corporation 82576 nic，this is not the I350 Gigabit Network Connection NIC. But both of them use the igb driver. So I can use it to test this bug. Steps to Reproduce: 1.SSH into RHEV-H node 2."modprobe igb" 3.set the kdump via nfs (10.66.11.91:/home/vol/cshao/data) and ssh (root@10.66.70.137). 4.echo c > /proc/sysrq-trigger 5.check the dumpfile wheather exists from the kdump path. 6.check the /var/log/boot.log 7.check the /boot/initrd-2.6.32-358.2.1.el6.x86_64kdump.img Actual results: on the rhev-hypervisor6-6.4-20130409.0.auto586.el6_4.iso,the kdump can be succeed via booth nfs and ssh. And there is no warning message in the boot.log. And there is no warning and error message when run 'modprobe igb' so this bug is fixed on rhev-hypervisor6-6.4-20130409.0.auto586.el6_4.iso.
Test version: rhev-hypervisor6-6.5-20131017.0 ovirt-node-3.0.1-4.el6.noarch Testing NIC is intel Corporation 82579 nic，this is not the I350 Gigabit Network Connection NIC. But both of them use the igb driver. So I can use it to test this bug. Steps to Reproduce: 1. Install RHEV-H. 2. "modprobe igb" 3. lsmod | grep igb 4. set the kdump via nfs and ssh. 5. echo c > /proc/sysrq-trigger 6. check the dumpfile wheather exists from the kdump path. Test result: 2. # modprobe igb WARNING: Could not open '/lib/modules/2.6.32-424.el6.x86_64/kernel/drivers/i2c/i2c-core.ko': No such file or directory WARNING: Could not open '/lib/modules/2.6.32-424.el6.x86_64/kernel/drivers/i2c/algos/i2c-algo-bit.ko': No such file or directory Some warning info pop-up, but there is no effect on this mode. 3. # lsmod | grep igb igb 197536 0 dca 7101 1 igb ptp 9614 2 igb,e1000e i2c_algo_bit 5935 2 igb,i915 i2c_core 31084 5 igb,i915,drm_kms_helper,drm,i2c_algo_bit 4. The kdump can be succeed via booth nfs and ssh. So the igb driver is loaded, the bug is fixed, change bug status to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0033.html