Bug 696988

Summary: Problem with installation on per210-01.lab.eng.brq.redhat.com
Product: Red Hat Enterprise Linux 6 Reporter: Kamil Kolakowski <kkolakow>
Component: anacondaAssignee: Radek Vykydal <rvykydal>
Status: CLOSED ERRATA QA Contact: Release Test Team <release-test-team>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.1CC: aokuliar, harald, ohudlick, rvykydal
Target Milestone: rcKeywords: Regression, Reopened, TestBlocker
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-05-19 12:40:30 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
anaconda.log
none
sys.log
none
compressed content of /var/log directory
none
compressed content of /etc directory none

Description Kamil Kolakowski 2011-04-15 14:13:31 UTC
Description of problem:

I have problem with installation RHEL6.1-20110406.0 on per210-01.lab.eng.brq.redhat.com. During installation I have choice to initialize em1 network device. If I choose this device it installs RHEL but when installation is finished and system is rebooted I get message:

Device em1 does not seem to be present, delaying initialization.

from console.log:

bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.1.6 (Mar 7, 2011) 
bnx2 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 
bnx2 0000:02:00.0: firmware: requesting bnx2/bnx2-mips-09-6.2.1a.fw 
bnx2 0000:02:00.0: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw 
bnx2 0000:02:00.0: eth1: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem da000000, IRQ 16, node addr b8:ac:6f:87:74:44 
bnx2 0000:02:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17 
bnx2 0000:02:00.1: firmware: requesting bnx2/bnx2-mips-09-6.2.1a.fw 
bnx2 0000:02:00.1: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw 
bnx2 0000:02:00.1: eth2: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem dc000000, IRQ 17, node addr b8:ac:6f:87:74:45

If I choose eth1 for installation in kickstart installation I get:
You have multiple network devices on this system.   ??? [8;13H??? Which would you like to install through? [12C??? [9;13H??? [53C??? [10;13H??? em1 - Ethernet device em1 - b8:ac:6f:87:74:44

When I will choose em1 after reboot I got Device em1 does not seem to be present, delaying initialization.

Version-Release number of selected component (if applicable):
RHEL6.1-20110406.0

How reproducible:
Submit this cloned job
- for eth1 case
https://beaker.engineering.redhat.com/jobs/clone?recipeset_id=125758
- and for em1 case
https://beaker.engineering.redhat.com/jobs/clone?recipeset_id=125722

In first case installation doesn't start - waiting for choice of network device. In second after finish you will have no network access.

Expected results:
After installation process RHEL is installed and network is connected.

Additional info: If you will need to have per210-01 please contact me.

Comment 2 Radek Vykydal 2011-04-15 14:35:14 UTC
Please attach /tmp/anaconda.log, /tmp/syslog, and /tmp/ifcfg.log (or respective logfiles from installed system's /var/log). Seems like biosdevname issue (the machine is dell) but it may also be caused by bug #690589.
Naturally, eth1 case doesn't work when devices are renamed emX (biosdevname feature on by default on dell)

Comment 3 Kamil Kolakowski 2011-04-15 17:57:48 UTC
Created attachment 492461 [details]
anaconda.log

Comment 4 Kamil Kolakowski 2011-04-15 17:58:19 UTC
Created attachment 492462 [details]
sys.log

Comment 5 Kamil Kolakowski 2011-04-15 17:59:29 UTC
Radek,

I'm unable to get ifcfg.log. I will contact you on Monday to get your help with that.

Thanks

Comment 6 Radek Vykydal 2011-04-18 14:26:38 UTC
From the logs it seems that device renaming didn't cause problems in installer environment, (the logs don't suggest bug #690589) and the installation went ok. The question is why device renaming fails after reboot, we need the logs from installed system.

Renaming in installation environment (comment #4):

13:15:59,771 INFO kernel:e1000e: Intel(R) PRO/1000 Network Driver - 1.2.20-k2
13:15:59,771 INFO kernel:e1000e: Copyright (c) 1999 - 2010 Intel Corporation.
13:15:59,771 INFO kernel:e1000e 0000:01:00.0: Disabling ASPM  L1
13:15:59,771 INFO kernel:e1000e 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
13:15:59,771 DEBUG kernel:e1000e 0000:01:00.0: setting latency timer to 64
13:15:59,771 DEBUG kernel:  alloc irq_desc for 31 on node -1
13:15:59,771 DEBUG kernel:  alloc kstat_irqs on node -1
13:15:59,771 DEBUG kernel:e1000e 0000:01:00.0: irq 31 for MSI/MSI-X
13:15:59,924 INFO kernel:e1000e 0000:01:00.0: eth0: (PCI Express:2.5GB/s:Width x1) 00:15:17:ec:ee:a6
13:15:59,924 INFO kernel:e1000e 0000:01:00.0: eth0: Intel(R) PRO/1000 Network Connection
13:15:59,924 INFO kernel:e1000e 0000:01:00.0: eth0: MAC: 1, PHY: 4, PBA No: D28777-005
13:15:59,933 INFO kernel:udev: renamed network interface eth0 to pci1p1
13:15:59,936 INFO kernel:bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.1.6 (Mar 7, 2011)
13:15:59,936 INFO kernel:bnx2 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
13:15:59,936 DEBUG kernel:bnx2 0000:02:00.0: setting latency timer to 64
13:15:59,937 INFO kernel:bnx2 0000:02:00.0: firmware: requesting bnx2/bnx2-mips-09-6.2.1a.fw
13:15:59,937 INFO kernel:bnx2 0000:02:00.0: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw
13:15:59,937 INFO kernel:bnx2 0000:02:00.0: eth0: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem da000000, IRQ 16, node addr b8:ac:6f:87:74:44
13:15:59,937 DEBUG kernel:  alloc irq_desc for 17 on node -1
13:15:59,937 DEBUG kernel:  alloc kstat_irqs on node -1
13:15:59,937 INFO kernel:bnx2 0000:02:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
13:15:59,937 DEBUG kernel:bnx2 0000:02:00.1: setting latency timer to 64
13:15:59,938 INFO kernel:bnx2 0000:02:00.1: firmware: requesting bnx2/bnx2-mips-09-6.2.1a.fw
13:15:59,938 INFO kernel:bnx2 0000:02:00.1: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw
13:15:59,938 INFO kernel:bnx2 0000:02:00.1: eth1: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem dc000000, IRQ 17, node addr b8:ac:6f:87:74:45
13:15:59,941 NOTICE firmware.sh: udev firmware loader misses sysfs directory
13:15:59,942 NOTICE firmware.sh: udev firmware loader misses sysfs directory
13:15:59,942 NOTICE firmware.sh: udev firmware loader misses sysfs directory
13:15:59,943 NOTICE firmware.sh: udev firmware loader misses sysfs directory
13:15:59,947 INFO kernel:udev: renamed network interface eth0 to em1

Comment 7 Radek Vykydal 2011-04-18 14:32:23 UTC
(In reply to comment #6)

> 13:15:59,941 NOTICE firmware.sh: udev firmware loader misses sysfs directory
> 13:15:59,942 NOTICE firmware.sh: udev firmware loader misses sysfs directory
> 13:15:59,942 NOTICE firmware.sh: udev firmware loader misses sysfs directory
> 13:15:59,943 NOTICE firmware.sh: udev firmware loader misses sysfs directory
> 13:15:59,947 INFO kernel:udev: renamed network interface eth0 to em1

... snip, snip, one more relevant line:

13:16:00,454 INFO kernel:udev: renamed network interface eth1 to em2

Comment 8 Adam Okuliar 2011-04-18 15:14:23 UTC
Hi,

I've done some investigation and here are results:

During installation ethernet devices has names em1, em2. Anacoda creates
network configuration files ifcfg-em[0,1]

cat ifcfg-em1
DEVICE="em1"
BOOTPROTO="dhcp"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"

After installation network interfaces has names eth0, eth1, so network
interfaces will stay down and not configured. Network is unavailable

cat /etc/udev/rules.d/70-persistent-net.rules
# PCI device 0x8086:0x107d (e1000e)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*",
ATTR{address}=="00:15:17:ec:ee:a6", ATTR{type}=="1", KERNEL=="eth*",
NAME="eth0"

# PCI device 0x14e4:0x163b (bnx2)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*",
ATTR{address}=="b8:ac:6f:87:74:44", ATTR{type}=="1", KERNEL=="eth*",
NAME="eth1"

# PCI device 0x14e4:0x163b (bnx2)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*",
ATTR{address}=="b8:ac:6f:87:74:45", ATTR{type}=="1", KERNEL=="eth*",
NAME="eth2"

If I will launch dhclient manualy: dhclient eth1, interface gets IP address and
network is reachable.

I added compressed content of /var/log directory and compressed content of /etc
directory after installation on affected system.

Thanks,
Adam

Comment 9 Adam Okuliar 2011-04-18 15:14:57 UTC
Created attachment 492912 [details]
compressed content of /var/log directory

Comment 10 Adam Okuliar 2011-04-18 15:16:22 UTC
Created attachment 492914 [details]
compressed content of /etc directory

Comment 11 Radek Vykydal 2011-04-18 16:12:47 UTC
Thanks for the logs. So the generated /etc/udev/rules.d/70-persistent-net.rules on installed system doesn't do renaming (see comment #8) while udev renames devices in installer environment (comment #6). Harald, do you have any hint?

(In reply to comment #2)
> ...  Seems like biosdevname issue (the
> machine is dell) but it may also be caused by bug #690589.
The fix for bug #690589 will be reverted in today's nightly and anaconda will write-out HWADDR into ifcfg files again so we should see tomorrow, but it doesn't seem likely to be the cause of the bug now.

Comment 12 Harald Hoyer 2011-04-18 16:23:22 UTC
I would guess biosdevname was not installed by the kickstart file

Comment 13 Adam Okuliar 2011-04-18 18:12:30 UTC
Hi,

I discovered that this problem is triggered by amount of packages, which is installed on system. We use following configuration in our kickstarts for kernel testing:

%packages --ignoremissing --nobase
@core
wget
python
perl-devel
parted
cpuspeed
perl
dhcpv6-client
dhclient
yum
yum-rhn-plugin
yum-security
yum-updatesd
openssh-server
openssh-clients
bc
screen
nfs-utils
seekwatcher
sysstat
xfsprogs
e2fsprogs
hdparm
sdparm
gcc
tuned
cpufrequtils
cryptsetup-luks
vim-enhanced
rsync
lvm2

This install extremely minimal system. We use this configuration for about two years and no problems related to this appeared. Now when using this minimal we have problem with bad ethernet device name, which is described above.

If we use 
%packages --ignoremissing --nobase
there is no problem at all. After installation ethernet devices has correct names em1,em2 and are properly configured and online.

Thanks, 
Adam

Comment 15 Adam Okuliar 2011-04-19 09:15:24 UTC
(In reply to comment #13)

Sorry, there is an error in my previous comment. Correct workaround for our issue is to use installation with default package set. So kick-start solving our troubles have 

%packages --ignoremissing --default

Sorry for confusing and thanks
Adam

Comment 16 Kamil Kolakowski 2011-04-19 11:06:55 UTC
Harald,

I think this should be solved. 

If you choose minimal installation you have no choice and it renames network devices and you are without network after installation reboot. 

In past we used this packages configuration and it works. 
I can imagine some RHEL customers who used it just for Routers/Mail servers would have this problems as well.

I'm reopening it.

Comment 17 Kamil Kolakowski 2011-04-19 11:56:19 UTC
I retested it on RHEL6.1-20110418.n.2.

Devices names em1 em2 as specified during installation.

Solved can be closed.

Comment 18 errata-xmlrpc 2011-05-19 12:40:30 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0530.html