Bug 638131 - Cannot perform network kickstart installation to iSCSI target accessed via different NIC
Cannot perform network kickstart installation to iSCSI target accessed via di...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: anaconda (Show other bugs)
6.0
x86_64 Linux
low Severity medium
: rc
: ---
Assigned To: Radek Vykydal
Release Test Team
:
Depends On: 665027 668417
Blocks:
  Show dependency treegraph
 
Reported: 2010-09-28 07:05 EDT by Tomasz Kepczynski
Modified: 2014-05-22 10:44 EDT (History)
9 users (show)

See Also:
Fixed In Version: anaconda-13.21.110-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 668417 (view as bug list)
Environment:
Last Closed: 2011-05-19 08:50:59 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
kickstart configuration (1.59 KB, text/plain)
2010-09-28 07:05 EDT, Tomasz Kepczynski
no flags Details
anaconda.log (6.28 KB, text/plain)
2010-10-08 05:22 EDT, Tomasz Kepczynski
no flags Details
syslog (44.70 KB, text/plain)
2010-10-08 05:23 EDT, Tomasz Kepczynski
no flags Details
ifcfg.log (2.82 KB, text/plain)
2010-10-08 05:23 EDT, Tomasz Kepczynski
no flags Details

  None (edit)
Description Tomasz Kepczynski 2010-09-28 07:05:08 EDT
Created attachment 450147 [details]
kickstart configuration

Description of problem:
I am trying to do fully automated network kickstart install from nfs server to iSCSI target. I cannot do this as nfs/kickstart server is on a separate subnet to iSCSI target and I cannot activate more then 1 ethernet port during installation

Version-Release number of selected component (if applicable):
anaconda-13.21.50-9.el6.x86_64.rpm

How reproducible:
always

Steps to Reproduce:
1.
I have to separate networks: 172.28.x.y/21 where nfs/dhcp/tftp server(s) live. This network also provides default gateway to the world. The second network is 192.168.18.0/24 where iSCSI target lives. This network also has DHCP server which provides iSCSI boot parameters. This network is separated from the first network and there is no routing between the two networks.

2.
Setup machine to which install will be performed with 2 NICs: 1 is PXE NIC to be used on the first network above, the other is Intel NIC with iSCSI option ROM and will connect to the second network above. The iSCSI option ROM is configured to get all parameters from DHCP and CHAP is not used on both initiator and target.

3.
Start kickstart installation from pxelinux. Boot command line is:
ip=dhcp ksdevice=bootif ks=nfs:www.xxx.yyy.zzz:/srv/nfs/install/linux/Fedora/ks/other/RHEL6S-x86_64.cfg
Kickstart file is attached.

4.
Installation fails as anaconda only configures one interface (on the first network) and tries to use it to connect to the target and the target is unreachable from that interface.

5.
As a workaround I tried to ask anaconda to bring up two interfaces with two 'network' statements in kickstart file but it ignored me.

Actual results:
Installation cannot proceed (no storage found)

Expected results:
Installation to iSCSI target succeeds

Additional info:
While I am not an expert on iBFT contents I believe it contains all the information to configure NIC connected to iSCSI SAN. If this information is present then anaconda should bring up the interface using data from iBFT and proceed with iSCSI connection over that interfce.

Point of consideration: iSCSI target may not be directly connected to the network to which iSCSI network port is connected. Static route to the iSCSI target then must be installed to reach the target.

Please note that Windows 2008 properly installs when installed from WDS server connected to one NIC to iSCSI disc reachable by another NIC also in scenario when target is on a different target.

The same problem exists on Fedora 12 x86_64 and RHEL 5.5 x86_64.
Comment 2 Chris Lumens 2010-10-07 14:46:52 EDT
Can you please attach /tmp/anaconda.log and /tmp/syslog to this bug report so we can see what's going on with the network?
Comment 3 Tomasz Kepczynski 2010-10-08 05:22:54 EDT
Created attachment 452306 [details]
anaconda.log
Comment 4 Tomasz Kepczynski 2010-10-08 05:23:19 EDT
Created attachment 452307 [details]
syslog
Comment 5 Tomasz Kepczynski 2010-10-08 05:23:46 EDT
Created attachment 452308 [details]
ifcfg.log
Comment 6 Tomasz Kepczynski 2010-10-08 05:24:52 EDT
I attached requested files plus ifcfg.log. The eth1 interface is the PXE interface used to boot installer. eth0 is iSCSI interface.
Comment 7 Chris Lumens 2010-10-19 15:29:04 EDT
Once we've fetched the kickstart file (using ksdevice=bootif), we should then attempt to bring up all network devices listed in the kickstart file.  And we are not doing that.
Comment 8 RHEL Product and Program Management 2010-11-22 14:39:33 EST
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.
Comment 9 Radek Vykydal 2010-11-29 10:28:24 EST
(In reply to comment #7)
> Once we've fetched the kickstart file (using ksdevice=bootif), we should then
> attempt to bring up all network devices listed in the kickstart file.  And we
> are not doing that.

We haven't been activating more than one device in rhel5, so I'd add something like network --activate option and leave "no" as default.
Comment 10 Alexander Todorov 2010-12-01 16:51:01 EST
qa_ack+ - this is not directly related to iSCSI. 

To test we need a system with 2 NICs which are on different networks and there's no routing between them. VMs can be used to do this. The do:

1) Boot with ksdevice=bootif, where booif is eth0
2) Instruct anaconda to access the second network (for example add a repository on the second network or iSCI disk).
3) ks.cfg will have something like:

network --bootproto=dhcp --device=eth0 --onboot=on
network --activate --bootproto=dhcp --device=eth1 --onboot=on


Expected results:
While anaconda is running switch to tty2 and verify both interfaces are up.

If resources on the 2nd network are available to anaconda then PASS, otherwise FAIL.
Comment 11 Tomasz Kepczynski 2010-12-10 10:23:26 EST
Please note that simply activating the additional network interface may not be enough to perform install to iSCSI target (which I already mentioned).

Let's assume:
1. eth0 is configured by DHCP, it exists on subnet1, there is default gateway on this subnet (provided by DHCP) and this is where network installation source is provided (may or may not be on subnet1).
2. eth1 is configured by DHCP, it exists on subnet2, there also is default gateway provided by DHCP. DHCP provides all the information needed by iSCSI option ROM and this information is stored in iBFT. Now, the iSCSI target is connected to subnet3 which is reachable by gateway configured on subnet2 but not on subnet1.

The windows 2008 (and R2) seem to cope with this scenario but I have my doubts that Linux will cope with it without any additional configuration.

I again suggest that iBFT is used for the interface which is used to connect to iSCSI target and that appropriate host route to iSCSI target is configured if needed.
Comment 12 Radek Vykydal 2010-12-22 09:26:25 EST
It really is not as simple as I thought.

- We need to be able to activate the iSCSI device already in loader (stage 2 kickstart is too late as iscsi is set up very early in stage 2). I have a patch that would allow to activate more than one NICs in loader. It also adds ks network --activate option.

- We need to be able to set DEFROUTE=no for NIC used for iSCSI. I have a patch adding --nodefroute to ks network command.

- We should use NM (now that it is ready having BOOTPROTO=ibft setting in 6.1) instead of reading the ibft values ourselves in anaconda. This has one issue, I filed bug #665027 and posted a patch.

- As for the routing information for subnet3 from comment #11, I think it can be set using route command in %pre kickstart section.

- I'd like to work on related bug #634016 while I'm on iSCSI stuff.
Comment 13 Tomasz Kepczynski 2010-12-22 09:38:28 EST
(In reply to comment #12)
> 
> - As for the routing information for subnet3 from comment #11, I think it can
> be set using route command in %pre kickstart section.
> 

I don't think it is doable if we grab all the stuff from the iBFT. How can we access target IP from iBFT from %pre script in anaconda?
While it may be an option for one-two targets that the addresses are hardcoded in kickstart file, it is a nightmare for larger deployments. And why must I repeat information I already have in DHCP in kickstart file?
I would say that if NM is going to use iBFT for interface configuration it should also take care of setting the host route to target. You will really need it after the installation.
Comment 14 Radek Vykydal 2011-01-04 07:52:18 EST
(In reply to comment #13)

> I don't think it is doable if we grab all the stuff from the iBFT. How can we
> access target IP from iBFT from %pre script in anaconda?

You should be able to use iscsiadm tool, e.g:
iscsiadm -m fw

> I would say that if NM is going to use iBFT for interface configuration it
> should also take care of setting the host route to target. You will really need
> it after the installation.

I'd leave the case when target is on subnet different from subnet to which NIC used for iscsi is connected ("Point of consideration:" from the bug Description) out of scope of this bug where I'd solve only the case from "Steps to reproduce". To address the point of consideration, please file a bug (RFE/request for enhancement) for NetworkManager as this is the place where the change should happen so that the configuration is working post installation too.
Comment 15 Tomasz Kepczynski 2011-01-04 07:57:16 EST
Fair enough. I will return to routing issue once your fix is in place and I have a chance to touch the installation and see if we still need it.
Comment 16 Radek Vykydal 2011-01-26 09:10:45 EST
This should be fixed in anaconda-13.21.92-1.



These kickstart network command options were added (require pykickstart-1.74.5-1):

--activate
causes the device to be activated in loader (by default it is off). For the first network command, it is always on to keep rhel 5 behaviour.

--nodefroute
sets DEFROUTE=no in ifcfg file to prevent grabbing of default route by the device.

--bootproto=ibft
read configuration values from iBFT


* Example of kickstart configuration for iSCSI on separate NIC configured manually:

network --device=eht0 --bootproto=dhcp
network --device=eth1 --activate --nodefroute <NIC configuration>

where <NIC configuration> can be bootproto=dhcp or static configuration with ip=XXX etc.

* Example of kickstart configuration for iSCSI on separate NIC configured with iBFT:
- repository is accessed via eth0; iSCSI is accessed via eth1, both on separate networks
- iBFT is configured properly
- iSCSI target must be accessible via eth1 subnet
- depends on https://bugzilla.redhat.com/show_bug.cgi?id=665027 that should make --nodefroute option work.

network --device=eth0 --bootproto=dhcp
network --device=eth1 --activate --nodefroute --bootproto=ibft
Comment 17 Radek Vykydal 2011-01-26 09:15:50 EST
(In reply to comment #16)

> 
> * Example of kickstart configuration for iSCSI on separate NIC configured
> manually:
> 

I forgot to add two notes to this (the first) example:

- repository is accessed via eth0, iSCSI is accessed via eth1, the NICs are on separate subnets
- iSCSI target must be accessible via eth1 subnet
Comment 19 Dan Williams 2011-02-03 18:53:47 EST
The NM patch for ibft+GATEWAYDEV was applied and built in NetworkManager-0.8.1-7.el6 today.
Comment 20 Radek Vykydal 2011-02-25 08:49:31 EST
This patch was added:

commit bcf83ca7a9c2d9e61ffb2b25e7ad1efba1ff0297
Author: Radek Vykydal <rvykydal@redhat.com>
Date:   Sun Feb 20 23:32:42 2011 +0100

    Do not activate first ks network device in non-network installs.
    
    To resotre rhel5 behaviour ... one more case found:
    for media (non-network) installs kickstart network command should
    not bring network up.
    
    Related: rhbz#638131

For QA:
see update of RHEL 6 Installation guide:
https://bugzilla.redhat.com/show_bug.cgi?id=679104
and
http://fedoraproject.org/wiki/Anaconda/Network#kickstart
Comment 22 Radek Vykydal 2011-04-01 08:49:09 EDT
I tested RHEL 6.1 nigtly 20110328 with this setup:

Have two separate subnets linked to two network devices:
- INSTDEV attached to subnet with kickstart, stage2 and repositories
- ISCSIDEV having iBFT configured, attached to subnet with iSCSI target

I used this boot option:

ksdevice=<mac address of INSTDEV>

and this kickstart network configuration:

network --device=<mac address of INSTDEV> --bootproto=dhcp --onboot=yes                                                                     
network --device=ibft --bootproto=ibft --onboot=yes --activate --nodefroute

and came across a bug in stage 2 kickstart processing which can't handle --device="ibft" option and raises error. Patch fixing it is here:

http://www.redhat.com/archives/anaconda-devel-list/2011-April/msg00016.html
Comment 23 Alexander Todorov 2011-04-04 03:26:21 EDT
Moving to assigned to pull in the patch from comment #22.
Comment 26 Radek Vykydal 2011-04-05 04:32:36 EDT
(In reply to comment #23)
> Moving to assigned to pull in the patch from comment #22.

The patch will be in anaconda-13.21.110-1.
Comment 28 Radek Vykydal 2011-04-21 05:37:45 EDT
(In reply to comment #23)
> Moving to assigned to pull in the patch from comment #22.

Note for QA: when testing please use Snapshot 5 or later. Older snapshots would probably fail due to another bug.
Comment 30 errata-xmlrpc 2011-05-19 08:50:59 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0530.html
Comment 31 Tomasz Kepczynski 2011-05-24 08:33:19 EDT
This issue is not resolved, please reopen.

Anaconda activates only one network interface, contrary to comments in #22.
Comment 32 Radek Vykydal 2011-05-24 08:53:02 EDT
(In reply to comment #31)
> This issue is not resolved, please reopen.
> 
> Anaconda activates only one network interface, contrary to comments in #22.

Can you please attach /tmp/anaconda.log and /tmp/syslog, used kickstart file, /etc/sysconfig/network-scripts/ifcfg-* files, and your ibft configuration (output of "iscsiadm -m -fw" command run in tty2) from the installation that fails for you?
Comment 33 Tomasz Kepczynski 2011-05-25 02:54:56 EDT
I've just found out that the problem exists only when specifying "text" directive in kickstart file. GUI install (the same kickstart file but with "text" commented out) works correctly.

If you still want the log files please let me know.
Comment 34 Radek Vykydal 2011-08-24 10:49:09 EDT
(In reply to comment #33)
> I've just found out that the problem exists only when specifying "text"
> directive in kickstart file. GUI install (the same kickstart file but with
> "text" commented out) works correctly.
> 

This issue will be fixed with bug #731274 in rhel 6.2. It is caused by network command preceding text command in this case.
Comment 35 Trapier Marshall 2011-10-31 12:53:46 EDT
(In reply to comment #34)
> (In reply to comment #33)
> > I've just found out that the problem exists only when specifying "text"
> > directive in kickstart file. GUI install (the same kickstart file but with
> > "text" commented out) works correctly.
> > 
> 
> This issue will be fixed with bug #731274 in rhel 6.2. It is caused by network
> command preceding text command in this case.

I have a case where the work-around was to:

place 'network' entries after 'url', 'graphical', 'text', 'harddrive', ’cdrom’, and 'nfs' entries.  

Just wanted to make a note of it here.

Note You need to log in before you can comment on or make changes to this bug.