Bug 1289341 - Multi-NIC server fails to kickstart when only one NIC is cabled
Multi-NIC server fails to kickstart when only one NIC is cabled
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: dracut (Show other bugs)
7.2
Unspecified Linux
high Severity high
: rc
: ---
Assigned To: dracut-maint
Release Test Team
: Regression, ZStream
Depends On:
Blocks: 1203710 1289485 1313485 1315709
  Show dependency treegraph
 
Reported: 2015-12-07 16:39 EST by Dylan Gross
Modified: 2017-05-18 10:31 EDT (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1315709 (view as bug list)
Environment:
Last Closed: 2016-11-04 04:02:01 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dylan Gross 2015-12-07 16:39:34 EST
Description of problem:

  Kickstarting a server with multiple NICs where only one has a link, the first few un-cabled cards are configured with the network setup of the targeted card.

  Server with 4 (un-cabled) on-board NICs and one (cabled) 10Gb NIC.

  enp3s0f0
  enp3s0f1
  enp3s0f2
  enp3s0f3
  ens1     (This NIC is Cabled, with link - When finally started, produces error that "enp3s0f0" has the IP address.   Ip is applied to enp3s0f0, then dracut attempts to apply same static ip info to ens1 and fails, as it is already assigned.  



Version-Release number of selected component (if applicable):

  RHEL 7.2


How reproducible:


Steps to Reproduce:
1. kickstart multi-NIC server with only one NIC cabled using boot line args:    ipv6.disable ksdevice=link nameserver=<DNS> netmask=<NETMASK> ip=<IP Address> gateway=<Gateway> hostname=<Hostname>



Actual results:

   Drops to a dracut shell after an error about Duplicate IP being held by the first discovered (uncabled) device

Expected results:

   Cabled NIC is assigned the correct IP and boots.

Additional info:

  - This kickstart is happening against a Satellite 5.7 using a boot ISO.  No DHCP/pxe.

  --device=link was working previously, but no longer works with 7.2
  - The ksdevice was working in 7.0 but stopped working in 7.1
Comment 1 Ben Breard 2015-12-07 16:53:06 EST
For the same environment we've also tested the following: 


initrd=initrd-7.img inst.ks=http://[satellite]/cblr/svc/op/ks/profile/RHEL-7-Prod:1 ipv6.disable nameserver=10.10.17.10 ip=10.10.10.142::10.10.10.129:255.255.255.128:hostname:ens1:none


This allows the kickstart file to be downloaded, but towards the end of the install the network cannot start as all interfaces ifcfg files are configured as static with the same ip address.
Comment 2 Jason Dickerson 2015-12-07 17:22:40 EST
Addition to comment 1.  The build works, but the static ifcfg-ens1 file is not created correctly by the network include kickstart snippet written by the customer.  That snippet depends on the old syntax where ip, netmask, gw, hostname, are all separate variables.  This results in a built server with no network.  (I am not sure if the kickstart postinstall is completing correctly or not.)  However, the result is a server that can boot.
Comment 3 Radek Vykydal 2015-12-08 09:58:29 EST
Could you please post anaconda logs for the case from comment #1 ? Also the kickstart file used for installation. The logs we need are:

/tmp/anaconda.log
/tmp/syslog
/tmp/ifcfg.log
Comment 4 Radek Vykydal 2015-12-08 10:33:22 EST
I am not able to reproduce the issue from the Description. I see the duplicate IP message but after some time (waiting for link on unplugged device) dracut proceeds to installation.

Could you please attach rd.debug option and try to gather output of journalctl -a from dracut shell?
Comment 6 Radek Vykydal 2015-12-09 06:51:02 EST
I suppose the kickstart used is the one attached in the customer case.

(In reply to Dylan Gross from comment #5)
> Created attachment 1103710 [details]
> journalctl -a output from case 01550617

This is the output for this case:

ip=161.166.234.142::161.166.234.129:255.255.255.128:osel400639::none

I'd like to see also journalctl -a with rd.debug for the original problem from the description, ie

ksdevice=link nameserver=161.173.7.10 netmask=255.255.255.128 ip=161.166.234.142 gateway=161.166.234.129 hostname=osel400639
Comment 7 Radek Vykydal 2015-12-09 07:01:54 EST
(In reply to Dylan Gross from comment #5)
> Created attachment 1103710 [details]
> journalctl -a output from case 01550617

It seems that enp3s0f0 behaves as it actually has a link, so it gets the static address (before ens1). From the log:

Dec 08 14:39:43 localhost dracut-initqueue[735]: /sbin/ifup@396(): do_static
Dec 08 14:39:43 localhost dracut-initqueue[735]: /sbin/ifup@156(do_static): strstr 161.166.234.142 '*:*:*'
Dec 08 14:39:43 localhost dracut-initqueue[735]: /lib/dracut-lib.sh@29(strstr): '[' 161.166.234.142 '!=' 161.166.234.142 ']'
Dec 08 14:39:43 localhost dracut-initqueue[735]: /sbin/ifup@158(do_static): linkup enp3s0f0
Dec 08 14:39:43 localhost dracut-initqueue[735]: /lib/net-lib.sh@607(linkup): wait_for_if_link enp3s0f0
Dec 08 14:39:43 localhost dracut-initqueue[735]: /lib/net-lib.sh@608(linkup): ip link set enp3s0f0 up
Dec 08 14:39:43 localhost dracut-initqueue[735]: /lib/net-lib.sh@609(linkup): wait_for_if_up enp3s0f0
Dec 08 14:39:43 localhost dracut-initqueue[735]: /sbin/ifup@163(do_static): ip route get 161.166.234.142
Dec 08 14:39:43 localhost dracut-initqueue[735]: /sbin/ifup@164(do_static): read a rest
Dec 08 14:39:43 localhost dracut-initqueue[735]: RTNETLINK answers: Network is unreachable
Dec 08 14:39:43 localhost dracut-initqueue[735]: /sbin/ifup@165(do_static): '[' '' = local ']'
Dec 08 14:39:43 localhost dracut-initqueue[735]: /sbin/ifup@169(do_static): return 0
Dec 08 14:39:43 localhost dracut-initqueue[735]: /sbin/ifup@172(do_static): '[' -n '' ']'
Dec 08 14:39:43 localhost dracut-initqueue[735]: /sbin/ifup@173(do_static): '[' -n '' ']'
Dec 08 14:39:43 localhost dracut-initqueue[735]: /sbin/ifup@174(do_static): strstr 161.166.234.142 '*:*:*'
Dec 08 14:39:43 localhost dracut-initqueue[735]: /lib/dracut-lib.sh@29(strstr): '[' 161.166.234.142 '!=' 161.166.234.142 ']'
Dec 08 14:39:43 localhost dracut-initqueue[735]: /sbin/ifup@179(do_static): arping -f -q -D -c 2 -I enp3s0f0 161.166.234.142
Dec 08 14:39:44 localhost kernel: floppy0: no floppy controllers found
Dec 08 14:39:44 localhost dracut-initqueue[735]: /sbin/ifup@183(do_static): ip addr flush dev enp3s0f0
Dec 08 14:39:44 localhost dracut-initqueue[735]: /sbin/ifup@184(do_static): ip addr add 161.166.234.142/255.255.255.128 brd + dev enp3s0f0

compared to other devices:

Dec 08 14:40:47 osel400639 dracut-initqueue[735]: /sbin/ifup@158(do_static): linkup enp3s0f1
Dec 08 14:40:47 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@607(linkup): wait_for_if_link enp3s0f1
Dec 08 14:40:47 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@608(linkup): ip link set enp3s0f1 up
Dec 08 14:40:47 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@609(linkup): wait_for_if_up enp3s0f1
Dec 08 14:41:06 osel400639 dracut-initqueue[735]: /sbin/ifup@159(do_static): warn 'Could not bring interface enp3s0f1 up!'
Dec 08 14:41:06 osel400639 dracut-initqueue[735]: /lib/dracut-lib.sh@60(warn): echo 'Warning: Could not bring interface enp3s0f1 up!'
Dec 08 14:41:06 osel400639 dracut-initqueue[735]: Warning: Could not bring interface enp3s0f1 up!
Dec 08 14:41:06 osel400639 dracut-initqueue[735]: /sbin/ifup@160(do_static): return 1

ec 08 14:41:07 osel400639 dracut-initqueue[735]: /sbin/ifup@158(do_static): linkup enp4s0f0
Dec 08 14:41:07 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@607(linkup): wait_for_if_link enp4s0f0
Dec 08 14:41:07 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@608(linkup): ip link set enp4s0f0 up
Dec 08 14:41:07 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@609(linkup): wait_for_if_up enp4s0f0
Dec 08 14:41:27 osel400639 dracut-initqueue[735]: /sbin/ifup@159(do_static): warn 'Could not bring interface enp4s0f0 up!'
Dec 08 14:41:27 osel400639 dracut-initqueue[735]: /lib/dracut-lib.sh@60(warn): echo 'Warning: Could not bring interface enp4s0f0 up!'
Dec 08 14:41:27 osel400639 dracut-initqueue[735]: Warning: Could not bring interface enp4s0f0 up!
Dec 08 14:41:27 osel400639 dracut-initqueue[735]: /sbin/ifup@160(do_static): return 1


Dec 08 14:41:28 osel400639 dracut-initqueue[735]: /sbin/ifup@396(): do_static
Dec 08 14:41:28 osel400639 dracut-initqueue[735]: /sbin/ifup@156(do_static): strstr 161.166.234.142 '*:*:*'
Dec 08 14:41:28 osel400639 dracut-initqueue[735]: /lib/dracut-lib.sh@29(strstr): '[' 161.166.234.142 '!=' 161.166.234.142 ']'
Dec 08 14:41:28 osel400639 dracut-initqueue[735]: /sbin/ifup@158(do_static): linkup enp4s0f1
Dec 08 14:41:28 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@607(linkup): wait_for_if_link enp4s0f1
Dec 08 14:41:28 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@608(linkup): ip link set enp4s0f1 up
Dec 08 14:41:28 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@609(linkup): wait_for_if_up enp4s0f1
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /sbin/ifup@159(do_static): warn 'Could not bring interface enp4s0f1 up!'
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /lib/dracut-lib.sh@60(warn): echo 'Warning: Could not bring interface enp4s0f1 up!'
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: Warning: Could not bring interface enp4s0f1 up!
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /sbin/ifup@160(do_static): return 1


Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /sbin/ifup@396(): do_static
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /sbin/ifup@156(do_static): strstr 161.166.234.142 '*:*:*'
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /lib/dracut-lib.sh@29(strstr): '[' 161.166.234.142 '!=' 161.166.234.142 ']'
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /sbin/ifup@158(do_static): linkup ens1
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@607(linkup): wait_for_if_link ens1
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@608(linkup): ip link set ens1 up
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /lib/net-lib.sh@609(linkup): wait_for_if_up ens1
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /sbin/ifup@163(do_static): ip route get 161.166.234.142
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /sbin/ifup@164(do_static): read a rest
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /sbin/ifup@165(do_static): '[' local = local ']'
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /sbin/ifup@166(do_static): warn 'Not assigning 161.166.234.142 to interface ens1, cause it is already assigned!'
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /lib/dracut-lib.sh@60(warn): echo 'Warning: Not assigning 161.166.234.142 to interface ens1, cause it is already assigned!'
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: Warning: Not assigning 161.166.234.142 to interface ens1, cause it is already assigned!
Dec 08 14:41:48 osel400639 dracut-initqueue[735]: /sbin/ifup@167(do_static): return 1

So even the one-string ip= configuration should work without specifying the device, given only one device has link.
That would require changing kickstart %pre script to parse this form of configuration though.

I suspect the multiple values configuration (ip= gateway= ...) is not working for the same reason, though we'd need to see the log requested in comment #6 to confirm.

Now the question is: is the ens1 really the only plugged device? And if so, why linkup on enp3s0f0 succeeds?
Comment 12 Radek Vykydal 2015-12-10 04:11:47 EST
From the logs in comment #9 I'd say both forms of boot options (multiple values with ksdevice, and ip= dracut boot option) fail due to the same root cause identified by Masahiro (great job!).

Reassigning to dracut based on comment #10.
Comment 25 Mike McCune 2016-03-28 18:31:02 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 26 Harald Hoyer 2016-06-20 08:30:21 EDT
please verify, if this is fixing your problem:

http://people.redhat.com/harald/downloads/dracut/dracut-033-360.el7_2.2/
Comment 32 errata-xmlrpc 2016-11-04 04:02:01 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2530.html

Note You need to log in before you can comment on or make changes to this bug.