Bug 721101

Summary: ifup-ib breaks on previously working ifcfg-ib configuration files.
Product: Red Hat Enterprise Linux 6 Reporter: C Speare <carl.speare>
Component: rdmaAssignee: Doug Ledford <dledford>
Status: CLOSED ERRATA QA Contact: Infiniband QE <infiniband-qe>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.1CC: borgan, carl.speare, honli, i.sakamoto, kzhang, mpoole, pasteur, prescott
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 15:33:07 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ifcfg-ib0 using netmask
none
ifcfg-ib0 using prefix
none
Calculate PREFIX if needed from NETMASK none

Description C Speare 2011-07-13 18:03:03 UTC 
Description of problem:
An /etc/sysconfig/network-scripts/ifcfg-ib0 file from a previously working RHEL 5 installation was used as a basis for a RHEL 6.1 system.  We also compared the file against a working RHEL 6.0 system and the configuration file should work.

However, during either a "service network restart" or an "ifup ib0", ipcalc complains and causes the interface to be marked as already in use.  This effectively renders the ib0 (ipoib) interface useless.

Version-Release number of selected component (if applicable):
rdma-1.0-9.el6.noarch

How reproducible:
Consistently breaks without fail.

Steps to Reproduce:
1. Perform a vanilla installation of RHEL 6.1 (kernel-2.6.32-131.0.15.el6.x86_64) with Infiniband Support group selected.
2. Create the ifcfg-ib0 interface file in /etc/sysconfig/network-scripts.  See attachments for both the NETMASK based version as well as the prefix based version.  (We tried both.)
3. Perform a "service network restart" or an "ifup ib0".
4. arpping will complain about unknown host if using the prefix method; if using the NETMASK method, ifup will complain that an inet prefix was expected.
5. Interface ib0 will not have an IP address assigned.  ibhosts verifies that connectivity is present (and ibstat shows both links are up).

Actual results:
ib0 will not have an IP address, which breaks ipoib.

Expected results:
ib0 should have an IP address.

Additional info:
Please see attachments for copies of the configuration files used.
Comment 1 C Speare 2011-07-13 18:07:24 UTC 
Created attachment 512717 [details]
ifcfg-ib0 using netmask
Comment 2 C Speare 2011-07-13 18:07:57 UTC 
Created attachment 512718 [details]
ifcfg-ib0 using prefix
Comment 3 C Speare 2011-07-13 18:12:19 UTC 
When using the netmask method, this is the output of "ifup ib0"

# ifup ib0
Error: an inet prefix is expected rather than "10.116.36.86/".
Error adding address 10.116.36.86 for ib0.
bind: Cannot assign requested address
Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
#

When using the prefix method, this is the output of "ifup ib0"

# ifup ib0
ipcalc: both netmask and prefix specified
Usage: ipcalc [OPTION...]
  -c, --check         Validate IP address for specified address family
  -4, --ipv4          IPv4 address family (default)
  -6, --ipv6          IPv6 address family
  -b, --broadcast     Display calculated broadcast address
  -h, --hostname      Show hostname determined via DNS
  -m, --netmask       Display default netmask for IP (class A, B, or C)
  -n, --network       Display network address
  -p, --prefix        Display network prefix
  -s, --silent        Don't ever display error messages

Help options:
  -?, --help          Show this help message
  --usage             Display brief usage message
ipcalc: both netmask and prefix specified
Usage: ipcalc [OPTION...]
  -c, --check         Validate IP address for specified address family
  -4, --ipv4          IPv4 address family (default)
  -6, --ipv6          IPv6 address family
  -b, --broadcast     Display calculated broadcast address
  -h, --hostname      Show hostname determined via DNS
  -m, --netmask       Display default netmask for IP (class A, B, or C)
  -n, --network       Display network address
  -p, --prefix        Display network prefix
  -s, --silent        Don't ever display error messages

Help options:
  -?, --help          Show this help message
  --usage             Display brief usage message
ipcalc: both netmask and prefix specified
Usage: ipcalc [OPTION...]
  -c, --check         Validate IP address for specified address family
  -4, --ipv4          IPv4 address family (default)
  -6, --ipv6          IPv6 address family
  -b, --broadcast     Display calculated broadcast address
  -h, --hostname      Show hostname determined via DNS
  -m, --netmask       Display default netmask for IP (class A, B, or C)
  -n, --network       Display network address
  -p, --prefix        Display network prefix
  -s, --silent        Don't ever display error messages

Help options:
  -?, --help          Show this help message
  --usage             Display brief usage message
arping: unknown host 10.116.36.86/26
Error, some other host already uses address 10.116.36.86/26.
#

In either case, "ip addr show dev ib0" shows that the device has no IPv4 address, which we would hope for.

Note that the netmask-based ifcfg-ib0 file attached works on a RHEL 6.0 host.
Comment 5 C Speare 2011-07-14 22:23:34 UTC 
Evidently, if we specify the netmask in the form:

PREFIX="26"

instead of:

NETMASK="255.255.255.192"

That works.
Comment 6 Craig Prescott 2011-07-26 15:54:12 UTC 
I ran into this same problem as well, and had a look.  I don't think ifup-ib should be different than ifup-eth in this respect - ifup-eth supports NETMASK=, then so should ifup-ib.  

In my case I just added the part that calculates the prefix from the netmask.  I'll upload the patch I used.
Comment 7 Craig Prescott 2011-07-26 16:00:06 UTC 
Created attachment 515310 [details]
Calculate PREFIX if needed from NETMASK
Comment 8 Doug Ledford 2011-08-04 17:04:38 UTC 
The problem is that expand_config used to set PREFIX if it wasn't set already.  Now, due to adding support for multiple IP addresses on an interface, it sets an array of ${ipaddr[$index]}, ${netmask[$index]}, ${network[$index]}, ${broadcast[$index]}, and ${prefix[$index]} instead.  I've modified the ifup-ib script to deal with this array instead of the old single address variables.
Comment 9 Doug Ledford 2011-08-04 17:05:15 UTC 
*** Bug 714669 has been marked as a duplicate of this bug. ***
Comment 12 Honggang LI 2011-08-31 03:30:42 UTC 
(In reply to comment #9)
> *** Bug 714669 has been marked as a duplicate of this bug. ***

Netmask method works properly on X86 and X86_64 platform had been verified. However prefix method still failed, since the expand_config assume that prefix would be provided in 'PREFIX' item not in 'IPADDR'.
Comment 13 Doug Ledford 2011-08-31 17:27:58 UTC 
Adding the PREFIX to the end of IPADDR was never intended to work (it simply worked around the incompatibility between the rhel6.1 ifup-ib file and the expand_config function).
Comment 14 Honggang LI 2011-09-01 01:28:52 UTC 
--------------------- reproduce -------------------------------------------
[root@rdma3 yum.repos.d]# uname -a
Linux rdma3.rhts.eng.bos.redhat.com 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@rdma3 yum.repos.d]# rpm -qa rdma
rdma-1.0-9.el6.noarch
[root@rdma3 yum.repos.d]# cat /etc/sysconfig/network-scripts/ifcfg-ib0
DEVICE="ib0"
ONBOOT="yes"
MTU="1500"
NETWORKING_IPV6="yes"
IPADDR="10.116.36.86"
NETMASK="255.255.255.192"
[root@rdma3 yum.repos.d]# ifdown ib0
[root@rdma3 yum.repos.d]# ibstat
CA 'qib0'
	CA type: InfiniPath_QLE7140
	Number of ports: 1
	Firmware version: 
	Hardware version: 2
	Node GUID: 0x0011750000ffdaa5
	System image GUID: 0x0011750000ffdaa5
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 10
		Base lid: 1
		LMC: 0
		SM lid: 1
		Capability mask: 0x0661086a
		Port GUID: 0x0011750000ffdaa5
		Link layer: InfiniBand
CA 'qib1'
	CA type: InfiniPath_QLE7140
	Number of ports: 1
	Firmware version: 
	Hardware version: 2
	Node GUID: 0x0011750000ffd832
	System image GUID: 0x0011750000ffdaa5
	Port 1:
		State: Initializing
		Physical state: LinkUp
		Rate: 10
		Base lid: 65535
		LMC: 0
		SM lid: 65535
		Capability mask: 0x06610868
		Port GUID: 0x0011750000ffd832
		Link layer: InfiniBand
[root@rdma3 yum.repos.d]# ifup  ib0
Error: an inet prefix is expected rather than "10.116.36.86/".
Error adding address 10.116.36.86 for ib0.
bind: Cannot assign requested address
Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
[root@rdma3 yum.repos.d]# cd /etc/sysconfig/network-scripts/
[root@rdma3 network-scripts]# vi ifcfg-ib0 
[root@rdma3 network-scripts]# ifdown ib0
[root@rdma3 network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-ib0
DEVICE="ib0"
ONBOOT="yes"
MTU="1500"
NETWORKING_IPV6="yes"
IPADDR="10.116.36.86/26"
[root@rdma3 network-scripts]# ifup ib0
ipcalc: both netmask and prefix specified
Usage: ipcalc [OPTION...]
  -c, --check         Validate IP address for specified address family
  -4, --ipv4          IPv4 address family (default)
  -6, --ipv6          IPv6 address family
  -b, --broadcast     Display calculated broadcast address
  -h, --hostname      Show hostname determined via DNS
  -m, --netmask       Display default netmask for IP (class A, B, or C)
  -n, --network       Display network address
  -p, --prefix        Display network prefix
  -s, --silent        Don't ever display error messages

Help options:
  -?, --help          Show this help message
  --usage             Display brief usage message
ipcalc: both netmask and prefix specified
Usage: ipcalc [OPTION...]
  -c, --check         Validate IP address for specified address family
  -4, --ipv4          IPv4 address family (default)
  -6, --ipv6          IPv6 address family
  -b, --broadcast     Display calculated broadcast address
  -h, --hostname      Show hostname determined via DNS
  -m, --netmask       Display default netmask for IP (class A, B, or C)
  -n, --network       Display network address
  -p, --prefix        Display network prefix
  -s, --silent        Don't ever display error messages

Help options:
  -?, --help          Show this help message
  --usage             Display brief usage message
ipcalc: both netmask and prefix specified
Usage: ipcalc [OPTION...]
  -c, --check         Validate IP address for specified address family
  -4, --ipv4          IPv4 address family (default)
  -6, --ipv6          IPv6 address family
  -b, --broadcast     Display calculated broadcast address
  -h, --hostname      Show hostname determined via DNS
  -m, --netmask       Display default netmask for IP (class A, B, or C)
  -n, --network       Display network address
  -p, --prefix        Display network prefix
  -s, --silent        Don't ever display error messages

Help options:
  -?, --help          Show this help message
  --usage             Display brief usage message
arping: unknown host 10.116.36.86/26
Error, some other host already uses address 10.116.36.86/26.
[root@rdma3 network-scripts]# vi ifcfg-ib0
[root@rdma3 network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-ib0
DEVICE="ib0"
ONBOOT="yes"
MTU="1500"
NETWORKING_IPV6="yes"
IPADDR="10.116.36.86"
PREFIX="26"
[root@rdma3 network-scripts]# ifdown ib0
[root@rdma3 network-scripts]# ifup  ib0
Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
[root@rdma3 network-scripts]# ip addr show dev ib0
4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 256
    link/infiniband 80:00:00:02:fe:80:00:00:00:00:00:00:00:11:75:00:00:ff:da:a5 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.116.36.86/26 brd 10.116.36.127 scope global ib0
    inet6 fe80::211:7500:ff:daa5/64 scope link 
       valid_lft forever preferred_lft forever
[root@rdma3 network-scripts]# 


------------------------ VERIFY ----------------------------------------
[root@rdma4 network-scripts]# uname -a
Linux rdma4.rhts.eng.bos.redhat.com 2.6.32-191.el6.x86_64 #1 SMP Wed Aug 17 20:22:22 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@rdma4 network-scripts]# pwd
/etc/sysconfig/network-scripts
[root@rdma4 network-scripts]# cat ifcfg-ib0
DEVICE="ib0"
ONBOOT="yes"
MTU="1500"
NETWORKING_IPV6="yes"
IPADDR="10.116.36.86"
NETMASK="255.255.255.192"
[root@rdma4 network-scripts]# /sbin/service rdma stop
Unloading OpenIB kernel modules:[  OK  ]
[root@rdma4 network-scripts]# /sbin/service rdma start
Loading OpenIB kernel modules:[  OK  ]
[root@rdma4 network-scripts]# ifup ib0
Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
[root@rdma4 network-scripts]# ip addr show ib0
6: ib0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 256
    link/infiniband 80:00:00:03:fe:80:00:00:00:00:00:00:00:11:75:00:00:68:70:9f brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.116.36.86/26 brd 10.116.36.127 scope global ib0
[root@rdma4 network-scripts]# ifdown ib0
[root@rdma4 network-scripts]# vi ifcfg-ib0
[root@rdma4 network-scripts]# cat ifcfg-ib0
DEVICE="ib0"
ONBOOT="yes"
MTU="1500"
NETWORKING_IPV6="yes"
IPADDR="10.116.36.86/26"
[root@rdma4 network-scripts]# ifup ib0
ipcalc: both netmask and prefix specified
Usage: ipcalc [OPTION...]
  -c, --check         Validate IP address for specified address family
  -4, --ipv4          IPv4 address family (default)
  -6, --ipv6          IPv6 address family
  -b, --broadcast     Display calculated broadcast address
  -h, --hostname      Show hostname determined via DNS
  -m, --netmask       Display default netmask for IP (class A, B, or C)
  -n, --network       Display network address
  -p, --prefix        Display network prefix
  -s, --silent        Don't ever display error messages

Help options:
  -?, --help          Show this help message
  --usage             Display brief usage message
ipcalc: both netmask and prefix specified
Usage: ipcalc [OPTION...]
  -c, --check         Validate IP address for specified address family
  -4, --ipv4          IPv4 address family (default)
  -6, --ipv6          IPv6 address family
  -b, --broadcast     Display calculated broadcast address
  -h, --hostname      Show hostname determined via DNS
  -m, --netmask       Display default netmask for IP (class A, B, or C)
  -n, --network       Display network address
  -p, --prefix        Display network prefix
  -s, --silent        Don't ever display error messages

Help options:
  -?, --help          Show this help message
  --usage             Display brief usage message
ipcalc: both netmask and prefix specified
Usage: ipcalc [OPTION...]
  -c, --check         Validate IP address for specified address family
  -4, --ipv4          IPv4 address family (default)
  -6, --ipv6          IPv6 address family
  -b, --broadcast     Display calculated broadcast address
  -h, --hostname      Show hostname determined via DNS
  -m, --netmask       Display default netmask for IP (class A, B, or C)
  -n, --network       Display network address
  -p, --prefix        Display network prefix
  -s, --silent        Don't ever display error messages

Help options:
  -?, --help          Show this help message
  --usage             Display brief usage message
arping: unknown host 10.116.36.86/26
Error, some other host already uses address 10.116.36.86/26.
Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
[root@rdma4 network-scripts]# ip addr show ib0
6: ib0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 256
    link/infiniband 80:00:00:03:fe:80:00:00:00:00:00:00:00:11:75:00:00:68:70:9f brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
[root@rdma4 network-scripts]#
Comment 15 Honggang LI 2011-09-01 01:33:05 UTC 
(In reply to comment #13)
> Adding the PREFIX to the end of IPADDR was never intended to work (it simply
> worked around the incompatibility between the rhel6.1 ifup-ib file and the
> expand_config function).

Based on Doug's comment, the reproduction and verification of comment 14, this bug had been set as 'VERIFIED'.
Comment 16 Doug Ledford 2011-10-07 13:44:22 UTC 
*** Bug 744128 has been marked as a duplicate of this bug. ***
Comment 17 errata-xmlrpc 2011-12-06 15:33:07 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2011-1639.html