Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2034799

Summary: IPv6 Default gateway fail to be configured when interface have both IPv4 and IPv6 address - IPv4 brings iface up, then IPv6 initilalization set accept_ra = 0. Since interface was already up default route was added upon receiving Router Advertisement.
Product: Red Hat Enterprise Linux 8 Reporter: Shailesh Chhabdiya <schhabdi>
Component: initscriptsAssignee: Jan Macku <jamacku>
Status: CLOSED ERRATA QA Contact: Daniel Rusek <drusek>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.2CC: abhijadh, bfournie, chrisw, dsneddon, fgadkano, hbrock, hjensas, initscripts-maint-list, jamacku, jslagle, lnykryn, mburns, msekleta, rskvaril, sbaker, scohen, tkajinam
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: 8.2Flags: pm-rhel: mirror+
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2040679 2118274 2118275 (view as bug list) Environment:
Last Closed: 2022-11-08 10:56:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2040679, 2118274, 2118275    
Attachments:
Description Flags
journal -u network.service from reproducer described in Comment 11 none

Description Shailesh Chhabdiya 2021-12-22 08:03:15 UTC 
Description of problem:
IPv6 Default gateway over Management Network doesn't gets added on nodes

Version-Release number of selected component (if applicable):
RHOSP 16.1.6


Actual results:
IPv6 Routes are not created 

Expected results:
IPv6 Route Should be created 

Additional info:
Comment 11 Harald Jensås 2021-12-23 14:47:14 UTC 
I think I managed to reproduce this with this configuration:

network_config:
- name: api-bond
  type: linux_bond
  use_dhcp: false
  mtu: 1442
  bonding_options: mode=active-backup updelay=1000 miimon=100
  members:
  - name: ens3
    type: interface
    mtu: 1442
    primary: true
  addresses:
  - ip_netmask: 192.168.100.10/26
  - ip_netmask: fd00:fd00:fd00:1::1:1bb/64
  routes:
  - default: true
    next_hop: 192.168.100.10
  - default: true
    next_hop: fd00:fd00:fd00:1::fffe

[root@controller-0 ~]# ip -6 route
::1 dev lo proto kernel metric 256 pref medium
unreachable ::/96 dev lo metric 1024 pref medium
unreachable ::ffff:0.0.0.0/96 dev lo metric 1024 pref medium
unreachable 2002:a00::/24 dev lo metric 1024 pref medium
unreachable 2002:7f00::/24 dev lo metric 1024 pref medium
unreachable 2002:a9fe::/32 dev lo metric 1024 pref medium
unreachable 2002:ac10::/28 dev lo metric 1024 pref medium
unreachable 2002:c0a8::/32 dev lo metric 1024 pref medium
unreachable 2002:e000::/19 dev lo metric 1024 pref medium
unreachable 3ffe:ffff::/32 dev lo metric 1024 pref medium
fd00:fd00:fd00:1::/64 dev api-bond proto kernel metric 256 pref medium
fe80::/64 dev api-bond proto kernel metric 256 pref medium

[root@controller-0 ~]# ip route
default via 192.168.100.10 dev api-bond 
192.168.100.0/26 dev api-bond proto kernel scope link src 192.168.100.10 

[root@controller-0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-api-bond 
# This file is autogenerated by os-net-config
DEVICE=api-bond
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
MACADDR="fa:16:3e:a6:86:55"
BONDING_OPTS="mode=active-backup updelay=1000 miimon=100"
MTU=1442
IPV6INIT=yes
IPV6_MTU=1442
BOOTPROTO=static
IPADDR=192.168.100.10
NETMASK=255.255.255.192
IPV6_AUTOCONF=no
IPV6ADDR=fd00:fd00:fd00:1::1:1bb/64

[root@controller-0 ~]# cat /etc/sysconfig/network-scripts/route-api-bond 
default via 192.168.100.10 dev api-bond

[root@controller-0 ~]# cat /etc/sysconfig/network-scripts/route6-api-bond 
default via fd00:fd00:fd00:1::fffe dev api-bond


I only see the problem after adding both IPv4 and IPv6 address to the same interface.

Dec 23 14:33:03 controller-0 systemd[1]: Starting LSB: Bring up/down networking...
Dec 23 14:33:03 controller-0 network[2653]: WARN      : [network] You are using 'network' service provided by 'network-scripts', which ar>
Dec 23 14:33:03 controller-0 network[2653]: WARN      : [network] 'network-scripts' will be removed in one of the next major releases of >
Dec 23 14:33:03 controller-0 network[2653]: WARN      : [network] It is advised to switch to 'NetworkManager' instead for network managem>
Dec 23 14:33:04 controller-0 network[2653]: Bringing up loopback interface:  [  OK  ]
Dec 23 14:33:08 controller-0 network[2653]: Bringing up interface api-bond:  INFO      : [ipv6_wait_tentative] Waiting for interface api->
Dec 23 14:33:09 controller-0 network[2653]: INFO      : [ipv6_wait_tentative] Waiting for interface api-bond IPv6 address(es) to leave th>
Dec 23 14:33:10 controller-0 network[2653]: RTNETLINK answers: File exists


After enabling rc.debug on the kernel command line, I can see that the "RTNETLINK answers: File exists" is logger right after running ifup-routes:

Dec 23 14:35:43 controller-0 network[1155]: + /etc/sysconfig/network-scripts/ifup-routes api-bond api-bond
Dec 23 14:35:43 controller-0 network[1155]: RTNETLINK answers: File exists

I will attach the full log from my reproducer to this Bz.


It would be very helpful if we can get the journal from a customer system with 'rc.debug' enabled on the kernel command line to confirm that we see similar events as to what I see in the reproducer.
@Shailesh can you collect that for us? Thank you!
Comment 12 Harald Jensås 2021-12-23 14:48:53 UTC 
Created attachment 1847596 [details]
journal -u network.service from reproducer described in Comment 11

journal -u network.service from reproducer described in Comment 11
Comment 13 Harald Jensås 2021-12-23 15:08:28 UTC 
Adding 'set -x' to the beginning of /etc/sysconfig/network-scripts/ifup-routes and adding '/sbin/ip -6 route show' command just before the '/sbin/ip -6 route add'.

Dec 23 15:01:09 controller-0 ipv6_wait_tentative[1484]: Waiting for interface api-bond IPv6 address(es) to leave the 'tentative' state
Dec 23 15:01:10 controller-0 network[1147]: INFO      : [ipv6_wait_tentative] Waiting for interface api-bond IPv6 address(es) to leave the 'tentative' state
Dec 23 15:01:12 controller-0 network[1147]: + '[' -z api-bond ']'
Dec 23 15:01:12 controller-0 network[1147]: + MATCH='^[[:space:]]*(\#.*)?$'
Dec 23 15:01:12 controller-0 network[1147]: + FILES='/etc/sysconfig/network-scripts/route-api-bond /etc/sysconfig/network-scripts/route6-api-bond'
Dec 23 15:01:12 controller-0 network[1147]: + '[' -n api-bond -a api-bond '!=' api-bond ']'
Dec 23 15:01:12 controller-0 network[1147]: + for file in $FILES
Dec 23 15:01:12 controller-0 network[1147]: + '[' -f /etc/sysconfig/network-scripts/route-api-bond ']'
Dec 23 15:01:12 controller-0 network[1147]: + grep -Eq '^[[:space:]]*ADDRESS[0-9]+=' /etc/sysconfig/network-scripts/route-api-bond
Dec 23 15:01:12 controller-0 network[1147]: + handle_ip_file /etc/sysconfig/network-scripts/route-api-bond
Dec 23 15:01:12 controller-0 network[1147]: + local f t type= file=/etc/sysconfig/network-scripts/route-api-bond proto=-4
Dec 23 15:01:12 controller-0 network[1147]: + f=route-api-bond
Dec 23 15:01:12 controller-0 network[1147]: + t=route
Dec 23 15:01:12 controller-0 network[1147]: + type=route
Dec 23 15:01:12 controller-0 network[1147]: + '[' route '!=' route ']'
Dec 23 15:01:12 controller-0 network[1147]: + read line
Dec 23 15:01:12 controller-0 network[1147]: + cat /etc/sysconfig/network-scripts/route-api-bond
Dec 23 15:01:12 controller-0 network[1147]: + [[ ! default via 192.168.100.10 dev api-bond =~ ^[[:space:]]*(\#.*)?$ ]]
Dec 23 15:01:12 controller-0 network[1147]: + echo
Dec 23 15:01:12 controller-0 network[1147]: + /sbin/ip -6 route show
Dec 23 15:01:12 controller-0 network[1147]: ::1 dev lo proto kernel metric 256 pref medium
Dec 23 15:01:12 controller-0 network[1147]: fd00:fd00:fd00:1::/64 dev api-bond proto kernel metric 256 pref medium
Dec 23 15:01:12 controller-0 network[1147]: fe80::/64 dev api-bond proto kernel metric 256 pref medium
Dec 23 15:01:12 controller-0 network[1147]: default via fe80::f816:3eff:fe6d:4a4d dev api-bond proto ra metric 1024 expires 1794sec hoplimit 64 pref medium
Dec 23 15:01:12 controller-0 network[1147]: + /sbin/ip -4 route add default via 192.168.100.10 dev api-bond
Dec 23 15:01:12 controller-0 network[1147]: + read line
Dec 23 15:01:12 controller-0 network[1147]: + [[ ! '' =~ ^[[:space:]]*(\#.*)?$ ]]
Dec 23 15:01:12 controller-0 network[1147]: + read line
Dec 23 15:01:12 controller-0 network[1147]: + for file in $FILES
Dec 23 15:01:12 controller-0 network[1147]: + '[' -f /etc/sysconfig/network-scripts/route6-api-bond ']'
Dec 23 15:01:12 controller-0 network[1147]: + grep -Eq '^[[:space:]]*ADDRESS[0-9]+=' /etc/sysconfig/network-scripts/route6-api-bond
Dec 23 15:01:12 controller-0 network[1147]: + handle_ip_file /etc/sysconfig/network-scripts/route6-api-bond
Dec 23 15:01:12 controller-0 network[1147]: + local f t type= file=/etc/sysconfig/network-scripts/route6-api-bond proto=-4
Dec 23 15:01:12 controller-0 network[1147]: + f=route6-api-bond
Dec 23 15:01:12 controller-0 network[1147]: + t=route6
Dec 23 15:01:12 controller-0 network[1147]: + type=route
Dec 23 15:01:12 controller-0 network[1147]: + '[' route '!=' route6 ']'
Dec 23 15:01:12 controller-0 network[1147]: + proto=-6
Dec 23 15:01:12 controller-0 network[1147]: + read line
Dec 23 15:01:12 controller-0 network[1147]: + cat /etc/sysconfig/network-scripts/route6-api-bond
Dec 23 15:01:12 controller-0 network[1147]: + [[ ! default via fd00:fd00:fd00:1::fffe dev api-bond =~ ^[[:space:]]*(\#.*)?$ ]]
Dec 23 15:01:12 controller-0 network[1147]: + echo

# Added /sbin/ip -6 route show command for debug:

Dec 23 15:01:12 controller-0 network[1147]: + /sbin/ip -6 route show
Dec 23 15:01:12 controller-0 network[1147]: ::1 dev lo proto kernel metric 256 pref medium
Dec 23 15:01:12 controller-0 network[1147]: fd00:fd00:fd00:1::/64 dev api-bond proto kernel metric 256 pref medium
Dec 23 15:01:12 controller-0 network[1147]: fe80::/64 dev api-bond proto kernel metric 256 pref medium
Dec 23 15:01:12 controller-0 network[1147]: default via fe80::f816:3eff:fe6d:4a4d dev api-bond proto ra metric 1024 expires 1794sec hoplimit 64 pref medium

# There is an existing default route, which was most likely learned from RA since it has the "expires ###sec" and "pref medium" and has the link local address

Dec 23 15:01:12 controller-0 network[1147]: + /sbin/ip -6 route add default via fd00:fd00:fd00:1::fffe dev api-bond
Dec 23 15:01:12 controller-0 network[1147]: RTNETLINK answers: File exists
Comment 14 Harald Jensås 2021-12-23 15:36:58 UTC 
I think the problem is that the link is brought up prior to disabling accept_ra.

Dec 23 14:35:37 controller-0 network[1155]: + ip link set dev api-bond up

Dec 23 14:35:41 controller-0 network[1155]: + /etc/sysconfig/network-scripts/ifup-ipv6 ifcfg-api-bond
Dec 23 14:35:41 controller-0 network[1155]: + /sbin/sysctl -e -w net.ipv6.conf.api-bond.accept_ra=0


This only happens when both IPv4 and IPv6 address is on the interface, because network-scripts does IPv4 first and brings up the interface.
Then it does IPv6 which disables the RA's, at that point the node has already learned the route from RA and setting a default route fails.
Comment 17 Dan Sneddon 2021-12-23 16:24:18 UTC 
(In reply to Harald Jensås from comment #14)
> I think the problem is that the link is brought up prior to disabling
> accept_ra.
> 
> Dec 23 14:35:37 controller-0 network[1155]: + ip link set dev api-bond up
> 
> Dec 23 14:35:41 controller-0 network[1155]: +
> /etc/sysconfig/network-scripts/ifup-ipv6 ifcfg-api-bond
> Dec 23 14:35:41 controller-0 network[1155]: + /sbin/sysctl -e -w
> net.ipv6.conf.api-bond.accept_ra=0
> 
> 
> This only happens when both IPv4 and IPv6 address is on the interface,
> because network-scripts does IPv4 first and brings up the interface.
> Then it does IPv6 which disables the RA's, at that point the node has
> already learned the route from RA and setting a default route fails.

I wonder if we could mitigate the issue by using two /1 routes instead of one default route for IPv6:

computeovsdpdksriov.yaml

              - type: vlan
                device: api-bond
                mtu: 9000
                vlan_id:
                  get_param: ManagementNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: ManagementIpSubnet
                - ip_netmask:
                    get_param: ManagementIpv6IpSubnet
                routes:
                  - ip_netmask: 0.0.0.0/0
                    next_hop:
                      get_param: ManagementInterfaceDefaultRoute
                    default: true
                  - ip_netmask: '::/1'
                    next_hop:
                      get_param: ManagementIpv6InterfaceDefaultRoute
                  - ip_netmask: '8000::/1'
                    next_hop:
                      get_param: ManagementIpv6InterfaceDefaultRoute
Comment 20 Harald Jensås 2021-12-24 18:34:52 UTC 
Another workaround:

a) Create a script file: '/sbin/ifup-local' with below content. (NOTE: replace "api-bond" with the correct interface name for the IPv6 default gateway.)

cat << EOF > /sbin/ifup-local
if [[ "$1" == "api-bond" ]]; then
  NEXT_WAIT_TIME=0
  COMMAND_STATUS=0

  /sbin/ip -6 route show | /usr/bin/grep --quiet "default via .* dev $1"
  COMMAND_STATUS=$?

  # Wait for default route from RA to be removed.
  until [[ $COMMAND_STATUS -eq 1 || $NEXT_WAIT_TIME -eq 5 ]]; do
    /sbin/ip -6 route show | /usr/bin/grep --quiet "default via .* dev $1"
    COMMAND_STATUS=$?
    echo "Wait for IPv6 default route from RA to be removed"
    /usr/bin/sleep $NEXT_WAIT_TIME
    let NEXT_WAIT_TIME=NEXT_WAIT_TIME+1
  done
  
  # Add IPv6 routes
  cat /etc/sysconfig/network-scripts/route6-$1 | while read line
  do
    # Only add route if the line starts with "default"
    if [[ $line =~ ^"default".* ]]; then
      /sbin/ip -6 route add $line
    fi
  done
fi
EOF

b) Make the file executable:

   chmod 700 /sbin/ifup-local
Comment 41 Jan Macku 2022-07-22 10:42:23 UTC 
Hi,

Once I'll have a fix for an issue I'm going to backport it to RHEL 8.4 as well.
Comment 50 errata-xmlrpc 2022-11-08 10:56:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (initscripts bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7798