Bug 782042 - network service finishes before assigned IPv6 addresses leave tentative state
Summary: network service finishes before assigned IPv6 addresses leave tentative state
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: initscripts
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Lukáš Nykrýn
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 896235 (view as bug list)
Depends On:
Blocks: network-online.target
TreeView+ depends on / blocked
 
Reported: 2012-01-16 11:56 UTC by Michal Schmidt
Modified: 2016-05-25 09:57 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 781708
Environment:
Last Closed: 2015-06-29 11:37:46 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Michal Schmidt 2012-01-16 11:56:56 UTC
+++ This bug was initially created as a clone of Bug #781708 +++

Description of problem:

This is a problem which started to show up today.  After reboot, sshd
failed to listen on my local IPv6 address.

I have a static network configuration using the network init script.
I have a sshd configuration which defines various ListenAddresses
in /etc/ssh/sshd_config, like this:

  # localhost
  ListenAddress 127.0.0.1:22
  ListenAddress [::1]:22
  # local only addresses
  ListenAddress 192.168.1.1:22
  ListenAddress [fc00::1]:22
  # external address
  ListenAddress a.b.c.d:12345

The last time before today I rebooted the machine was 2 days ago.  This
was right after I updated to the new 3.1.7 kernel via yum.  Everything
worked fine.

Today I installed the last set of updates via yum and rebooted again.
This time I was not able to connect to that machine via IPv6.  So
I tried netstat:

  $ netstat -tnl | grep :22
  tcp   0  0 127.0.0.1:22      0.0.0.0:*   LISTEN
  tcp   0  0 192.168.1.1:22    0.0.0.0:*   LISTEN
  tcp   0  0 ::1:22            :::*        LISTEN

Where is fc00::1?  ifconfig showed clearly that the fc00::1 address
was configured and ready.  `systemctl restart sshd.service' worked fine
and afterwards sshd was listening on fc00::1:22 as well.

I had a look into /var/log/secure:

  12:32:24 sshd[1360]: Server listening on a.b.c.d port 12345.
  12:32:24 sshd[1360]: error: Bind to port 22 on fc00::1 failed:
                       Cannot assign requested address.
  12:32:24 sshd[1360]: Server listening on 192.168.1.1 port 22.
  12:32:24 sshd[1360]: Server listening on ::1 port 22.
  12:32:24 sshd[1360]: Server listening on 127.0.0.1 port 22.

Ok, so it was no problem to set up the listening sockets on any other
requested IP address, just trying to lsten on fc00::1 failed with
"Cannot assign requested address".  So what about setting up the network?

[...]

--- Additional comment from mschmidt on 2012-01-14 16:42:07 EST ---
[...]
When an IPv6 address is assigned to an interface, it does not become usable immediately. It is in a "tentative" state for a moment for the purpose of a mandatory duplicate address detection.

You can see it with an experiment:
 ifdown eth0
 ifup eth0; ip addr show dev eth0
You'll see something like:
 ...
 inet6 fc00::1/64 scope global tentative
 ...
When you ask ip the same question again a little later, the "tentative" flag will be gone.

Ideally sshd would use IP_FREEBIND to avoid the need for ordering.
Reassigning to openssh.

These discussion threads are relevant:
http://lists.fedoraproject.org/pipermail/devel/2011-May/151272.html
http://lists.debian.org/debian-devel/2011/05/msg00801.html

--- Additional comment from vinschen on 2012-01-15 05:56:04 EST ---

Ah, I see.  But I don't think openssh is the culprit in the first place,
even if it might make sense to patch IP_FREEBIND into it.

The underlying problem is that the dependency mechanism is broken by the
fact that the initscripts network script reports success before all
addresses are actually available.  From my POV network should only
report success when all network addresses are available *and* left the
tentative state.  This would do the right thing in non-NetworkManager
scenarios which don't have to cope with dynamic addresses.

Comment 1 Corinna Vinschen 2012-01-24 10:34:10 UTC
Ping?  Am I wrong in my assumption that the network initscript should
make sure that all configured addresses have left the tentative state?


Corinna

Comment 2 Bill Nottingham 2012-01-24 17:11:54 UTC
The problem is finding a good way to do so... busywaiting on /sbin/ip screenscraping isn't a good answer.

Comment 3 Corinna Vinschen 2012-01-24 21:05:23 UTC
I see what you mean.  I searched for this problem and apparently I'm not
the first one who got a problem with this behaviour.  I found the following
potential solutions/workarounds/hacks:

  while [ -n "$(ip -6 addr show tentative)" ]; do sleep 1; done

or something along the lines of

  echo 0 >/proc/sys/net/ipv6/conf/${interface}/accept_dad
  ifup
  echo 1 >/proc/sys/net/ipv6/conf/${interface}/accept_dad

However, while looking into this I found just a few minutes ago that there
is already a command in the network initscript, right at the end of the
"start" case, line 171:

  [ -n "${NETWORKDELAY}" ] && /bin/sleep ${NETWORKDELAY}

NETWORKDELAY isn't mentioned anywhere and it doesn't show up in
/etc/sysconfig/network either.  For the time being this is a working
workaround, but a simple sleep looks rather lame.


Corinna

Comment 4 Bill Nottingham 2012-01-24 22:30:00 UTC
NETWORKDELAY is generally used for cases where the switch takes time to negotiate and won't pass packets - there is *no* good way to test for that, so it's just a unilateral admin-defined delay.

I suspect disabling DAD would be enough of a behavior change that it would break for some people.

Comment 5 Corinna Vinschen 2012-01-25 08:58:03 UTC
However, shouldn't NETWORKDELAY be mentioned in /etc/sysconfig/network at least?

I understand that disabling DAD might be dangerous in the general case, but
is DAD really a problem for servers which are configured with fixed IP
addresses?


Corinna

Comment 6 Bill Nottingham 2012-01-25 16:54:12 UTC
NETWORKDELAY is mentioned in /usr/share/doc/initscripts-<version>/sysconfig.txt

The duplicate IP check is only done for static IP (for IPv4)... the assumption  is that if your server is dishing out static IPs, it's smart enough to not get in this situation.

Comment 7 Fedora End Of Life 2013-01-16 16:08:34 UTC
This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 8 Bill Nottingham 2013-01-18 21:49:50 UTC
*** Bug 896235 has been marked as a duplicate of this bug. ***

Comment 9 Fedora Admin XMLRPC Client 2013-09-04 14:49:27 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 10 Fedora End Of Life 2013-12-21 08:31:58 UTC
This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 11 Fedora End Of Life 2014-02-05 11:53:04 UTC
Fedora 18 changed to end-of-life (EOL) status on 2014-01-14. Fedora 18 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 12 Sander Steffann 2014-02-24 12:53:52 UTC
This bug still exists in Fedora 20. Please re-open.

Comment 13 Scott Shambarger 2014-03-14 05:06:30 UTC
ifup-ipv6 probably needs a test similar to one I use in my NetworkManager dispatcher scripts before starting daemons:

# 2.5 sec max                                                                   
for i in $(seq 25); do                                                          
  [ -z "$(/sbin/ip -6 addr show dev $DEVICE tentative)" ] && break              
  /bin/sleep 0.1                                                                
done                                                                            
fail=$(/sbin/ip -6 addr show dev $DEVICE dadfailed)                             

Ideally, it'd be nice if both initscripts and NetworkManager would report success only after all configured addresses leaves the tentative state, so daemons that bind to specific (ie manually configured) addresses wouldn't fail with "no such address" errors... guessing NetworkManager probably needs its own bug.

Bug does need to be re-opened and updated to F20.

Comment 14 Matthew Miller 2014-07-14 15:49:38 UTC
Reopening and moving to F20 as per comments above.

Comment 15 Pavel Šimerda (pavlix) 2014-07-15 10:07:51 UTC
(In reply to Scott Shambarger from comment #13)
> ifup-ipv6 probably needs a test similar to one I use in my NetworkManager
> dispatcher scripts before starting daemons:

By the way, what's the current status of NetworkManager and IPv6 tentative addresses? I would guess that if you run `nm-online -s`, you should get a working environment, otherwise it's a bug and should be reported.

> Ideally, it'd be nice if both initscripts and NetworkManager would report
> success only after all configured addresses leaves the tentative state,

+1

Comment 16 Pavel Šimerda (pavlix) 2014-08-14 17:07:21 UTC
(In reply to Scott Shambarger from comment #13)
> Ideally, it'd be nice if both initscripts and NetworkManager would report
> success only after all configured addresses leaves the tentative state, so
> daemons that bind to specific (ie manually configured) addresses wouldn't
> fail with "no such address" errors... guessing NetworkManager probably needs
> its own bug.

+1 for the initscripts but I think NetworkManager handles this already (at least in upstream).

Comment 17 Fedora End Of Life 2015-05-29 08:42:09 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 18 Fedora End Of Life 2015-06-29 11:37:46 UTC
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 19 Jari Korva 2016-05-25 09:57:12 UTC
I ran into a similar looking problem with up-to-date Centos 7. I have eth2 for LAN (IPv4&IPv6) and eth3 with statically configured IPv6 ULA address for iscsi (no IPv4). NetworkManager is disabled. During boot I get:

kernel: IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
kernel: tg3 0000:03:00.0 eth2: Link is up at 1000 Mbps, full duplex
kernel: tg3 0000:03:00.0 eth2: Flow control is off for TX and off for RX
kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
network[842]: Bringing up interface eth2:  [  OK  ]
kernel: tg3 0000:04:00.0: irq 29 for MSI/MSI-X
kernel: IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
network[842]: Bringing up interface eth3:  [  OK  ]
systemd[1]: Started LSB: Bring up/down networking.
systemd[1]: Reached target Network.
systemd[1]: Starting Network.
systemd[1]: Reached target Network is Online.
systemd[1]: Started Open-iSCSI.
systemd[1]: Starting Logout off all iSCSI sessions on shutdown...
systemd[1]: Starting Login and scanning of iSCSI devices...
systemd[1]: Started Logout off all iSCSI sessions on shutdown.
systemd[1]: Started NFS Mount Daemon.
iscsi-mark-root-nodes[1286]: iscsiadm: No active sessions.
iscsiadm[1298]: iscsiadm: No records found
systemd[1]: Started Login and scanning of iSCSI devices.
systemd[1]: Starting Availability of block devices...
systemd[1]: Reached target Remote File Systems (Pre).
systemd[1]: Starting Remote File Systems (Pre).
systemd[1]: Started Availability of block devices.
kernel: tg3 0000:04:00.0 eth3: Link is up at 1000 Mbps, full duplex
kernel: tg3 0000:04:00.0 eth3: Flow control is on for TX and on for RX
kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready

After boot is complete, manually executed "iscsiadm -m node --login" mounts failed iscsi disks.

Should the bug be reopened for RHEL/Centos?

My ifcfg-eth3:

DEVICE=eth3
HWADDR=00:...
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
IPV6_AUTOCONF=no
IPV6ADDR=fd00::1/64


Note You need to log in before you can comment on or make changes to this bug.