Bug 1130804

This bug is just a kind request to backport the following fix to EL7. Thank you in advance! :-)

for i in $(seq 50); do
    linklocal=$(ip -6 addr show dev ${interface} scope link)
    tentative=$(echo "${linklocal}" | grep tentative)
    [[ -n "${linklocal}" &&  -z "${tentative}" ]] && exit_with_hooks 0
    sleep 0.1
done

+++ This bug was initially created as a clone of Bug #1129500 +++

For testing VDSM's (part of oVirt) IPv6 capabilities, I have set up a simple virtual ethernet environment with dnsmasq acting as a DHCPv6 server on one side and ifup (run by VDSM or alone) on the other.

Only DHCPV6C is enabled in an ifcfg file – the setup is meant to rely on DHCPv6.

As there is no delay after assigning a link-local address to the 'client' side, the address is reported as 'tentative' (by e.g. 'ip addr show veth_45') and cannot yet be used. dhclient -6 thus refuses to start. From the journal, its error output is as following:

Can't bind to dhcp address: Cannot assign requested address
Please make sure there is no other dhcp server
running and that there's no entry for dhcp or
bootp in /etc/inetd.conf.   Also make sure you
are not running HP JetAdmin software, which
includes a bootp server.

Giving the address some time to "stabilize" is enough (and a must). As a preliminary solution, I added a 'sleep 1' before 'dhclient -6' invocation in ifup-eth (before line 311 in https://git.fedorahosted.org/cgit/initscripts.git/tree/sysconfig/network-scripts/ifup-eth#n311)

A simple way to fix this issue is to busy-wait until 'ip -6 -o addr show veth_45' stops reporting the 'tentative' flag. A harder one is to wait in dhclient.

ifcfg-veth_45 is attached, I will also add a shell script with commands used for testing. For convenience, they are also below:

# Generated by VDSM version 4.16.0-175.gite9e8110.fc20
DEVICE=veth_45
#HWADDR=52:27:15:53:22:04
ONBOOT=no
BOOTPROTO=none
DEFROUTE=no
NM_CONTROLLED=no
IPV6INIT=yes
DHCPV6C=yes
IPV6_AUTOCONF=no

ip link add name veth_63 type veth peer name veth_45
ip addr add dev veth_63 fdb3:84e5:4ff4:55e3::1/64
ip link set dev veth_63 up 
firewall-cmd --zone=trusted --change-interface=veth_63
dnsmasq --dhcp-authoritative -p 0 --dhcp-option=3 -O 6 -i veth_63 -I lo --bind-dynamic --dhcp-range=fdb3:84e5:4ff4:55e3::a,fdb3:84e5:4ff4:55e3::64,2m
ifup veth_45

--- Additional comment from Ondřej Svoboda on 2014-08-12 20:55:24 EDT ---



--- Additional comment from Jiri Popelka on 2014-08-13 10:19:31 EDT ---

(In reply to Ondřej Svoboda from comment #0)
> A simple way to fix this issue is to busy-wait until 'ip -6 -o addr show
> veth_45' stops reporting the 'tentative' flag. A harder one is to wait in
> dhclient.

We might add such a waiting into dhclient-script after we 'up' the interface in PREINIT6, see line 707 in
http://pkgs.fedoraproject.org/cgit/dhcp.git/tree/dhclient-script

--- Additional comment from Jiri Popelka on 2014-08-13 10:41:04 EDT ---

Something like this works for me:

    PREINIT6)
        # ensure interface is up
        ip link set dev ${interface} up
        
        # remove any stale addresses from aborted clients
        ip -6 addr flush dev ${interface} scope global permanent

        for i in $(seq 3); do
            # tentative flag == DAD is still not complete
            tentative=$(ip -6 addr show dev ${interface} scope link | grep tentative)
            if [ -z "${tentative}" ] ; then
                # DAD is over
                exit_with_hooks 0
            fi
            sleep 1
        done

        exit_with_hooks 0
        ;;

Could you try to modify PREINIT6 in /usr/sbin/dhclient-script like that and see if it fixes the problem for you Ondrej ?

--- Additional comment from Ondřej Svoboda on 2014-08-13 12:21:17 EDT ---

I confirm this approach works. It takes ~1.5 second for the link-local address to lose the tentative flag and approx. 2 seconds for the global address, "measured" using the following snippet in the same environment as originally.

 while true; do date +%X.%N; ip addr show dev veth_45; sleep 0.1; done;

To avoid waiting too much, could we shorten the period to less than half a second? If it would not be too resource-intensive (which I doubt), even go down to 0.1 s and have 30–50 cycles (in a similar situation, there is a 5s limit in dhclient-script [1]).

[1] http://pkgs.fedoraproject.org/cgit/dhcp.git/tree/dhclient-script#n563

--- Additional comment from Jiri Popelka on 2014-08-14 05:43:21 EDT ---

Fixed in rawhide/F21 with:
http://pkgs.fedoraproject.org/cgit/dhcp.git/commit/?h=f21&id=991bd354d956cc2f31ba75689ddcca0021706f0b

I'd rather not push this into F20.

--- Additional comment from Ondřej Svoboda on 2014-08-14 07:38:38 EDT ---

It does not happen 100% of time but the very existence of the link-local address is not guaranteed that early (after 100ms). We also have to check that

 linklocal=$(ip -6 addr show dev ${interface} scope link)
 …
 if [ ! -z "${linklocal}" -a -z "${tentative}" ] ; then

Dan, I assume we need this change backported to EL because that is where VDSM is supposed to run. What systems do we want to support DHCPv6 in?

--- Additional comment from Jiri Popelka on 2014-08-14 07:56:44 EDT ---

Ondrej, what about this ?

for i in $(seq 50); do
    linklocal=$(ip -6 addr show dev ${interface} scope link)
    tentative=$(echo "${linklocal}" | grep tentative)
    [[ -n "${linklocal}" &&  -z "${tentative}" ]] && exit_with_hooks 0
    sleep 0.1
done

--- Additional comment from Ondřej Svoboda on 2014-08-14 08:11:32 EDT ---

Jirka, thanks, this is it exactly and work reliably :-)

--- Additional comment from Dan Kenigsberg on 2014-08-14 08:52:33 EDT ---

Jiri, can we have this backported to f20, el7 and el6? We would oVirt's Vdsm to start using dhcp6.

--- Additional comment from Jiri Popelka on 2014-08-14 09:07:19 EDT ---

(In reply to Dan Kenigsberg from comment #9)
> Jiri, can we have this backported to f20

Sure, dhcp-4.2.7-2.fc20 should do the trick:
http://pkgs.fedoraproject.org/cgit/dhcp.git/commit/?h=f20&id=88bb3be48ee88795f43ba90c4ef1bf3a33a03c96

> el7 and el6? 

You have to clone this bug for these products if you want it to be fixed there.

--- Additional comment from Fedora Update System on 2014-08-15 03:56:52 EDT ---

dhcp-4.2.7-2.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/FEDORA-2014-8287/dhcp-4.2.7-2.fc20

--- Additional comment from Fedora Update System on 2014-08-15 20:29:50 EDT ---

Package dhcp-4.2.7-2.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dhcp-4.2.7-2.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-8287/dhcp-4.2.7-2.fc20
then log in and leave karma (feedback).