Bug 1539098

Summary: Configuring metadata network-interfaces with "iface eth2 inet6 static" breaks all networking on cloud-init instance
Product: Red Hat Enterprise Linux 7 Reporter: Andreas Karis <akaris>
Component: cloud-initAssignee: Ryan McCabe <rmccabe>
Status: CLOSED CURRENTRELEASE QA Contact: Vratislav Hutsky <vhutsky>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 7.4CC: anthony.mcmahon, eterrell, huzhao, jgreguske, linl, mkalinin, rmccabe, yacao, yuxisun
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-11 13:23:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andreas Karis 2018-01-26 16:04:26 UTC
Description of problem:
Configuring metadata network-interfaces with "iface eth2 inet6 static" breaks all networking on cloud-init instance. This only works when configuring a sub interface.

The following breaks networking:
~~~
auto eth2
iface eth2 inet static
address 192.168.0.166
network services
netmask 255.255.0.0
broadcast 192.168.255.255
gateway 192.168.0.1
iface eth2 inet6 static
address 2001::166
gateway 2001::1
hwaddress aa:aa:aa:bb:bb:bb
~~~

According to https://wiki.debian.org/NetworkConfiguration , this should work, though:
+++
If you're configuring it manually then something like this will set the default gateway (network, broadcast and gateway are optional):

    auto eth0
    iface eth0 inet static
        address 192.0.2.7
        netmask 255.255.255.0
        gateway 192.0.2.254

If you want to add an IPv6 address, too, append something like:

    iface eth0 inet6 static
        address 2001:db8::c0ca:1eaf
        netmask 64
        gateway 2001:db8::1ead:ed:beef
+++

And https://manpages.debian.org/jessie/ifupdown/interfaces.5.en.html
+++
Options are usually indented for clarity (as in the example above) but are not required to be. 
+++

The following works correctly:
~~~
iface eth2 inet static
address 192.168.0.166
network services
netmask 255.255.0.0
broadcast 192.168.255.255
gateway 192.168.0.1
auto eth2:0
iface eth2:0 inet6 static
address 2001::166
gateway 2001::1
hwaddress aa:aa:aa:bb:bb:bb
~~~

Additional info:
result working:


Working instance: ./sosreport-20180125-075718/svc-1-lvsrouter/var/log/cloud-init-output.log
~~~
Cloud-init v. 0.7.9 running 'init-local' at Thu, 25 Jan 2018 07:48:44 +0000. Up 12.41 seconds.
Cloud-init v. 0.7.9 running 'init' at Thu, 25 Jan 2018 07:48:47 +0000. Up 16.00 seconds.
ci-info: +++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: | Device |  Up  |    Address     |      Mask     | Scope |     Hw-Address    |
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: |  lo:   | True |   127.0.0.1    |   255.0.0.0   |   .   |         .         |
ci-info: |  lo:   | True |       .        |       .       |   d   |         .         |
ci-info: | eth1:  | True | 10.250.246.171 | 255.255.252.0 |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth1:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth2:  | True | 192.168.0.166  |  255.255.0.0  |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth2:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True | 10.247.246.174 | 255.255.252.0 |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth3:  | True | 172.16.30.226  | 255.255.255.0 |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth3:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
ci-info: | Route | Destination  |   Gateway   |    Genmask    | Interface | Flags |
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
ci-info: |   0   |   0.0.0.0    | 192.168.0.1 |    0.0.0.0    |    eth2   |   UG  |
ci-info: |   1   | 10.247.244.0 |   0.0.0.0   | 255.255.252.0 |    eth0   |   U   |
ci-info: |   2   | 10.250.244.0 |   0.0.0.0   | 255.255.252.0 |    eth1   |   U   |
ci-info: |   3   | 172.16.30.0  |   0.0.0.0   | 255.255.255.0 |    eth3   |   U   |
ci-info: |   4   | 192.168.0.0  |   0.0.0.0   |  255.255.0.0  |    eth2   |   U   |
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
Cloud-init v. 0.7.9 running 'modules:config' at Thu, 25 Jan 2018 07:48:50 +0000. Up 19.03 seconds.
~~~

result not working:
~~~
Cloud-init v. 0.7.9 running 'init-local' at Thu, 25 Jan 2018 11:50:02 +0000. Up 7.98 seconds.
Cloud-init v. 0.7.9 running 'init' at Thu, 25 Jan 2018 11:50:10 +0000. Up 15.71 seconds.
2018-01-25 06:50:10,348 - util.py[WARNING]: Route info failed: Unexpected error while running command.
Command: ['netstat', '-rn']
Exit code: 1
Reason: -
Stdout: Kernel IP routing table
        Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
Stderr: -
ci-info: +++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: | Device |  Up  |  Address  |    Mask   | Scope |     Hw-Address    |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: |  lo:   | True | 127.0.0.1 | 255.0.0.0 |   .   |         .         |
ci-info: |  lo:   | True |     .     |     .     |   d   |         .         |
ci-info: | eth1:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth2:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True |     .     |     .     |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth3:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Cloud-init v. 0.7.9 running 'modules:config' at Thu, 25 Jan 2018 11:50:12 +0000. Up 17.90 seconds.
~~~

Comment 4 Andreas Karis 2018-01-26 16:21:13 UTC
Tested with cloud-init-0.7.9-20.el7.x86_64

Comment 5 Ryan McCabe 2018-01-26 18:30:53 UTC
I don't believe this is intended to work. Did defining the interface twice previously work for the customer?

The current upstream release explicitly errors out when you do this with a parse error.

Comment 8 Andreas Karis 2018-01-26 19:38:14 UTC
Hi,

cloud-init upstrea in cloudinit/net/eni.py
https://github.com/number5/cloud-init/blob/master/cloudinit/net/eni.py#L193
~~~
 elif option == "iface":
            iface, family, method = split[1:4]
            if iface not in ifaces:
                ifaces[iface] = {
                    # Include the source path this interface was found in.
                    "_source_path": src_path
                }
            elif 'family' in ifaces[iface]:
                raise ParserError(
                    "Interface %s can only be defined once. "
                    "Re-defined in '%s'." % (iface, src_path))
            ifaces[iface]['family'] = family
            ifaces[iface]['method'] = method
            currif = iface
~~~

a) iface eth2 inet static
b) iface eth2 inet6 static

a)
iface = eth2
family = inet
method = static

b)
iface = eth2
family = inet6
method = static

inet will define ifaces['eth2']['family'] as 'inet' and then on the next pass with inet6, `elif 'family' in ifaces[iface]` will evaluate to true and this throw an error.

The actual error message doesn't seem to figure in any of the collected customer data, but from the above, it seems pretty clear that the ENI format accepts only one interface and not different tuples of iface/family.

Comment 9 Andreas Karis 2018-01-26 19:44:20 UTC
I'm bringing this upstream ...

http://cloudinit.readthedocs.io/en/latest/topics/network-config-format-eni.html#network-config-eni

states that:
~~~
Please reference existing documentation for the /etc/network/interfaces(5) format.
~~~

From https://manpages.debian.org/jessie/ifupdown/interfaces.5.en.html   - the man page doesn't seem to say anything else about duplicate iface declarations with different families:
~~~
Stanzas defining logical interfaces start with a line consisting of the word "iface" followed by the name of the logical interface. In simple configurations without mapping stanzas this name should simply be the name of the physical interface to which it is to be applied. (The default mapping script is, in effect, the echo command.) The interface name is followed by the name of the address family that the interface uses. This will be "inet" for TCP/IP networking, but there is also some support for IPX networking ("ipx"), and IPv6 networking ("inet6"). Following that is the name of the method used to configure the interface. 
(...)
~~~

According to https://wiki.debian.org/NetworkConfiguration , this should work, though:
~~~
If you're configuring it manually then something like this will set the default gateway (network, broadcast and gateway are optional):

    auto eth0
    iface eth0 inet static
        address 192.0.2.7
        netmask 255.255.255.0
        gateway 192.0.2.254

If you want to add an IPv6 address, too, append something like:

    iface eth0 inet6 static
        address 2001:db8::c0ca:1eaf
        netmask 64
        gateway 2001:db8::1ead:ed:beef
~~~