Bug 1714142

+++ This bug was initially created as a clone of Bug #1688450 +++

Description of problem:
In the previous cycle a change added to 
tempest which allows to use rhel7 image instead of very small test only image.
Unfortunately the change is not compatible with RHEL8,
first of all, the 'serivce network restart' Does not exists.

The change for RHEL7 in tempest:
https://github.com/openstack/tempest/commit/c5c7ba997e


tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet.* tests are failing.


Additional info:
RHEL guest image does not have the legacy part of the initscripts,
network-scripts was splinted out from initscripts (package build time) to a dedicated package since Jun 14 2018 9.83-1 .

Steps how to reproduce:

$ source ~/overcloudrc && openstack flavor create --id 200 --ram 1024 --disk 10 --vcpus 1 rhel
$ source ~/overcloudrc && openstack flavor create --id 201 --ram 2048 --disk 11 --vcpus 2 rhel_alt

$ discover-tempest-config --debug --create --image <path/to/rhel8> --deployer-input ../ir-tempest-deployer-input.conf compute.flavor_ref 200 compute.flavor_ref_alt 201  compute.image_ssh_user cloud-user validation.image_ssh_user cloud-user scenario.dhcp_client dhclient volume.volume_size 12

I executed one of the tests from tempest/scenario/test_network_v6.py:
$ tempest run --regex tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_dhcp6_stateless_from_os

<output omitted>
    2019-04-26 09:21:50,876 89245 ERROR    [tempest.lib.common.utils.linux.remote_client] (TestGettingAddress:test_dualnet_dhcp6_stateless_from_os) Executing command on 10.0.0.214 failed. Error: Command 'set -eu -o pipefail; PATH=$PATH:/sbin; echo -e "DEVICE=eth1\nIPV6INIT=yes" | sudo tee /etc/sysconfig/network-scripts/ifcfg-eth1; sudo /sbin/service network restart', exit status: 5, stderr:
    Redirecting to /bin/systemctl restart network.service
    Failed to restart network.service: Unit network.service not found.
    
    stdout:
    DEVICE=eth1
    IPV6INIT=yes
    
    2019-04-26 09:21:50,876 89245 DEBUG    [tempest.scenario.test_network_v6] Error during restarting eth1 interface on instance. Error message: Command 'set -eu -o pipefail; PATH=$PATH:/sbin; echo -e "DEVICE=eth1\nIPV6INIT=yes" | sudo tee /etc/sysconfig/network-scripts/ifcfg-eth1; sudo /sbin/service network restart', exit status: 5, stderr:
    Redirecting to /bin/systemctl restart network.service
    Failed to restart network.service: Unit network.service not found.
<output omitted>

So the first problem is that network can't be restarted on RHEL-8 the way as it used to be on RHEL-7 and older, https://access.redhat.com/discussions/3791081
Therefore the test should either restart the network a unified way (it works on RHEL-8 and the older versions as well) or the test should check what kind of image is used. I like more the first idea, I'm gonna have a look.

--- Additional comment from Martin Kopec on 2019-04-30 09:05:58 UTC ---

[1] in the test can be changed for 'nmcli connection reload', which will work also for RHEL8 and anyway the tests should not restart the whole network service at all.

The second problem with the tests is that RHEL8 doesn't calculate ipv6 addresses based on the mac address of the interface anymore, therefore, the tests can't assume what ipv6 address will be given to the interface (ips_from_api_* variable set by prepare_server method [2]). We need to modify the tests and instead of assuming what address will be given to the interface, the test should rather check if any ipv6 address was given to it and maybe ping it.

[1] https://github.com/openstack/tempest/blob/c248f5962ec4abfd716822c3b3ba496fe0e9cfbb/tempest/scenario/test_network_v6.py#L171
[2] https://github.com/openstack/tempest/blob/c248f5962ec4abfd716822c3b3ba496fe0e9cfbb/tempest/scenario/test_network_v6.py#L192-L193

--- Additional comment from Martin Kopec on 2019-05-02 09:08:54 UTC ---

$ openstack server create --image <rhel8> --flavor <flavor> --network <network1> --network <network2> --key-name <key_name> instance_name
+-------------------------------------+--------------------------------------------------------------------------------------------------------+
| Field                               | Value                                                                                                  |
+-------------------------------------+--------------------------------------------------------------------------------------------------------+
<output omitted>
| accessIPv4                          |                                                                                                        |
| accessIPv6                          |                                                                                                        |
| addresses                           | tempest-network-smoke--884367252=10.100.0.5; tempest-network-smoke--18828977=2003::f816:3eff:febb:7456 |
<output omitted>

The test expects that eth1 interface of the instance will have ipv6 address equal to 2003::f816:3eff:febb:7456. This address is returned by 'openstack server create' command and I guess it's calculated based on the mac address of eth1 interface which is fa:16:3e:bb:74:56, however, when I ssh to the instance, I see the following output of 'ip a' command:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:48:e8:b5 brd ff:ff:ff:ff:ff:ff
    inet 10.100.0.3/28 brd 10.100.0.15 scope global dynamic noprefixroute eth0
      valid_lft 86363sec preferred_lft 86363sec
    inet6 fe80::f816:3eff:fe48:e8b5/64 scope link
      valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:bb:74:56 brd ff:ff:ff:ff:ff:ff
    inet6 2003::b47f:f400:ecca:2a55/64 scope global dynamic noprefixroute
      valid_lft 86385sec preferred_lft 14385sec
    inet6 fe80::7615:8d57:775d:fae/64 scope link noprefixroute
      valid_lft forever preferred_lft forever

Notice that eth1 interface has an ipv6 address which seems not to be derived from its mac address.

If the calculation of ipv6 addresses has changed in rhel8, 'openstack server create' should know about it and should return the proper address.

--- Additional comment from Luigi Toscano on 2019-05-02 09:18:07 UTC ---

(In reply to Martin Kopec from comment #6)
> [1] in the test can be changed for 'nmcli connection reload', which will
> work also for RHEL8 and anyway the tests should not restart the whole
> network service at all.

But if you use systemctl restart network, shouldn't that work also on the other supported distributions and avoid a distribution-specific branch?

--- Additional comment from Martin Kopec on 2019-05-03 19:05:23 UTC ---

(In reply to Luigi Toscano from comment #8)
> (In reply to Martin Kopec from comment #6)
> > [1] in the test can be changed for 'nmcli connection reload', which will
> > work also for RHEL8 and anyway the tests should not restart the whole
> > network service at all.
> 
> But if you use systemctl restart network, shouldn't that work also on the
> other supported distributions and avoid a distribution-specific branch?

Based on the comments in this [1] commit, the distribution-specific branch was created because a config file in a certain location needs to be changed. The restart command is just a part of the process of changing the config file. The problem which this BZ describes is that RHEL8 doesn't have network service as it used to have in the previous version [2] so restarting of the network service will not work there (either via systemctl or service command).

[1] https://github.com/openstack/tempest/commit/c5c7ba997e
[2] https://access.redhat.com/discussions/3791081