Hi Rik,

Please provide us more information and logs, vdsm and engine logs.
The described scenario is working as expected. 
Also the vdsm and engine versions. 

Thanks,

Created attachment 1404883 [details]
vdsm log on the host during attach of network

Created attachment 1404884 [details]
engine log during create and attach of network

Hi,

On the engine:
ovirt-engine-4.2.1.7-1.el7.centos.noarch

On the host:
vdsm-4.20.17-1.el7.centos.x86_64

I've attached a tail of the vdsm and engine log.

In the engine log it shows that I've added the 'ea-psi' network and that it fails to add the network to the host as the regex for the ip address and netmask does not match. There doesn't seem to be any relevant log on the host during this action (the engine does not contact the host but fails the validation before this step).


Regards,

Rik

(In reply to Rik Theys from comment #3)
> Created attachment 1404884 [details]
> engine log during create and attach of network
> 
> Hi,
> 
> On the engine:
> ovirt-engine-4.2.1.7-1.el7.centos.noarch
> 
> On the host:
> vdsm-4.20.17-1.el7.centos.x86_64
> 
> I've attached a tail of the vdsm and engine log.
> 
> In the engine log it shows that I've added the 'ea-psi' network and that it
> fails to add the network to the host as the regex for the ip address and
> netmask does not match. There doesn't seem to be any relevant log on the
> host during this action (the engine does not contact the host but fails the
> validation before this step).
> 
> 
> Regards,
> 
> Rik

Thank you, we will take a look

This is sounds like BZ 1528906 (same error), at least very similar.
Although it was verified on 4.2.1.1-0.1.el7 and you using ovirt-engine-4.2.1.7-1.el7.centos.noarch 

Alona would you agree this is the same issue?

Rik, 
Are you sure that the network attachment has no IP address nor netmask/prefix? Are you sure that the network has none of the IPconfigurations(prefix,IP, gateway) when trying to attach it to the host?

I can't reproduce this issue. Can you give me a clear steps to reproduce it? is it 100% reproducible for you?

Created attachment 1405171 [details]
screenshots

Hi,

Yes it is 100% reproducible on our installation.

I add a new logical network to our datacenter and the only change I make in the dialog is to enable vlan tagging and enter vlan 724.

I then go to one of the hosts in that datacenter and use the setup host network button. I drag the network from the unassigned list and drop in on the bond (where I have multiple other networks that were already there).

I don't make any changes to the settings of the network. As you can see from the screenshots, the IP configuration is set to None for ipv4 and ipv6.

When I click on OK, the message appears.

Regards,

Rik

Alona, see need info requested in comment 5#^^

Hi Rik,

Does it happen only on host 'amazone'?
Does it work for a vlan id other than '724'?
Are you able to attach the network 'ea-psi' to another nic?

(In reply to Michael Burman from comment #8)
> Alona, see need info requested in comment 5#^^

Hi Michael,

The bug here has two errors, gateway and address.
BZ 1528906 may fix the gateway error, it won't fix the address one.

(In reply to Alona Kaplan from comment #9)
> Does it happen only on host 'amazone'?
No, it also happens for the other host in that cluster.

> Does it work for a vlan id other than '724'?
No.

> Are you able to attach the network 'ea-psi' to another nic?

I tried to attach it to a physical nic that was not in use but this throws the same error.

But! I've configured the network in another datacenter that's managed by the same engine and there it works!?

When I look in the engine database, both networks look identical:

engine=# select * from network where name='ea-psi';
                  id                  |  name  |  description  | type | addr | subnet | gateway | vlan_id | stp |           storage_pool_id            | mtu | vm_network | provider_network_provider_id | provider_
network_external_id | free_text_comment | label | qos_id | vdsm_name | dns_resolver_configuration_id | provider_physical_network_id
--------------------------------------+--------+---------------+------+------+--------+---------+---------+-----+--------------------------------------+-----+------------+------------------------------+----------
--------------------+-------------------+-------+--------+-----------+-------------------------------+------------------------------
 c155cb16-3c89-419a-adca-a2a18fd82f0c | ea-psi | PSI A network |      |      |        |         |     724 | f   | 4c47403d-2815-484c-abc1-c279ff7a25b5 |   0 | t          |                              |
                    |                   |       |        | ea-psi    |                               |
 ae10e5a8-19ed-4d42-b962-6f4d2d374227 | ea-psi |               |      |      |        |         |     724 | f   | e7bdba88-e718-41a9-8d2b-0ca79c517630 |   0 | t          |                              |
                    |                   |       |        | ea-psi    |                               |
(2 rows)

the one with id that starts with c155 is the broken one.

engine=# select * from network_cluster where network_id='c155cb16-3c89-419a-adca-a2a18fd82f0c' or network_id='ae10e5a8-19ed-4d42-b962-6f4d2d374227';
              network_id              |              cluster_id              | status | is_display | required | migration | management | is_gluster | default_route
--------------------------------------+--------------------------------------+--------+------------+----------+-----------+------------+------------+---------------
 c155cb16-3c89-419a-adca-a2a18fd82f0c | 38ce6d46-5097-4d6e-9355-6903d84656ff |      0 | f          | t        | f         | f          | f          | f
 ae10e5a8-19ed-4d42-b962-6f4d2d374227 | 96b07d57-24cf-44dc-b067-4580fe3013c1 |      1 | f          | f        | f         | f          | f          | f
(2 rows)

engine=# select * from network_attachments where network_id='c155cb16-3c89-419a-adca-a2a18fd82f0c' or network_id='ae10e5a8-19ed-4d42-b962-6f4d2d374227';
                  id                  |              network_id              |                nic_id                | boot_protocol | address | netmask | gateway | custom_properties |         _create_date
  | _update_date | ipv6_boot_protocol | ipv6_address | ipv6_prefix | ipv6_gateway | dns_resolver_configuration_id
--------------------------------------+--------------------------------------+--------------------------------------+---------------+---------+---------+---------+-------------------+-----------------------------
--+--------------+--------------------+--------------+-------------+--------------+-------------------------------
 64321247-8d34-4804-ab9e-fe004c91d6d9 | ae10e5a8-19ed-4d42-b962-6f4d2d374227 | a12a5f16-2711-436c-9846-310def7648ad | NONE          |         |         |         |                   | 2018-03-08 12:48:49.751938+0
1 |              | NONE               |              |             |              |
(1 row)

How can I determine the differences between these two?

The bad datacenter is compat version 4.2, with one cluster in 4.2 level and two hosts (haswell-noTSX IBRS family).
The good datacenter is compat version 4.2, with one cluster in 4.2 level and one host (sandybridge family)

Regards,

Rik

Hi,

I think I know why it fails.

I get the same error when I click OK on the "setup host networks" with no changes. The reason is that this cluster has a dedicated migrate network where no gateway is configured as it is an isolated network.

So I get the error because the migrate network has not gateway (which should be an OK setup???)

Regards,

Rik

Hi Rik,

Thanks for the information.

Just to make sure I understand correctly -

1. The problematic cluster has more than one host. Opening the Setup Networks dialog, doing nothing and just clicking ok causes the error on all the cluster hosts. Right? (Can you please attach an engine log?) What about hosts in the same dc but different cluster?

2. Does performing Setup Networks on the problematic cluster/host via the rest api works fine?
 
3. Setup Networks on different data center works fine. Right?
 
4. The problematic cluster has a migration network which is configured with no gateway on all the cluster hosts. Right?
Does changing the migration network to another network (ovirtmgmt for example) solves the issue? (I couldn't reproduce the issue having on the host migration network with no gateway).

5. Please provide the output of -  
- select * from network_attachments where gateway = '' or address = '';
- select * from vds_interface where (addr = '' or gateway = '') and boot_protocol != 0;
- If possible, dump of the db.

I can't reproduce the issue a well :(

(I couldn't reproduce the issue having on the host migration network with no gateway).

(In reply to Alona Kaplan from comment #13)
> Just to make sure I understand correctly -
> 
> 1. The problematic cluster has more than one host. Opening the Setup
> Networks dialog, doing nothing and just clicking ok causes the error on all
> the cluster hosts. Right? (Can you please attach an engine log?) What about
> hosts in the same dc but different cluster?

Yes, it happens on all hosts in that cluster. The datacenter only has one cluster, so I can not test it with another cluster. I will attach a tail of the engine log while doing this.

> 2. Does performing Setup Networks on the problematic cluster/host via the
> rest api works fine?

Do you have any examples on how to do this?

>  
> 3. Setup Networks on different data center works fine. Right?

Yes. I have verified on another datacenter on the same engine that also has a migration network without gateway and there it works :-/.

>  
> 4. The problematic cluster has a migration network which is configured with
> no gateway on all the cluster hosts. Right?
Yes

> Does changing the migration network to another network (ovirtmgmt for
> example) solves the issue? (I couldn't reproduce the issue having on the
> host migration network with no gateway).
> 
> 5. Please provide the output of -  
> - select * from network_attachments where gateway = '' or address = '';
> - select * from vds_interface where (addr = '' or gateway = '') and
> boot_protocol != 0;
> - If possible, dump of the db.

I will attach the output from these commands.

Regards,

Rik

Created attachment 1407103 [details]
engine log during setup network

Created attachment 1407104 [details]
sql output of requested commands

Given that the issue does not happen in the other data center, I'm no longer convinced that the migrate network alone is the culprit.

Created attachment 1407159 [details]
engine log from engine startup to trigger of issue

This is the engine log from an engine start to when I've triggered the issue.

Hi,

The only differences I can find between datacenters that work and that don't are:

* The one that doesn't work also has two additional network cards that are not used and are down. They are not configured.

The ifcfg file is:

TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=em4
UUID=befec9ff-60be-4943-846f-125057b1b593
DEVICE=em4
ONBOOT=no

It does not show the 'created by vdsm' line in this file.

According to the supervdsm log during startup, the interface has the following configuration:

'em3': {'ipv6autoconf': False, 'addr': '', 'speed': 0, 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': 'ec:f4:bb:d6:5f:0c', 'ipv6gateway': '::', 'gateway': ''},

'em4': {'ipv6autoconf': False, 'addr': '', 'speed': 0, 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': 'ec:f4:bb:d6:5f:0d', 'ipv6gateway': '::', 'gateway': ''},

So according to this output, dhcp is disabled (but the ifcfg file says it is enabled). I can not configure these interfaces in the web interface (no pencil), see screen shot I will attach in a minute.

I don't know if this is relevant. Maybe the validator is also checking these values?

* When I compare the network_attachments table for the migration network of the working vs non-working datacenter, the working one has '{ }' in the custom_properties field, and the non-working one has nothing in this field. I don't see any other differences.

Regards,

Rik

Created attachment 1407174 [details]
screenshot that shows two extra (not used) interfaces

Eureka!

I was able to find the root cause of the issue. The issue was not that the address/netmask/gateway was incorrectly/not set, but that for older network_attachments the address/netmask/gateway are set to '' (empty string) in the database and for newer networks to NULL (for networks that have boot_protocol NONE).

When you use psql to list it there's no visual difference, but when I pulled the data using a python script I was able to tell that all my older networks had '' instead of NULL.

After manually updating the database (setting all those fields to NULL) and restarting the engine, I can save the network changes again.

I assume that during some previous upgrade oVirt started using NULL values instead of ''. The cluster on which I experienced the issue is several years old and many oVirt releases old (I believe it started at 3.3 or something).

So the fix would be to update the fields during an engine update.

It would be awesome if this could get fixed in an ovirt update. It may save others a lot of time.

Regards,

Rik

Thanks! We will add an upgrade script replace the empty string with null.

Putting empty string in the db in network_attachments#address field of one of the networks and then following the steps of the reporter should reproduce this.

Reporter is outside Redhat so I doubt they can verify...

Thanks, 

Verified on - 4.2.3.2-0.1.el7

This bugzilla is included in oVirt 4.2.3 release, published on May 4th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.3 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.