Bug 1622550

Can you please attach the host log collection?

QE cannot reproduced this bug, using the below two ovirt-node version. The hosted engine wizard is loaded successfully according to the steps in comment #0. 

Test Version
1. ovirt-node-ng-4.2.5.1-0.20180731.0
2. ovirt-node-ng-4.2.4-0.20180626.0

1. ovirt-node 4.2.5
[root@ibm-x3650m5-04 ~]# rpm -qa|grep cockpit
cockpit-169-1.el7.centos.x86_64
cockpit-bridge-169-1.el7.centos.x86_64
cockpit-storaged-169-1.el7.centos.noarch
cockpit-networkmanager-169-1.el7.centos.noarch
cockpit-system-169-1.el7.centos.noarch
cockpit-ovirt-dashboard-0.11.31-1.el7.noarch
cockpit-ws-169-1.el7.centos.x86_64
cockpit-dashboard-169-1.el7.centos.x86_64
cockpit-machines-ovirt-169-1.el7.centos.noarch

2. ovirt-node 4.2.4
[root@ibm-x3650m5-04 ~]# rpm -qa|grep cockpit
cockpit-ovirt-dashboard-0.11.28-1.el7.noarch
cockpit-169-1.el7.x86_64
cockpit-storaged-169-1.el7.noarch
cockpit-bridge-169-1.el7.x86_64
cockpit-system-169-1.el7.noarch
cockpit-networkmanager-169-1.el7.noarch
cockpit-ws-169-1.el7.x86_64
cockpit-dashboard-169-1.el7.x86_64
cockpit-machines-ovirt-169-1.el7.noarch

agmeshs,
Could you please give the detail steps to reproduced it.

Created attachment 1480727 [details]
Ovirt setup log

Here is the attach log (i hope it's the good one) of the failed setup of ovirt hosted engine with cockpit

Dear Wei Wang,

i will try to give you a few more detail to reproduced it :

this is a standard node installation with usb key of ovirt 4.2.4 (now upgraded in 4.2.5 with yum)
the server is a Proliant DL 380 Gen9 (Intel(R) Xeon(R) CPU E5-2620  @2.10GHz)

the only operation i did after the node installation and before the engine installation is to create 2 bonds with 4 ethernets links (2x2 802.3ad) one with a lan address, the other one with a public ip address

If you need any more information or test, please tell me (i'm available on ovirt irc chan if needed too)

regards,
Slx

(In reply to Slx from comment #3)
> Created attachment 1480727 [details]
> Ovirt setup log
> 
> Here is the attach log (i hope it's the good one) of the failed setup of
> ovirt hosted engine with cockpit

Everything there seams fine.
Can you please reproduce (try reloading the page) keeping your browser developer tools (ctrl-shift-i) open on the console tab and attach the result here?

(if needed, i can do it with an english browser)

Here are the log :
13:51:04,540 ReferenceError: reference to undefined property "disabled"[En savoir plus] jquery.js:5227:32
13:51:04,545 ReferenceError: reference to undefined property "fromElement"[En savoir plus] app.js:11:53035
13:51:04,545 ReferenceError: reference to undefined property "_dispatchListeners"[En savoir plus] app.js:11:36709
13:51:04,545 ReferenceError: reference to undefined property ".8"[En savoir plus] app.js:11:35841
13:51:04,822 Failed to read file /var/lib/ovirt-hosted-engine-setup/answers/he-answer.conf. Check that the file exists and is not empty app.js:11:60774
13:51:04,886 ReferenceError: reference to undefined property "deploymentTypeOption"[En savoir plus] app.js:25:25099
13:51:04,935 [Afficher/Masquer les détails du message.] GET
https://x.x.x.x:9090/cockpit/$513f545ad673eaaef8563181606847f5187dd5305b7fd37074434b3f22fa277c/ovirt-dashboard/3aef1304bdd5e379ba9299cdf43bfdda.png
[HTTP/1.1 200 OK 233ms]
13:51:15,444 General system data retrieval started. app.js:25:315951
13:51:15,445 Get network interfaces started. app.js:25:41824
13:51:15,449 No gdeploy answer files found. app.js:25:151281
13:51:15,450 ReferenceError: reference to undefined property "backdrop"[En savoir plus] jquery.js:4266:9
13:51:22,604 Get network interfaces completed successfully. app.js:25:42560
13:51:22,608 Network interfaces retrieval failed app.js:25:317001
13:51:22,987 General system data retrieved successfully.

Hi,
can you please share the output of:
  ansible-playbook -vvv -i localhost, /usr/share/ovirt-hosted-engine-setup/ansible/get_network_interfaces.yml

directly executed on your host?

Created attachment 1480786 [details]
result of ansible-playbook -vvv -i localhost, /usr/share/ovirt-hosted-engine-setup/ansible/get_network_interfaces.yml

(In reply to Slx from comment #4)
> Dear Wei Wang,
> 
> i will try to give you a few more detail to reproduced it :
> 
> this is a standard node installation with usb key of ovirt 4.2.4 (now
> upgraded in 4.2.5 with yum)
> the server is a Proliant DL 380 Gen9 (Intel(R) Xeon(R) CPU E5-2620  @2.10GHz)
> 
> the only operation i did after the node installation and before the engine
> installation is to create 2 bonds with 4 ethernets links (2x2 802.3ad) one
> with a lan address, the other one with a public ip address
> 
> If you need any more information or test, please tell me (i'm available on
> ovirt irc chan if needed too)
> 
> regards,
> Slx

Dear Slx,
There is not the same environment from QE, so could you please help to verify this issue after it's status change to ON_QA? Thanks!

The issue comes from here:
    "ansible_facts": {
        "otopi_host_net": []
    }, 

No valid interfaces are available on that host.

I see that you have two bonds named ippub0 and ippriv0 but they are not usable since we support only bond named as bond0, bond1.

For QE, reprodution step:
1. configure a bond on the host with an unsupported name like ippub0. Be sure that no other interface is active.
2. try to deploy hosted-engine from cockpit

(In reply to Simone Tiraboschi from comment #10)
> The issue comes from here:
>     "ansible_facts": {
>         "otopi_host_net": []
>     }, 
> 
> No valid interfaces are available on that host.
> 
> I see that you have two bonds named ippub0 and ippriv0 but they are not
> usable since we support only bond named as bond0, bond1.
> 
> For QE, reprodution step:
> 1. configure a bond on the host with an unsupported name like ippub0. Be
> sure that no other interface is active.
> 2. try to deploy hosted-engine from cockpit

Test with the above steps, QE can reproduce this bug. Change flag testing_ack to '+'.

On my side, changing the bond's name into bond0 & bond1 has indeed make the cockpit go until the first step for Deployment. So it seems the problem was indeed here, sorry about the renaming thing (to be honest, it's hard to know what we can't do or not without breaking something ;).
I'll fully deploy the engine to verify there's no other problem.

If i may, i would suggest some warning into the node UI about that (and about the bond type since it seems only the four first modes are supported for ovirtmngt) if the user step outside of what is acceptable

i'm available if you need more test.

thanks again, and regards,
Slx

Another thing maybe worth mentionning, even if i guess it's not the good place to say it.
During deployment, i have been deconnected before the end (probably after step 3, but before any storage have been setup). So, after log back in, i try to redeploy the engine. But the only bond who was available was the bond who hasn't been used during the previous installation.

I guess the bond unavailable has been already been locked up. 

Since the network part is in a bad shape on this machine, i will have no other choice than to wipe it to reinstall a full node again.

I'm closing with insufficient data for reproducing the issue. If you are able to reproduce please provide steps for reproducing.

Need to add a check on bond naming:
> 1. configure a bond on the host with an unsupported name like ippub0. Be
> sure that no other interface is active.

Let's check documentation is updated about this and print a warning if we find bond with unsupported names.

(In reply to Sandro Bonazzola from comment #15)
> Need to add a check on bond naming:
> > 1. configure a bond on the host with an unsupported name like ippub0. Be
> > sure that no other interface is active.
> 
> Let's check documentation is updated about this and print a warning if we
> find bond with unsupported names.

Check RHVH 4.2 docs, there is a naming conventions for bond interfaces.
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html-single/planning_and_prerequisites_guide/#networking-recommendations

Bond interfaces: bondnumber (for example, bond0, bond1) 

one question:
Need we still print a warning if we find bond with unsupported names for the mistaken of naming conventions?

(In reply to Wei Wang from comment #16)
> (In reply to Sandro Bonazzola from comment #15)
> > Need to add a check on bond naming:
> > > 1. configure a bond on the host with an unsupported name like ippub0. Be
> > > sure that no other interface is active.
> > 
> > Let's check documentation is updated about this and print a warning if we
> > find bond with unsupported names.
> 
> Check RHVH 4.2 docs, there is a naming conventions for bond interfaces.
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/
> html-single/planning_and_prerequisites_guide/#networking-recommendations
> 
> Bond interfaces: bondnumber (for example, bond0, bond1) 
> 
> one question:
> Need we still print a warning if we find bond with unsupported names for the
> mistaken of naming conventions?

From my perspective, it's better to have a warning on cockpit UI if the bond name is not correct.

What's more, can you try to reproduce comment #13? If it's reproducible, I think maybe it's better to file another bug to track it.

(In reply to Qin Yuan from comment #17)
> (In reply to Wei Wang from comment #16)
> > (In reply to Sandro Bonazzola from comment #15)
> > > Need to add a check on bond naming:
> > > > 1. configure a bond on the host with an unsupported name like ippub0. Be
> > > > sure that no other interface is active.
> > > 
> > > Let's check documentation is updated about this and print a warning if we
> > > find bond with unsupported names.
> > 
> > Check RHVH 4.2 docs, there is a naming conventions for bond interfaces.
> > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/
> > html-single/planning_and_prerequisites_guide/#networking-recommendations
> > 
> > Bond interfaces: bondnumber (for example, bond0, bond1) 
> > 
> > one question:
> > Need we still print a warning if we find bond with unsupported names for the
> > mistaken of naming conventions?
> 
> From my perspective, it's better to have a warning on cockpit UI if the bond
> name is not correct.
Yes, I agree with you. It's better to warn end user on cockpit UI.
> 
> What's more, can you try to reproduce comment #13? If it's reproducible, I
> think maybe it's better to file another bug to track it.
Ok, I will reproduce it latter. If it is reproducible, I will file another bug to track it.

(In reply to Wei Wang from comment #18)
> (In reply to Qin Yuan from comment #17)
> > (In reply to Wei Wang from comment #16)
> > > (In reply to Sandro Bonazzola from comment #15)
> > > > Need to add a check on bond naming:
> > > > > 1. configure a bond on the host with an unsupported name like ippub0. Be
> > > > > sure that no other interface is active.
> > > > 
> > > > Let's check documentation is updated about this and print a warning if we
> > > > find bond with unsupported names.
> > > 
> > > Check RHVH 4.2 docs, there is a naming conventions for bond interfaces.
> > > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/
> > > html-single/planning_and_prerequisites_guide/#networking-recommendations
> > > 
> > > Bond interfaces: bondnumber (for example, bond0, bond1) 
> > > 
> > > one question:
> > > Need we still print a warning if we find bond with unsupported names for the
> > > mistaken of naming conventions?
> > 
> > From my perspective, it's better to have a warning on cockpit UI if the bond
> > name is not correct.
> Yes, I agree with you. It's better to warn end user on cockpit UI.
> > 
> > What's more, can you try to reproduce comment #13? If it's reproducible, I
> > think maybe it's better to file another bug to track it.
> Ok, I will reproduce it latter. If it is reproducible, I will file another
> bug to track it.
Retest according to comment #13, but I cannot reproduce this issue, the same bond can be available during the re-deployment.
My Steps:
1. Install host with rhvh-4.2.8.0-0.20181212.0
2. Configure two bonds, bond1 is public IP, bond2 is private Lan IP
3. Using bond1 to deploy hosted-engine to the steps before storage setting
4. Close the session, and redeploy it

Result:
The bond1 is still available to redeploy hosted-engine, and it is the default one.

Slx, can you please re-test with 4.2.8 RC or 4.3.0 RC to see if it reproduces for you?

Any idea why we do not allow any bond name? IMO it's enough that we have no conflicts (e.g. no bond called 'ovirtmgmt' etc.). As an admin, I also liked (re)naming interfaces, and policies such as "bondNUMBER" are not easy to follow:

1. You might have conflicts with some other policy (internal or otherwise)

2. You have to very carefully read the docs (this will be fixed if we issue a warning, but adding the code to add the warning might take us more time than removing the restriction :-) )

(In reply to Sandro Bonazzola from comment #20)
> Slx, can you please re-test with 4.2.8 RC or 4.3.0 RC to see if it
> reproduces for you?

i don't have atm a test environment available, i hope to get one back in one or two months,
sorry. I'll test that asap.

re-targeting to 4.3.1 since this BZ has not been proposed as blocker for 4.3.0.
If you think this bug should block 4.3.0 please re-target and set blocker flag.

Moving to 4.3.2 not being identified as blocker for 4.3.1.

Did the host have any interface at all?

interfaces was up but skipped because of existing bonds (with bad names)

Created attachment 1546996 [details]
Cockpit based failure on Ovirt 4.3

Following on to @Yedidyah's comments I ran into this as well. In my environment I'm not using bonding but instead using teaming with two teamed interfaces:

# private teamed interface
priv-team
  - enp6s0
  - enp7s0

pub-team
  - enp8s0
  - enp9s0


It seems the structure of the playbooks has changed and this is now failing in the "pre_checks".  I'd like to also point out that there is a discrepancy in the documentation linked above.  Based on the structuring of the playbooks these aren't "recommendations" but "requirements".  If I don't conform to your naming scheme, the install fails.  I see where more variables have been added to support this (after doing acrobatics just to see what cockpit is doing under the hood for the install):

TASK [ovirt.hosted_engine_setup : Detecting interface on existing management bridge] ****************************************************************************************
skipping: [localhost] => (item=ovirtmgmt) 
skipping: [localhost] => (item=rhevm) 

TASK [ovirt.hosted_engine_setup : debug] ************************************************************************************************************************************
ok: [localhost] => {
    "bridge_interface": "VARIABLE IS NOT DEFINED!"
}


The challenge is that the cockpit based interface isn't providing me any way of configuring those or useful telemetry as to what's failing or what to check (screenshot attached).

Created attachment 1551073 [details]
warning message

Add a message when network interfaces aren't valid

Test Version
RHVH-4.3-20190404.1-RHVH-x86_64-dvd1.iso
cockpit-system-176-4.el7.noarch
cockpit-ws-176-4.el7.x86_64
cockpit-bridge-176-4.el7.x86_64
cockpit-storaged-176-4.el7.noarch
cockpit-ovirt-dashboard-0.12.7-1.el7ev.noarch
cockpit-machines-ovirt-176-4.el7.noarch
cockpit-dashboard-176-4.el7.x86_64
cockpit-176-4.el7.x86_64
ovirt-hosted-engine-ha-2.3.1-1.el7ev.noarch
ovirt-hosted-engine-setup-2.3.7-1.el7ev.noarch
rhvm-appliance-4.3-20190404.1.el7.x86_64

Test Steps:
According to comment 10 and comment 28

Result:
Warning message displays when network interfaces aren't valid

Bug is fixed, move it to "VERIFIED"

This bugzilla is included in oVirt 4.3.3 release, published on April 16th 2019.

Since the problem described in this bug report should be
resolved in oVirt 4.3.3 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.