Bug 1560684

Created attachment 1413291 [details]
vm engine log

Created attachment 1413292 [details]
engine server log

attachment 1413291 [details] and attachment 1413292 [details] seem to contain the same log. Would you attach engine.log from within the VM?

Could it be that you specify a network name with a space or another special character?

Created attachment 1413772 [details]
vm engine log

I've put the correct vm server log up.

I do not specify anything other than what interface to use, hosted-engine --deploy does that..

Also, rc4 worked, rc5 and newer is where this popped up.

Is it the engine, or is it hosted-engine --deploy doing the wrong thing?

There is no network label being passed in via host-deploy/ansible; from

/usr/share/ovirt-hosted-engine-setup/ansible/bootstrap_local_vm.yml


      - name: Set VLAN ID at datacenter level
        ovirt_networks:
          data_center: "{{ DATA_CENTER }}"
          name: "{{ MGMT_NETWORK }}"
          vlan_tag: "{{ vlan_id_out.stdout }}"
          auth: "{{ ovirt_auth }}"
        when: vlan_id_out.stdout|length > 0

is what it is trying to do.  Network Label is not supported till ansible 2.5, the hosted-engine deploy uses:

[root@d8-r13-c0-n1 library]# yum list *ansible*
Loaded plugins: fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
 * base: mirrors.usc.edu
 * extras: dallas.tx.mirror.xygenhosting.com
 * ovirt-4.2: mirrors.ibiblio.org
 * ovirt-4.2-epel: mirror.prgmr.com
 * updates: mirror.texas3006.com
Installed Packages
ansible.noarch                                                             2.4.2.0-2.el7                                          @extras                  
Available Packages
ansible-doc.noarch                                                         2.4.2.0-2.el7                                          extras                   
ovirt-ansible-cluster-upgrade.noarch                                       1.1.6-1.el7.centos                                     ovirt-4.2-pre            
ovirt-ansible-disaster-recovery.noarch                                     0.3-1.el7.centos                                       ovirt-4.2-pre            
ovirt-ansible-engine-setup.noarch                                          1.1.0-1.el7.centos                                     ovirt-4.2                
ovirt-ansible-image-template.noarch                                        1.1.5-1.el7.centos                                     ovirt-4.2                
ovirt-ansible-infra.noarch                                                 1.1.4-1.el7.centos                                     ovirt-4.2-pre            
ovirt-ansible-manageiq.noarch                                              1.1.6-1.el7.centos                                     ovirt-4.2-pre            
ovirt-ansible-repositories.noarch                                          1.1.0-1.el7.centos                                     ovirt-4.2                
ovirt-ansible-roles.noarch                                                 1.1.3-1.el7.centos                                     ovirt-4.2                
ovirt-ansible-vm-infra.noarch                                              1.1.5-1.el7.centos                                     ovirt-4.2-pre            
skydive-ansible.x86_64                                                     0.16.0-1.el7                                           ovirt-4.2-centos-opstools

so I'm wondering if the engine is complaining about a 'null' or non-existent label?

The bug was caused by https://gerrit.ovirt.org/#/c/88620/4/packaging/dbscripts/upgrade/04_02_0960_default_all_search_engine_fields_to_empty_string.sql

Null and empty labels are treated differently in the code, changing all the nulls to empty strings breaks the functionality.

(In reply to Alona Kaplan from comment #8)
> The bug was caused by
> https://gerrit.ovirt.org/#/c/88620/4/packaging/dbscripts/upgrade/
> 04_02_0960_default_all_search_engine_fields_to_empty_string.sql
> 
> Null and empty labels are treated differently in the code, changing all the
> nulls to empty strings breaks the functionality.

Also, the labels that were changed to be empty should be reverted to be null.

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Network update:
- Empty label has gone from the networks in the UI drop down list
- All networks that had the empty label, after upgrade has no label as expected('-' sign in the Label column)
- The empty label gone from the setup networks dialogue > labels section
- No regression has caused to network functionality

SHE on these components:
ovirt-hosted-engine-ha-2.2.9-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.15-1.el7ev.noarch
rhvm-appliance-4.2-20180322.0.el7.noarch.rpm
Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

I've tried to deploy over regular untagged network interface and deployment was successful.


Over tagged VLAN it fails, with this error:
[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Network label must be formed only from: English letters, numbers, hyphen or underscore., Attribute: network.label]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Network label must be formed only from: English letters, numbers, hyphen or underscore., Attribute: network.label]\". HTTP response code is 400."}
[ INFO  ] TASK [include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Remove local vm dir]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
[ INFO  ] TASK [Gathering Facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Remove local vm dir]
[ INFO  ] ok: [localhost]
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180401145330.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180401144424-be9gbo.log

Moving back to assigned.

Created attachment 1415792 [details]
engine logs

Created attachment 1415793 [details]
sosreport from puma19

sosreport-nsednev-he-7.scl.lab.tlv.redhat.com-20180401150029/sos_commands/rpm/sh_-c_rpm_--nodigest_-qa_--qf_NAME_-_VERSION_-_RELEASE_._ARCH_INSTALLTIME_date_awk_-F_printf_-59s_s_n_1_2_sort_-f

ovirt-engine-4.2.2.5-0.1.el7.noarch                         Thu Mar 22 22:24:10 2018

Please retest with 4.2.2.6 which includes the fix of http://gerrit.ovirt.org/89555

(In reply to Nikolai Sednev from comment #13)
> SHE on these components:
> ovirt-hosted-engine-ha-2.2.9-1.el7ev.noarch
> ovirt-hosted-engine-setup-2.2.15-1.el7ev.noarch
> rhvm-appliance-4.2-20180322.0.el7.noarch.rpm
> Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64
> x86_64 x86_64 GNU/Linux
> Red Hat Enterprise Linux Server release 7.5 (Maipo)
> 
> I've tried to deploy over regular untagged network interface and deployment
> was successful.
> 
> 
> Over tagged VLAN it fails, with this error:
> [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is
> "[Network label must be formed only from: English letters, numbers, hyphen
> or underscore., Attribute: network.label]". HTTP response code is 400.
> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault
> reason is \"Operation Failed\". Fault detail is \"[Network label must be
> formed only from: English letters, numbers, hyphen or underscore.,
> Attribute: network.label]\". HTTP response code is 400."}
> [ INFO  ] TASK [include_tasks]
> [ INFO  ] ok: [localhost]
> [ INFO  ] TASK [Remove local vm dir]
> [ INFO  ] changed: [localhost]
> [ INFO  ] TASK [Notify the user about a failure]
> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The
> system may not be provisioned according to the playbook results: please
> check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
> [ ERROR ] Failed to execute stage 'Closing up': Failed executing
> ansible-playbook
> [ INFO  ] Stage: Clean up
> [ INFO  ] Cleaning temporary resources
> [ INFO  ] TASK [Gathering Facts]
> [ INFO  ] ok: [localhost]
> [ INFO  ] TASK [include_tasks]
> [ INFO  ] ok: [localhost]
> [ INFO  ] TASK [Remove local vm dir]
> [ INFO  ] ok: [localhost]
> [ INFO  ] Generating answer file
> '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180401145330.conf'
> [ INFO  ] Stage: Pre-termination
> [ INFO  ] Stage: Termination
> [ ERROR ] Hosted Engine deployment failed: please check the logs for the
> issue, fix accordingly or re-deploy from scratch.
>           Log file is located at
> /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180401144424-
> be9gbo.log
> 
> Moving back to assigned.

Hi Nikolai, but you are not testing with the correct engine version.
You running an old appliance from last month 22.3, it doesn't have the engine version with this fix. Please re-test with the correct engine version.
Moving back to ON_QA

Moving to modified, until QA gets the appliance with the fix.

(In reply to Nikolai Sednev from comment #21)
> Moving to modified, until QA gets the appliance with the fix.

Hi Nikolai, 
We got the new appliance(rhvm-appliance-4.2-20180401.0)with rhvm 4.2.2.6-0.1.el7 which has the fix to this bug. Please re-test

(In reply to Michael Burman from comment #12)
> Network update:
> - Empty label has gone from the networks in the UI drop down list
> - All networks that had the empty label, after upgrade has no label as
> expected('-' sign in the Label column)
> - The empty label gone from the setup networks dialogue > labels section
> - No regression has caused to network functionality

Tested on - 4.2.2.6-0.1.el7

Failed on these components:
rhvm-appliance-4.2-20180401.0.el7.noarch.rpm
ovirt-hosted-engine-ha-2.2.9-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.15-1.el7ev.noarch
Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Engine inside the appliance:
ovirt-engine-setup-base-4.2.2.6-0.1.el7.noarch

Deployed over NFS.

[ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_vms": [{"affinity_labels": [], "applications": [], "bios": {"boot_menu": {"enabled": false}}, "cdroms": [], "cluster": {"href": "/ovirt-engine/api/clusters/1748a294-3676-11e8-b3bd-00163eeeeee1", "id": "1748a294-3676-11e8-b3bd-00163eeeeee1"}, "cpu": {"architecture": "x86_64", "topology": {"cores": 1, "sockets": 4, "threads": 1}}, "cpu_profile": {"href": "/ovirt-engine/api/cpuprofiles/58ca604e-01a7-003f-01de-000000000250", "id": "58ca604e-01a7-003f-01de-000000000250"}, "cpu_shares": 0, "creation_time": "2018-04-02 16:06:05.791000+03:00", "delete_protected": false, "disk_attachments": [], "display": {"address": "127.0.0.1", "allow_override": false, "copy_paste_enabled": true, "disconnect_action": "LOCK_SCREEN", "file_transfer_enabled": true, "monitors": 1, "port": 5900, "single_qxl_pci": false, "smartcard_enabled": false, "type": "vnc"}, "graphics_consoles": [], "high_availability": {"enabled": false, "priority": 0}, "host": {"href": "/ovirt-engine/api/hosts/043bc7e9-5580-48d8-8953-6d84a33ed596", "id": "043bc7e9-5580-48d8-8953-6d84a33ed596"}, "host_devices": [], "href": "/ovirt-engine/api/vms/b537b175-8545-426d-8571-d6652d56d2ef", "id": "b537b175-8545-426d-8571-d6652d56d2ef", "io": {"threads": 0}, "katello_errata": [], "large_icon": {"href": "/ovirt-engine/api/icons/e57746a0-a95b-019c-4355-27b4eac77170", "id": "e57746a0-a95b-019c-4355-27b4eac77170"}, "memory": 17179869184, "memory_policy": {"guaranteed": 17179869184, "max": 17179869184}, "migration": {"auto_converge": "inherit", "compressed": "inherit"}, "migration_downtime": -1, "name": "external-HostedEngineLocal", "next_run_configuration_exists": false, "nics": [], "numa_nodes": [], "numa_tune_mode": "interleave", "origin": "external", "original_template": {"href": "/ovirt-engine/api/templates/00000000-0000-0000-0000-000000000000", "id": "00000000-0000-0000-0000-000000000000"}, "os": {"boot": {"devices": ["hd"]}, "type": "other"}, "permissions": [], "placement_policy": {"affinity": "migratable"}, "quota": {"id": "2996557c-3676-11e8-8971-00163eeeeee1"}, "reported_devices": [], "run_once": false, "sessions": [], "small_icon": {"href": "/ovirt-engine/api/icons/5ba0b8a7-51c6-5ef5-7ed0-495a62737d13", "id": "5ba0b8a7-51c6-5ef5-7ed0-495a62737d13"}, "snapshots": [], "sso": {"methods": [{"id": "guest_agent"}]}, "start_paused": false, "stateless": false, "statistics": [], "status": "unknown", "storage_error_resume_behaviour": "auto_resume", "tags": [], "template": {"href": "/ovirt-engine/api/templates/00000000-0000-0000-0000-000000000000", "id": "00000000-0000-0000-0000-000000000000"}, "time_zone": {"name": "Etc/GMT"}, "type": "desktop", "usb": {"enabled": false}, "watchdogs": []}]}, "attempts": 24, "changed": false}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
[ INFO  ] TASK [Gathering Facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Remove local vm dir]
[ INFO  ] changed: [localhost]
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180402162515.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180402155548-4n0to5.log
You have new mail in /var/spool/mail/root

The bug now is different, I've opened a separate https://bugzilla.redhat.com/show_bug.cgi?id=1562787 here.
Now its failing because of the different error (affinity_labels).
Moving this specific bug to verified as its specific case had been fixed.

This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.