Bug 1560684 - [Labels] - Empty string on label instead of null fails hosted engine setup
Summary: [Labels] - Empty string on label instead of null fails hosted engine setup
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: Database.Core
Version: ---
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: ovirt-4.2.2
: ---
Assignee: Eli Mesika
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On:
Blocks: 1561483
TreeView+ depends on / blocked
 
Reported: 2018-03-26 18:21 UTC by Thomas Davis
Modified: 2018-04-05 09:38 UTC (History)
9 users (show)

Fixed In Version: ovirt-engine-4.2.2.6
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1561483 (view as bug list)
Environment:
Last Closed: 2018-04-05 09:38:31 UTC
oVirt Team: Network
Embargoed:
rule-engine: ovirt-4.2+
rule-engine: blocker+


Attachments (Terms of Use)
hosted-engine log (268.04 KB, text/plain)
2018-03-26 18:21 UTC, Thomas Davis
no flags Details
vm engine log (27.44 KB, text/plain)
2018-03-26 18:21 UTC, Thomas Davis
no flags Details
engine server log (27.44 KB, text/plain)
2018-03-26 18:22 UTC, Thomas Davis
no flags Details
vm engine log (86.38 KB, text/plain)
2018-03-27 15:03 UTC, Thomas Davis
no flags Details
engine logs (9.31 MB, application/x-xz)
2018-04-01 12:05 UTC, Nikolai Sednev
no flags Details
sosreport from puma19 (9.28 MB, application/x-xz)
2018-04-01 12:06 UTC, Nikolai Sednev
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 89553 0 master MERGED core: setting network label to NULL 2018-03-28 14:36:40 UTC
oVirt gerrit 89555 0 ovirt-engine-4.2 MERGED core: setting network label to NULL 2018-03-28 18:06:21 UTC

Description Thomas Davis 2018-03-26 18:21:17 UTC
Created attachment 1413290 [details]
hosted-engine log

Description of problem:

[ INFO  ] TASK [Enable GlusterFS at cluster level]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [Set VLAN ID at datacenter level]
[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Network label must be formed only from: English letters, numbers, hyphen or underscore., Attribute: network.label]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Network label must be formed only from: English letters, numbers, hyphen or underscore., Attribute: network.label]\". HTTP response code is 400."}
[ INFO  ] TASK [include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Remove local vm dir]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
[ INFO  ] TASK [Gathering Facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Remove local vm dir]
[ INFO  ] ok: [localhost]
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180326111210.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180326110506-6ga7ie.log


Version-Release number of selected component (if applicable):

ovirt-engine-appliance.noarch                                   4.2-20180321.1.el7.centos                                   @ovirt-4.2-pre
ovirt-engine-sdk-python.noarch                                  3.6.9.1-1.el7                                               @ovirt-4.2
ovirt-host.x86_64                                               4.2.2-2.el7.centos                                          @ovirt-4.2-pre
ovirt-host-dependencies.x86_64                                  4.2.2-2.el7.centos                                          @ovirt-4.2-pre
ovirt-host-deploy.noarch                                        1.7.3-1.el7.centos                                          @ovirt-4.2-pre
ovirt-hosted-engine-ha.noarch                                   2.2.8-1.el7.centos                                          @ovirt-4.2-pre
ovirt-hosted-engine-setup.noarch                                2.2.15-1.el7.centos                                         @ovirt-4.2-pre
ovirt-imageio-common.noarch                                     1.2.2-0.el7.centos                                          @ovirt-4.2-pre
ovirt-imageio-daemon.noarch                                     1.2.2-0.el7.centos                                          @ovirt-4.2-pre
ovirt-provider-ovn-driver.noarch                                1.2.9-1.el7.centos                                          @ovirt-4.2-pre
ovirt-release42-pre.noarch                                      4.2.2-0.6.rc6.20180326144545.git9bcecfa.el7.centos          @ovirt-4.2-pre
ovirt-setup-lib.noarch                                          1.1.4-1.el7.centos                                          @ovirt-4.2
ovirt-vmconsole.noarch                                          1.0.4-1.el7                                                 @ovirt-4.2
ovirt-vmconsole-host.noarch                                     1.0.4-1.el7                                                 @ovirt-4.2


How reproducible:

everytime.

Steps to Reproduce:
1.  yum install release candidate hosted-engine rpm.
2.  hosted-engine --deploy
3.

Actual results:

Failure.

Expected results:

Complete and running oVirt hosted-engine.

Additional info:

Comment 1 Thomas Davis 2018-03-26 18:21:50 UTC
Created attachment 1413291 [details]
vm engine log

Comment 2 Thomas Davis 2018-03-26 18:22:21 UTC
Created attachment 1413292 [details]
engine server log

Comment 3 Dan Kenigsberg 2018-03-27 14:36:37 UTC
attachment 1413291 [details] and attachment 1413292 [details] seem to contain the same log. Would you attach engine.log from within the VM?

Could it be that you specify a network name with a space or another special character?

Comment 4 Thomas Davis 2018-03-27 15:03:16 UTC
Created attachment 1413772 [details]
vm engine log

Comment 5 Thomas Davis 2018-03-27 15:05:11 UTC
I've put the correct vm server log up.

I do not specify anything other than what interface to use, hosted-engine --deploy does that..

Comment 6 Thomas Davis 2018-03-27 15:15:11 UTC
Also, rc4 worked, rc5 and newer is where this popped up.

Is it the engine, or is it hosted-engine --deploy doing the wrong thing?

Comment 7 Thomas Davis 2018-03-27 15:44:50 UTC
There is no network label being passed in via host-deploy/ansible; from

/usr/share/ovirt-hosted-engine-setup/ansible/bootstrap_local_vm.yml


      - name: Set VLAN ID at datacenter level
        ovirt_networks:
          data_center: "{{ DATA_CENTER }}"
          name: "{{ MGMT_NETWORK }}"
          vlan_tag: "{{ vlan_id_out.stdout }}"
          auth: "{{ ovirt_auth }}"
        when: vlan_id_out.stdout|length > 0

is what it is trying to do.  Network Label is not supported till ansible 2.5, the hosted-engine deploy uses:

[root@d8-r13-c0-n1 library]# yum list *ansible*
Loaded plugins: fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
 * base: mirrors.usc.edu
 * extras: dallas.tx.mirror.xygenhosting.com
 * ovirt-4.2: mirrors.ibiblio.org
 * ovirt-4.2-epel: mirror.prgmr.com
 * updates: mirror.texas3006.com
Installed Packages
ansible.noarch                                                             2.4.2.0-2.el7                                          @extras                  
Available Packages
ansible-doc.noarch                                                         2.4.2.0-2.el7                                          extras                   
ovirt-ansible-cluster-upgrade.noarch                                       1.1.6-1.el7.centos                                     ovirt-4.2-pre            
ovirt-ansible-disaster-recovery.noarch                                     0.3-1.el7.centos                                       ovirt-4.2-pre            
ovirt-ansible-engine-setup.noarch                                          1.1.0-1.el7.centos                                     ovirt-4.2                
ovirt-ansible-image-template.noarch                                        1.1.5-1.el7.centos                                     ovirt-4.2                
ovirt-ansible-infra.noarch                                                 1.1.4-1.el7.centos                                     ovirt-4.2-pre            
ovirt-ansible-manageiq.noarch                                              1.1.6-1.el7.centos                                     ovirt-4.2-pre            
ovirt-ansible-repositories.noarch                                          1.1.0-1.el7.centos                                     ovirt-4.2                
ovirt-ansible-roles.noarch                                                 1.1.3-1.el7.centos                                     ovirt-4.2                
ovirt-ansible-vm-infra.noarch                                              1.1.5-1.el7.centos                                     ovirt-4.2-pre            
skydive-ansible.x86_64                                                     0.16.0-1.el7                                           ovirt-4.2-centos-opstools

so I'm wondering if the engine is complaining about a 'null' or non-existent label?

Comment 8 Alona Kaplan 2018-03-28 09:47:11 UTC
The bug was caused by https://gerrit.ovirt.org/#/c/88620/4/packaging/dbscripts/upgrade/04_02_0960_default_all_search_engine_fields_to_empty_string.sql

Null and empty labels are treated differently in the code, changing all the nulls to empty strings breaks the functionality.

Comment 9 Alona Kaplan 2018-03-28 09:55:03 UTC
(In reply to Alona Kaplan from comment #8)
> The bug was caused by
> https://gerrit.ovirt.org/#/c/88620/4/packaging/dbscripts/upgrade/
> 04_02_0960_default_all_search_engine_fields_to_empty_string.sql
> 
> Null and empty labels are treated differently in the code, changing all the
> nulls to empty strings breaks the functionality.

Also, the labels that were changed to be empty should be reverted to be null.

Comment 10 Red Hat Bugzilla Rules Engine 2018-03-28 12:22:05 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 12 Michael Burman 2018-04-01 10:53:12 UTC
Network update:
- Empty label has gone from the networks in the UI drop down list
- All networks that had the empty label, after upgrade has no label as expected('-' sign in the Label column)
- The empty label gone from the setup networks dialogue > labels section
- No regression has caused to network functionality

Comment 13 Nikolai Sednev 2018-04-01 11:56:03 UTC
SHE on these components:
ovirt-hosted-engine-ha-2.2.9-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.15-1.el7ev.noarch
rhvm-appliance-4.2-20180322.0.el7.noarch.rpm
Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

I've tried to deploy over regular untagged network interface and deployment was successful.


Over tagged VLAN it fails, with this error:
[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Network label must be formed only from: English letters, numbers, hyphen or underscore., Attribute: network.label]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Network label must be formed only from: English letters, numbers, hyphen or underscore., Attribute: network.label]\". HTTP response code is 400."}
[ INFO  ] TASK [include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Remove local vm dir]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
[ INFO  ] TASK [Gathering Facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Remove local vm dir]
[ INFO  ] ok: [localhost]
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180401145330.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180401144424-be9gbo.log

Moving back to assigned.

Comment 14 Nikolai Sednev 2018-04-01 12:05:40 UTC
Created attachment 1415792 [details]
engine logs

Comment 15 Nikolai Sednev 2018-04-01 12:06:48 UTC
Created attachment 1415793 [details]
sosreport from puma19

Comment 16 Dan Kenigsberg 2018-04-01 12:18:35 UTC
sosreport-nsednev-he-7.scl.lab.tlv.redhat.com-20180401150029/sos_commands/rpm/sh_-c_rpm_--nodigest_-qa_--qf_NAME_-_VERSION_-_RELEASE_._ARCH_INSTALLTIME_date_awk_-F_printf_-59s_s_n_1_2_sort_-f

ovirt-engine-4.2.2.5-0.1.el7.noarch                         Thu Mar 22 22:24:10 2018

Please retest with 4.2.2.6 which includes the fix of http://gerrit.ovirt.org/89555

Comment 17 Michael Burman 2018-04-01 12:27:58 UTC
(In reply to Nikolai Sednev from comment #13)
> SHE on these components:
> ovirt-hosted-engine-ha-2.2.9-1.el7ev.noarch
> ovirt-hosted-engine-setup-2.2.15-1.el7ev.noarch
> rhvm-appliance-4.2-20180322.0.el7.noarch.rpm
> Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64
> x86_64 x86_64 GNU/Linux
> Red Hat Enterprise Linux Server release 7.5 (Maipo)
> 
> I've tried to deploy over regular untagged network interface and deployment
> was successful.
> 
> 
> Over tagged VLAN it fails, with this error:
> [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is
> "[Network label must be formed only from: English letters, numbers, hyphen
> or underscore., Attribute: network.label]". HTTP response code is 400.
> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault
> reason is \"Operation Failed\". Fault detail is \"[Network label must be
> formed only from: English letters, numbers, hyphen or underscore.,
> Attribute: network.label]\". HTTP response code is 400."}
> [ INFO  ] TASK [include_tasks]
> [ INFO  ] ok: [localhost]
> [ INFO  ] TASK [Remove local vm dir]
> [ INFO  ] changed: [localhost]
> [ INFO  ] TASK [Notify the user about a failure]
> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The
> system may not be provisioned according to the playbook results: please
> check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
> [ ERROR ] Failed to execute stage 'Closing up': Failed executing
> ansible-playbook
> [ INFO  ] Stage: Clean up
> [ INFO  ] Cleaning temporary resources
> [ INFO  ] TASK [Gathering Facts]
> [ INFO  ] ok: [localhost]
> [ INFO  ] TASK [include_tasks]
> [ INFO  ] ok: [localhost]
> [ INFO  ] TASK [Remove local vm dir]
> [ INFO  ] ok: [localhost]
> [ INFO  ] Generating answer file
> '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180401145330.conf'
> [ INFO  ] Stage: Pre-termination
> [ INFO  ] Stage: Termination
> [ ERROR ] Hosted Engine deployment failed: please check the logs for the
> issue, fix accordingly or re-deploy from scratch.
>           Log file is located at
> /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180401144424-
> be9gbo.log
> 
> Moving back to assigned.

Hi Nikolai, but you are not testing with the correct engine version.
You running an old appliance from last month 22.3, it doesn't have the engine version with this fix. Please re-test with the correct engine version.
Moving back to ON_QA

Comment 21 Nikolai Sednev 2018-04-01 13:08:22 UTC
Moving to modified, until QA gets the appliance with the fix.

Comment 22 Michael Burman 2018-04-02 08:04:21 UTC
(In reply to Nikolai Sednev from comment #21)
> Moving to modified, until QA gets the appliance with the fix.

Hi Nikolai, 
We got the new appliance(rhvm-appliance-4.2-20180401.0)with rhvm 4.2.2.6-0.1.el7 which has the fix to this bug. Please re-test

Comment 23 Michael Burman 2018-04-02 08:04:57 UTC
(In reply to Michael Burman from comment #12)
> Network update:
> - Empty label has gone from the networks in the UI drop down list
> - All networks that had the empty label, after upgrade has no label as
> expected('-' sign in the Label column)
> - The empty label gone from the setup networks dialogue > labels section
> - No regression has caused to network functionality

Tested on - 4.2.2.6-0.1.el7

Comment 24 Nikolai Sednev 2018-04-02 13:27:37 UTC
Failed on these components:
rhvm-appliance-4.2-20180401.0.el7.noarch.rpm
ovirt-hosted-engine-ha-2.2.9-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.15-1.el7ev.noarch
Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Engine inside the appliance:
ovirt-engine-setup-base-4.2.2.6-0.1.el7.noarch

Deployed over NFS.

[ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_vms": [{"affinity_labels": [], "applications": [], "bios": {"boot_menu": {"enabled": false}}, "cdroms": [], "cluster": {"href": "/ovirt-engine/api/clusters/1748a294-3676-11e8-b3bd-00163eeeeee1", "id": "1748a294-3676-11e8-b3bd-00163eeeeee1"}, "cpu": {"architecture": "x86_64", "topology": {"cores": 1, "sockets": 4, "threads": 1}}, "cpu_profile": {"href": "/ovirt-engine/api/cpuprofiles/58ca604e-01a7-003f-01de-000000000250", "id": "58ca604e-01a7-003f-01de-000000000250"}, "cpu_shares": 0, "creation_time": "2018-04-02 16:06:05.791000+03:00", "delete_protected": false, "disk_attachments": [], "display": {"address": "127.0.0.1", "allow_override": false, "copy_paste_enabled": true, "disconnect_action": "LOCK_SCREEN", "file_transfer_enabled": true, "monitors": 1, "port": 5900, "single_qxl_pci": false, "smartcard_enabled": false, "type": "vnc"}, "graphics_consoles": [], "high_availability": {"enabled": false, "priority": 0}, "host": {"href": "/ovirt-engine/api/hosts/043bc7e9-5580-48d8-8953-6d84a33ed596", "id": "043bc7e9-5580-48d8-8953-6d84a33ed596"}, "host_devices": [], "href": "/ovirt-engine/api/vms/b537b175-8545-426d-8571-d6652d56d2ef", "id": "b537b175-8545-426d-8571-d6652d56d2ef", "io": {"threads": 0}, "katello_errata": [], "large_icon": {"href": "/ovirt-engine/api/icons/e57746a0-a95b-019c-4355-27b4eac77170", "id": "e57746a0-a95b-019c-4355-27b4eac77170"}, "memory": 17179869184, "memory_policy": {"guaranteed": 17179869184, "max": 17179869184}, "migration": {"auto_converge": "inherit", "compressed": "inherit"}, "migration_downtime": -1, "name": "external-HostedEngineLocal", "next_run_configuration_exists": false, "nics": [], "numa_nodes": [], "numa_tune_mode": "interleave", "origin": "external", "original_template": {"href": "/ovirt-engine/api/templates/00000000-0000-0000-0000-000000000000", "id": "00000000-0000-0000-0000-000000000000"}, "os": {"boot": {"devices": ["hd"]}, "type": "other"}, "permissions": [], "placement_policy": {"affinity": "migratable"}, "quota": {"id": "2996557c-3676-11e8-8971-00163eeeeee1"}, "reported_devices": [], "run_once": false, "sessions": [], "small_icon": {"href": "/ovirt-engine/api/icons/5ba0b8a7-51c6-5ef5-7ed0-495a62737d13", "id": "5ba0b8a7-51c6-5ef5-7ed0-495a62737d13"}, "snapshots": [], "sso": {"methods": [{"id": "guest_agent"}]}, "start_paused": false, "stateless": false, "statistics": [], "status": "unknown", "storage_error_resume_behaviour": "auto_resume", "tags": [], "template": {"href": "/ovirt-engine/api/templates/00000000-0000-0000-0000-000000000000", "id": "00000000-0000-0000-0000-000000000000"}, "time_zone": {"name": "Etc/GMT"}, "type": "desktop", "usb": {"enabled": false}, "watchdogs": []}]}, "attempts": 24, "changed": false}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
[ INFO  ] TASK [Gathering Facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Remove local vm dir]
[ INFO  ] changed: [localhost]
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180402162515.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180402155548-4n0to5.log
You have new mail in /var/spool/mail/root

Comment 25 Nikolai Sednev 2018-04-02 13:49:41 UTC
The bug now is different, I've opened a separate https://bugzilla.redhat.com/show_bug.cgi?id=1562787 here.
Now its failing because of the different error (affinity_labels).
Moving this specific bug to verified as its specific case had been fixed.

Comment 26 Sandro Bonazzola 2018-04-05 09:38:31 UTC
This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.