Bug 1809070

Summary: [openvswitch] HE deployment is Failing on RHVH-4.4
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: milind <mwaykole>
Component: rhhiAssignee: Gobinda Das <godas>
Status: CLOSED ERRATA QA Contact: milind <mwaykole>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rhhiv-1.8CC: cshao, dholler, godas, mtessun, rhs-bugs, sasundar
Target Milestone: ---Keywords: TestBlocker
Target Release: RHHI-V 1.8   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-04 14:51:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1809208    
Bug Blocks: 1779977    
Attachments:
Description Flags
host-deploy.log
none
host-deploy.log none

Description milind 2020-03-02 11:36:49 UTC
Description of problem:
HE Deployment is failing in RHHI-V 1.8
--------------
[root@ ~]# imgbase w (if applicable)
You are on rhvh-4.4.0.14-0.20200226.0+1

[root@ ~]# rpm -qa | grep -i ansible 
ansible-2.9.5-1.el8ae.noarch
gluster-ansible-infra-1.0.4-4.el8rhgs.noarch
ovirt-ansible-hosted-engine-setup-1.0.35-1.el8ev.noarch
gluster-ansible-cluster-1.0-1.el8rhgs.noarch
gluster-ansible-repositories-1.0.1-2.el8rhgs.noarch
gluster-ansible-features-1.0.5-4.el8rhgs.noarch
gluster-ansible-roles-1.0.5-5.el8rhgs.noarch
ovirt-ansible-engine-setup-1.2.1-1.el8ev.noarch
gluster-ansible-maintenance-1.0.1-1.el8rhgs.noarch
----------------


Version-Release number of selected component (if applicable):
4.4.0
How reproducible:
Always

Steps to Reproduce:
1. from cockpit click in Hyperconverged and deploy gluster it will be successfully
2.Deploy HE this step will fail
3.

Actual results:
Failing 


Expected results:
HE deployment should not fail


Additional info:

[ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_hosts": [{"address": "rhsqa-grafton1.lab.eng.blr.redhat.com", "affinity_labels": [], "auto_numa_status": "unknown", "certificate": {"organization": "lab.eng.blr.redhat.com", "subject": "O=lab.eng.blr.redhat.com,CN=rhsqa-grafton1.lab.eng.blr.redhat.com"}, "cluster": {"href": "/ovirt-engine/api/clusters/5b41ee24-5c3a-11ea-8596-004554194801", "id": "5b41ee24-5c3a-11ea-8596-004554194801"}, "comment": "", "cpu": {"speed": 0.0, "topology": {}}, "device_passthrough": {"enabled": false}, "devices": [], "external_network_provider_configurations": [], "external_status": "ok", "hardware_information": {"supported_rng_sources": []}, "hooks": [], "href": "/ovirt-engine/api/hosts/26ba148a-79ac-4c5c-ba7e-c11b4e86f37b", "id": "26ba148a-79ac-4c5c-ba7e-c11b4e86f37b", "katello_errata": [], "kdump_status": "unknown", "ksm": {"enabled": false}, "max_scheduling_memory": 0, "memory": 0, "name": "rhsqa-grafton1.lab.eng.blr.redhat.com", "network_attachments": [], "nics": [], "numa_nodes": [], "numa_supported": false, "os": {"custom_kernel_cmdline": ""}, "permissions": [], "port": 54321, "power_management": {"automatic_pm_enabled": true, "enabled": false, "kdump_detection": true, "pm_proxies": []}, "protocol": "stomp", "se_linux": {}, "spm": {"priority": 5, "status": "none"}, "ssh": {"fingerprint": "SHA256:afIfjlqbi4e9fzOARDkN0wfg2IVI3qI/Dejc3kTUHPo", "port": 22}, "statistics": [], "status": "install_failed", "storage_connection_extensions": [], "summary": {"total": 0}, "tags": [], "transparent_huge_pages": {"enabled": false}, "type": "rhel", "unmanaged_networks": [], "update_available": false, "vgpu_placement": "consolidated"}]}, "attempts": 120, "changed": false, "deprecations": [{"msg": "The 'ovirt_host_facts' module has been renamed to 'ovirt_host_info', and the renamed one no longer returns ansible_facts", "version": "2.13"}]}
[ INFO ] TASK [ovirt.hosted_engine_setup : Fetch logs from the engine VM]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Set destination directory path]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Create destination directory]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Find the local appliance image]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Set local_vm_disk_path]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers]
[ INFO ] ok: [localhost -> localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Copy engine logs]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}

Comment 2 SATHEESARAN 2020-03-02 12:10:06 UTC
HE deployment fails during host deploy stage as seen in the host_deploy_ansible logs from engine

<snip>

2020-03-02 07:31:55 UTC - TASK [ovirt-provider-ovn-driver : Ensure Open vSwitch is started] **************
2020-03-02 07:31:58 UTC - fatal: [rhsqa-grafton1.lab.eng.blr.redhat.com]: FAILED! => {"changed": false, "msg": "Unable to start service openvswitch: A dependency job for openvswitch.ser
vice failed. See 'journalctl -xe' for details.\n"}
2020-03-02 07:31:58 UTC - {
  "status" : "OK",
  "msg" : "",
  "data" : {
    "event" : "runner_on_failed",
    "uuid" : "671acf62-2b78-40d3-b836-7703c33f28cc",
    "stdout" : "fatal: [rhsqa-grafton1.lab.eng.blr.redhat.com]: FAILED! => {\"changed\": false, \"msg\": \"Unable to start service openvswitch: A dependency job for openvswitch.service failed. See 'journalctl -xe' for details.\\n\"}",
    "counter" : 223,
    "pid" : 32046,
    "created" : "2020-03-02T07:31:57.564460",
    "end_line" : 218,
    "runner_ident" : "23d84128-5c57-11ea-a7df-004855204901",
    "start_line" : 217,
    "event_data" : {
      "play_pattern" : "all",
      "play" : "all",
      "event_loop" : null,
      "task_args" : "",
      "remote_addr" : "rhsqa-grafton1.lab.eng.blr.redhat.com",
      "res" : {
        "invocation" : {
          "module_args" : {
            "no_block" : false,
            "force" : null,
            "name" : "openvswitch",
            "daemon_reexec" : false,
            "enabled" : true,
            "daemon_reload" : false,
            "state" : "started",
            "masked" : null,
            "scope" : null,
            "user" : null
          }
        },
        "msg" : "Unable to start service openvswitch: A dependency job for openvswitch.service failed. See 'journalctl -xe' for details.\n",
        "changed" : false,
        "_ansible_no_log" : false
      },
      "pid" : 32046,
      "play_uuid" : "00485520-4901-1885-a5ed-000000000006",
      "task_uuid" : "00485520-4901-1885-a5ed-00000000002f",
      "task" : "Ensure Open vSwitch is started",
"playbook_uuid" : "e3125038-13f4-4cec-8a49-6b112bbd1cfb",
      "playbook" : "ovirt-host-deploy.yml",
      "task_action" : "service",
      "host" : "rhsqa-grafton1.lab.eng.blr.redhat.com",
      "ignore_errors" : null,
      "role" : "ovirt-provider-ovn-driver",
      "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml:8"
    },
    "parent_uuid" : "00485520-4901-1885-a5ed-00000000002f"
  }
}
</snip>

Comment 4 SATHEESARAN 2020-03-02 12:15:15 UTC
Created attachment 1666939 [details]
host-deploy.log

Comment 5 SATHEESARAN 2020-03-02 12:19:23 UTC
Created attachment 1666947 [details]
host-deploy.log

Comment 14 SATHEESARAN 2020-05-05 02:28:29 UTC
RHHI-V 1.8 deployment with 3 node works good with the workaround from Bug 1823423.
The particular issue on the bug is not seen

The builds used for the verification are:
RHVH-4.4-20200417.0-RHVH-x86_64-dvd1.iso 
rhvm-appliance-4.4-20200417.0.el8ev.x86_64.rpm 

@Milind, could you verify this bug also with single node RHHI-V 1.8 deployment
and verify this bug ?

Comment 15 milind 2020-05-05 11:42:32 UTC
As HE Deployment is successfully done
Hence marking this bug as verified  
=======
[root@rhsqa-grafton1 vdsm]# imgbase w
You are on rhvh-4.4.0.18-0.20200417.0+1

Comment 18 errata-xmlrpc 2020-08-04 14:51:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHHI for Virtualization 1.8 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:3314