Bug 1814940 - otopi fails with: Python 2 is disabled in RHEL8
Summary: otopi fails with: Python 2 is disabled in RHEL8
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: otopi
Classification: oVirt
Component: Core
Version: master
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ovirt-4.4.0
: 1.9.1
Assignee: Yedidyah Bar David
QA Contact: Wei Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-19 05:24 UTC by Wei Wang
Modified: 2020-05-20 20:00 UTC (History)
12 users (show)

Fixed In Version: otopi-1.9.1
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-20 20:00:20 UTC
oVirt Team: Integration
Embargoed:
sbonazzo: ovirt-4.4?
sbonazzo: blocker?
sbonazzo: planning_ack?
sbonazzo: devel_ack+
weiwang: testing_ack+


Attachments (Terms of Use)
log files (2.60 MB, application/gzip)
2020-03-19 05:24 UTC, Wei Wang
no flags Details
picture (124.41 KB, image/png)
2020-03-20 04:14 UTC, Wei Wang
no flags Details
log files (2.03 MB, application/gzip)
2020-03-20 04:30 UTC, Wei Wang
no flags Details
host_add_to_standalone_engine log files (1.86 MB, application/gzip)
2020-03-20 06:56 UTC, Wei Wang
no flags Details
engine log (2.49 MB, application/gzip)
2020-03-24 08:44 UTC, Wei Wang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 107812 0 master MERGED core: Try python3 first when checking distribution 2021-01-29 16:58:23 UTC

Description Wei Wang 2020-03-19 05:24:58 UTC
Created attachment 1671329 [details]
log files

Description of problem:
Hosted engine deploy failed at "Wait for the host to be up" ansible_task.

ovirt-hosted-engine-setup-ansible-bootstrap_local_vm-2020219122244-qs5126.log
2020-03-19 12:46:24,876+0800 ERROR ansible failed {
    "ansible_host": "localhost",
    "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
    "ansible_result": {
        "_ansible_no_log": false,
        "ansible_facts": {
            "ovirt_hosts": [
                {
                    "address": "hp-dl388g9-04.lab.eng.pek2.**FILTERED**.com",
                    "affinity_labels": [],
                    "auto_numa_status": "unknown",
                    "certificate": {
                        "organization": "lab.eng.pek2.**FILTERED**.com",
                        "subject": "O=lab.eng.pek2.**FILTERED**.com,CN=hp-dl388g9-04.lab.eng.pek2.**FILTERED**.com"
                    },
                    "cluster": {
                        "href": "/ovirt-engine/api/clusters/32caf9be-699a-11ea-b3ed-5254003404b0",
                        "id": "32caf9be-699a-11ea-b3ed-5254003404b0"
                    },
                    "comment": "",
                    "cpu": {
                        "speed": 0.0,
                        "topology": {}
                    },
                    "device_passthrough": {
                        "enabled": false
                    },
                    "devices": [],
                    "external_network_provider_configurations": [],
                    "external_status": "ok",
                    "hardware_information": {
                        "supported_rng_sources": []
                    },
                    "hooks": [],
                    "href": "/ovirt-engine/api/hosts/87423d20-8cd4-42fb-bed6-35a59d94e967",
                    "id": "87423d20-8cd4-42fb-bed6-35a59d94e967",
                    "katello_errata": [],
                    "kdump_status": "unknown",
                    "ksm": {
                        "enabled": false
                    },
                    "max_scheduling_memory": 0,
                    "memory": 0,
                    "name": "hp-dl388g9-04.lab.eng.pek2.**FILTERED**.com",
                    "network_attachments": [],
                    "nics": [],
                    "numa_nodes": [],
                    "numa_supported": false,
                    "os": {
                        "custom_kernel_cmdline": ""
                    },
                    "permissions": [],
                    "port": 54321,
                    "power_management": {
                        "automatic_pm_enabled": true,
                        "enabled": false,
                        "kdump_detection": true,
                        "pm_proxies": []
                    },
                    "protocol": "stomp",
                    "se_linux": {},
                    "spm": {
                        "priority": 5,
                        "status": "none"
                    },
                    "ssh": {
                        "fingerprint": "SHA256:H5jKj02OXvEtQsLyGj/n7a31uWgwRxRf4J7irqRhef8",
                        "port": 22
                    },
                    "statistics": [],
                    "status": "installing",
                    "storage_connection_extensions": [],
                    "summary": {
                        "total": 0
                    },
                    "tags": [],
                    "transparent_huge_pages": {
                        "enabled": false
                    },
                    "type": "rhel",
                    "unmanaged_networks": [],
                    "update_available": false,
                    "vgpu_placement": "consolidated"
                }
            ]
        },
        "attempts": 120,
        "changed": false,
        "deprecations": [
            {
                "msg": "The 'ovirt_host_facts' module has been renamed to 'ovirt_host_info', and the renamed one no longer returns ansible_facts",
                "version": "2.13"
            }
        ],
        "invocation": {
            "module_args": {
                "all_content": false,
                "cluster_version": null,
                "fetch_nested": false,
                "nested_attributes": [],
                "pattern": "name=hp-dl388g9-04.lab.eng.pek2.**FILTERED**.com"
            }
        }
    },
    "ansible_task": "Wait for the host to be up",
    "ansible_type": "task",
    "status": "FAILED",
    "task_duration": 733
}

supervdsm.log
MainProcess|vm/9dd5bff6::ERROR::2020-03-19 12:41:26,922::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in prepareVmChannel
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper
    res = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_api/virt.py", line 39, in prepareVmChannel
    fsinfo = os.stat(socketFile)
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/libvirt/qemu/channels/9dd5bff6-6ae1-42fa-8c66-11383dd3b038.com.redhat.rhevm.vdsm'


Version-Release number of selected component (if applicable):
RHVH-UNSIGNED-ISO-4.4-RHEL-8-20200318.0-RHVH-x86_64-dvd1
cockpit-system-211.3-1.el8.noarch
cockpit-ws-211.3-1.el8.x86_64
cockpit-ovirt-dashboard-0.14.2-1.el8ev.noarch
cockpit-211.3-1.el8.x86_64
cockpit-bridge-211.3-1.el8.x86_64
cockpit-dashboard-211.3-1.el8.noarch
cockpit-storaged-211.3-1.el8.noarch
ovirt-ansible-engine-setup-1.2.2-1.el8ev.noarch
ovirt-ansible-hosted-engine-setup-1.1.0-1.el8ev.noarch
rhvm-appliance-4.4-20200123.0.el8ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Clean install RHVH-UNSIGNED-ISO-4.4-RHEL-8-20200318.0-RHVH-x86_64-dvd1
2.Yum install rhvm-appliance-4.4-20200123.0.el8ev.x86_64
3.Deploy hosted engine via cockpit

Actual results:
Hosted engine deploy failed at "Wait for the host to be up" ansible_task.

Expected results:
Hosted engine deploy successful

Additional info:

Comment 1 Wei Wang 2020-03-19 08:03:45 UTC
The new rhvm-appliance for 4.4 is coming, I will retest this bug later and give the further update.

Comment 2 Yedidyah Bar David 2020-03-19 10:09:46 UTC
(In reply to Wei Wang from comment #1)
> The new rhvm-appliance for 4.4 is coming, I will retest this bug later and
> give the further update.

Thanks, please do.

That said, the failure is simply due to the fact that adding the host took too much time. We allow up to 10 minutes, and [1] (in the attached logs) starts with:

2020-03-19 12:34:12 CST - TASK [Gathering Facts] *********************************************************

And ends (cut in the middle) with:

2020-03-19 12:44:57 CST - TASK [ovirt-host-deploy-vnc-certificates : Setup VNC PKI] **********************

So:

1. It might be wise to simply allow more time, for now. That's a trivial change:

https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/commit/11fd4397d44b56d7f696016b772791efad5b1aa3

2. Perhaps try to understand why it took so long. E.g., can you try adding same host (perhaps with reprovisioned image of RHVH) to a standalone engine, and attach the host-deploy log? Skimming through the file, I see e.g.:

2020-03-19 12:37:50 CST - TASK [ovirt-host-deploy-vdsm-certificates : Add vdsm cacert files] *************
2020-03-19 12:38:29 CST - 
2020-03-19 12:38:29 CST - {
  "status" : "OK",

Meaning, copying 3 small files took 40 seconds. I think that's too much.

[1] log/ovirt-hosted-engine-setup/engine-logs-2020-03-19T04\:23\:56Z/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20200319123407-hp-dl388g9-04.lab.eng.pek2.redhat.com-3eeecdb2.log

Comment 3 Wei Wang 2020-03-20 04:12:33 UTC
Test Version:
RHVH-UNSIGNED-ISO-4.4-RHEL-8-20200318.0-RHVH-x86_64-dvd1
cockpit-system-211.3-1.el8.noarch
cockpit-ws-211.3-1.el8.x86_64
cockpit-ovirt-dashboard-0.14.2-1.el8ev.noarch
cockpit-211.3-1.el8.x86_64
cockpit-bridge-211.3-1.el8.x86_64
cockpit-dashboard-211.3-1.el8.noarch
cockpit-storaged-211.3-1.el8.noarch
ovirt-ansible-engine-setup-1.2.2-1.el8ev.noarch
ovirt-ansible-hosted-engine-setup-1.1.0-1.el8ev.noarch
rhvm-appliance-4.4-20200319.0.el8ev.x86_64

Test Steps:
1. Clean install RHVH-UNSIGNED-ISO-4.4-RHEL-8-20200318.0-RHVH-x86_64-dvd1
2. Yum install rhvm-appliance-4.4-20200319.0.el8ev.x86_64
3. Deploy hosted engine via cockpit

Result:
Failed at ansible task "ovirt.engine-setup : Run engine-setup with answerfile"

Please refer the picture "he_0320.png"
[INFO ] TASK [ovirt.engine-setup : Run engine-setup with answerfile]
[ ERROR ] fatal: [localhost -> rhevh-hostedengine-vm-06.lab.eng.pek2.redhat.com]: FAILED! => {"changed": true, "cmd": ["engine-setup", "--accept-defaults", "--config-append=/root/ovirt-engine-answers", "--offline"], "delta": "0:00:26.581864", "end": "2020-03-20 11:46:54.101298", "msg": "non-zero return code", "rc": 1, "start": "2020-03-20 11:46:27.519434", "stderr": "\nERROR: Python 2 is disabled in RHEL8.\n\n- For guidance on porting to Python 3, see the\n Conservative Python3 Porting Guide:\n http://portingguide.readthedocs.io/\n\n- If you need Python 2 at runtime:\n - Use the python27 module\n\n- If you do not have access to BZ#1533919:\n - Use the python27 module\n\n- If you need to use Python 2 only at RPM build time:\n - File a bug blocking BZ#1533919:\n https://bugzilla.redhat.com/show_bug.cgi?id=1533919\n - Set the environment variable RHEL_ALLOW_PYTHON2_FOR_BUILD=1\n (Note that if you do not file the bug as above,\n this workaround will break without warning in the future.)\n\n- If you need to use Python 2 only for tests:\n - File a bug blocking BZ#1533919:\n https://bugzilla.redhat.com/show_bug.cgi?id=1533919\n (If your test tool does not have a Bugzilla component,\n feel free to use `python2`.)\n - Use /usr/bin/python2-for-tests instead of python2 to run\n your tests.\n (Note that if you do not file the bug as above,\n this workaround will break without warning in the future.)\n\nFor details, see https://hurl.corp.redhat.com/rhel8-py2\n\nFatal Python error: Python 2 is disabled\n\nERROR: Python 2 is disabled in RHEL8.\n\n- For guidance on porting to Python 3, see the\n Conservative Python3 Porting Guide:\n http://portingguide.readthedocs.io/\n\n- If you need Python 2 at runtime:\n - Use the python27 module\n\n- If you do not have access to BZ#1533919:\n - Use the python27 module\n\n- If you need to use Python 2 only at RPM build time:\n - File a bug blocking BZ#1533919:\n https://bugzilla.redhat.com/show_bug.cgi?id=1533919\n - Set the environment variable RHEL_ALLOW_PYTHON2_FOR_BUILD=1\n (Note that if you do not file the bug as above,\n this workaround will break without warning in the future.)\n\n- If you need to use Python 2 only for tests:\n - File a bug blocking BZ#1533919:\n https://bugzilla.redhat.com/show_bug.cgi?id=1533919\n (If your test tool does not have a Bugzilla component,\n feel free to use `python2`.)\n - Use /usr/bin/python2-for-tests instead of python2 to run\n your tests.\n (Note that if you do not file the bug as above,\n this workaround will break without warning in the future.)\n\nFor details, see https://hurl.corp.redhat.com/rhel8-py2\n\nFatal Python error: Python 2 is disabled\n/usr/share/otopi/otopi-functions: line 47: [: : integer expression expected", "stderr_lines": ["", "ERROR: Python 2 is disabled in RHEL8.", "", "- For guidance on porting to Python 3, see the", " Conservative Python3 Porting Guide:", " http://portingguide.readthedocs.io/", "", "- If you need Python 2 at runtime:", " - Use the python27 module", "", "- If you do not have access to BZ#1533919:", " - Use the python27 module", "", "- If you need to use Python 2 only at RPM build time:", " - File a bug blocking BZ#1533919:", " https://bugzilla.redhat.com/show_bug.cgi?id=1533919", " - Set the environment variable RHEL_ALLOW_PYTHON2_FOR_BUILD=1", " (Note that if you do not file the bug as above,", " this workaround will break without warning in the future.)", "", "- If you need to use Python 2 only for tests:", " - File a bug blocking BZ#1533919:", " https://bugzilla.redhat.com/show_bug.cgi?id=1533919", " (If your test tool does not have a Bugzilla component,", " feel free to use `python2`.)", " - Use /usr/bin/python2-for-tests instead of python2 to run", " your tests.", " (Note that if you do not file the bug as above,", " this workaround will break without warning in the future.)", "", "For details, see https://hurl.corp.redhat.com/rhel8-py2", "", "Fatal Python error: Python 2 is disabled", "", "ERROR: Python 2 is disabled in RHEL8.", "", "- For guidance on porting to Python 3, see the", " Conservative Python3 Porting Guide:", " http://portingguide.readthedocs.io/", "", "- If you need Python 2 at runtime:", " - Use the python27 module", "", "- If you do not have access to BZ#1533919:", " - Use the python27 module", "", "- If you need to use Python 2 only at RPM build time:", " - File a bug blocking BZ#1533919:", " https://bugzilla.redhat.com/show_bug.cgi?id=1533919", " - Set the environment variable RHEL_ALLOW_PYTHON2_FOR_BUILD=1", " (Note that if you do not file the bug as above,", " this workaround will break without warning in the future.)", "", "- If you need to use Python 2 only for tests:", " - File a bug blocking BZ#1533919:", " https://bugzilla.redhat.com/show_bug.cgi?id=1533919", " (If your test tool does not have a Bugzilla component,", " feel free to use `python2`.)", " - Use /usr/bin/python2-for-tests instead of python2 to run", " your tests.", " (Note that if you do not file the bug as above,", " this workaround will break without warning in the future.)", "", "For details, see https://hurl.corp.redhat.com/rhel8-py2", "", "Fatal Python error: Python 2 is disabled", "/usr/share/otopi/otopi-functions: line 47: [: : integer expression expected"], "stdout": "[ INFO ] Stage: Initializing\n[ INFO ] Stage: Environment setup\n Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /root/ovirt-engine-answers\n Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20200320114633-ujxnbe.log\n Version: otopi-1.9.0 (otopi-1.9.0-1.el8ev)\n[ INFO ] Stage: Environment packages setup\n[ INFO ] Stage: Programs detection\n[ INFO ] Stage: Environment setup (late)\n[ INFO ] Stage: Environment customization\n \n --== PRODUCT OPTIONS ==--\n \n Set up Cinderlib integration\n (Currently in tech preview)\n (Yes, No) [No]: \n Configure ovirt-provider-ovn (Yes, No) [Yes]: \n \n * Please note * : Data Warehouse is required for the engine.\n If you choose to not configure it on this host, you have to configure\n it on a remote host, and then configure the engine on this host so\n that it can access the database of the remote Data Warehouse host.\n Configure Data Warehouse on this host (Yes, No) [Yes]: \n \n --== PACKAGES ==--\n \n \n --== NETWORK CONFIGURATION ==--\n \n Host fully qualified DNS name of this server [rhevh-hostedengine-vm-06.lab.eng.pek2.redhat.com]: \n[ INFO ] firewalld will be configured as firewall manager.\n \n --== DATABASE CONFIGURATION ==--\n \n Where is the DWH database located? (Local, Remote) [Local]: \n Setup can configure the local postgresql server automatically for the DWH to run. This may conflict with existing applications.\n Would you like Setup to automatically configure postgresql and create DWH database, or prefer to perform that manually? (Automatic, Manual) [Automatic]: \n \n --== OVIRT ENGINE CONFIGURATION ==--\n \n Use default credentials (admin@internal) for ovirt-provider-ovn (Yes, No) [Yes]: \n \n --== STORAGE CONFIGURATION ==--\n \n \n --== PKI CONFIGURATION ==--\n \n Organization name for certificate [lab.eng.pek2.redhat.com]: \n \n --== APACHE CONFIGURATION ==--\n \n \n --== SYSTEM CONFIGURATION ==--\n \n \n --== MISC CONFIGURATION ==--\n \n Please choose Data Warehouse sampling scale:\n (1) Basic\n (2) Full\n (1, 2)[1]: \n \n --== END OF CONFIGURATION ==--\n \n[ INFO ] Stage: Setup validation\n \n --== CONFIGURATION PREVIEW ==--\n \n Application mode : both\n Default SAN wipe after delete : False\n Host FQDN : rhevh-hostedengine-vm-06.lab.eng.pek2.redhat.com\n Firewall manager : firewalld\n Update Firewall : True\n Set up Cinderlib integration : False\n Configure local Engine database : True\n Set application as default page : True\n Configure Apache SSL : True\n Engine database host : localhost\n Engine database port : 5432\n Engine database secured connection : False\n Engine database host name validation : False\n Engine database name : engine\n Engine database user name : engine\n Engine installation : True\n PKI organization : lab.eng.pek2.redhat.com\n Set up ovirt-provider-ovn : True\n Configure WebSocket Proxy : True\n DWH installation : True\n DWH database host : localhost\n DWH database port : 5432\n Configure local DWH database : True\n Configure VMConsole Proxy : True\n[ INFO ] Stage: Transaction setup\n[ INFO ] Stopping engine service\n[ INFO ] Stopping ovirt-fence-kdump-listener service\n[ INFO ] Stopping dwh service\n[ INFO ] Stopping vmconsole-proxy service\n[ INFO ] Stopping websocket-proxy service\n[ INFO ] Stage: Misc configuration (early)\n[ INFO ] Stage: Package installation\n[ INFO ] Stage: Misc configuration\n[ INFO ] Upgrading CA\n[ INFO ] Initializing PostgreSQL\n[ INFO ] Creating PostgreSQL 'engine' database\n[ INFO ] Configuring PostgreSQL\n[ INFO ] Creating PostgreSQL 'ovirt_engine_history' database\n[ INFO ] Configuring PostgreSQL\n[ INFO ] Creating CA: /etc/pki/ovirt-engine/ca.pem\n[ INFO ] Creating CA: /etc/pki/ovirt-engine/qemu-ca.pem\n[ INFO ] Updating OVN SSL configuration\n[ INFO ] Creating/refreshing DWH database schema\n[ ERROR ] Failed to execute stage 'Misc configuration': Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute\n[ INFO ] Rolling back DWH database schema\n[ INFO ] Clearing DWH database ovirt_engine_history\n[ INFO ] Stage: Clean up\n Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20200320114633-ujxnbe.log\n[ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200320114653-setup.conf'\n[ INFO ] Stage: Pre-termination\n[ INFO ] Stage: Termination\n[ ERROR ] Execution of setup failed", "stdout_lines": ["[ INFO ] Stage: Initializing", "[ INFO ] Stage: Environment setup", " Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /root/ovirt-engine-answers", " Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20200320114633-ujxnbe.log", " Version: otopi-1.9.0 (otopi-1.9.0-1.el8ev)", "[ INFO ] Stage: Environment packages setup", "[ INFO ] Stage: Programs detection", "[ INFO ] Stage: Environment setup (late)", "[ INFO ] Stage: Environment customization", " ", " --== PRODUCT OPTIONS ==--", " ", " Set up Cinderlib integration", " (Currently in tech preview)", " (Yes, No) [No]: ", " Configure ovirt-provider-ovn (Yes, No) [Yes]: ", " ", " * Please note * : Data Warehouse is required for the engine.", " If you choose to not configure it on this host, you have to configure", " it on a remote host, and then configure the engine on this host so", " that it can access the database of the remote Data Warehouse host.", " Configure Data Warehouse on this host (Yes, No) [Yes]: ", " ", " --== PACKAGES ==--", " ", " ", " --== NETWORK CONFIGURATION ==--", " ", " Host fully qualified DNS name of this server [rhevh-hostedengine-vm-06.lab.eng.pek2.redhat.com]: ", "[ INFO ] firewalld will be configured as firewall manager.", " ", " --== DATABASE CONFIGURATION ==--", " ", " Where is the DWH database located? (Local, Remote) [Local]: ", " Setup can configure the local postgresql server automatically for the DWH to run. This may conflict with existing applications.", " Would you like Setup to automatically configure postgresql and create DWH database, or prefer to perform that manually? (Automatic, Manual) [Automatic]: ", " ", " --== OVIRT ENGINE CONFIGURATION ==--", " ", " Use default credentials (admin@internal) for ovirt-provider-ovn (Yes, No) [Yes]: ", " ", " --== STORAGE CONFIGURATION ==--", " ", " ", " --== PKI CONFIGURATION ==--", " ", " Organization name for certificate [lab.eng.pek2.redhat.com]: ", " ", " --== APACHE CONFIGURATION ==--", " ", " ", " --== SYSTEM CONFIGURATION ==--", " ", " ", " --== MISC CONFIGURATION ==--", " ", " Please choose Data Warehouse sampling scale:", " (1) Basic", " (2) Full", " (1, 2)[1]: ", " ", " --== END OF CONFIGURATION ==--", " ", "[ INFO ] Stage: Setup validation", " ", " --== CONFIGURATION PREVIEW ==--", " ", " Application mode : both", " Default SAN wipe after delete : False", " Host FQDN : rhevh-hostedengine-vm-06.lab.eng.pek2.redhat.com", " Firewall manager : firewalld", " Update Firewall : True", " Set up Cinderlib integration : False", " Configure local Engine database : True", " Set application as default page : True", " Configure Apache SSL : True", " Engine database host : localhost", " Engine database port : 5432", " Engine database secured connection : False", " Engine database host name validation : False", " Engine database name : engine", " Engine database user name : engine", " Engine installation : True", " PKI organization : lab.eng.pek2.redhat.com", " Set up ovirt-provider-ovn : True", " Configure WebSocket Proxy : True", " DWH installation : True", " DWH database host : localhost", " DWH database port : 5432", " Configure local DWH database : True", " Configure VMConsole Proxy : True", "[ INFO ] Stage: Transaction setup", "[ INFO ] Stopping engine service", "[ INFO ] Stopping ovirt-fence-kdump-listener service", "[ INFO ] Stopping dwh service", "[ INFO ] Stopping vmconsole-proxy service", "[ INFO ] Stopping websocket-proxy service", "[ INFO ] Stage: Misc configuration (early)", "[ INFO ] Stage: Package installation", "[ INFO ] Stage: Misc configuration", "[ INFO ] Upgrading CA", "[ INFO ] Initializing PostgreSQL", "[ INFO ] Creating PostgreSQL 'engine' database", "[ INFO ] Configuring PostgreSQL", "[ INFO ] Creating PostgreSQL 'ovirt_engine_history' database", "[ INFO ] Configuring PostgreSQL", "[ INFO ] Creating CA: /etc/pki/ovirt-engine/ca.pem", "[ INFO ] Creating CA: /etc/pki/ovirt-engine/qemu-ca.pem", "[ INFO ] Updating OVN SSL configuration", "[ INFO ] Creating/refreshing DWH database schema", "[ ERROR ] Failed to execute stage 'Misc configuration': Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute", "[ INFO ] Rolling back DWH database schema", "[ INFO ] Clearing DWH database ovirt_engine_history", "[ INFO ] Stage: Clean up", " Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20200320114633-ujxnbe.log", "[ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200320114653-setup.conf'", "[ INFO ] Stage: Pre-termination", "[ INFO ] Stage: Termination", "[ ERROR ] Execution of setup failed"]}


Try to test by manual and automatic for several times, QE got the same results.

Comment 4 Wei Wang 2020-03-20 04:14:14 UTC
Created attachment 1671689 [details]
picture

Comment 5 Wei Wang 2020-03-20 04:30:37 UTC
Created attachment 1671691 [details]
log files

Comment 6 Wei Wang 2020-03-20 06:55:59 UTC
(In reply to Yedidyah Bar David from comment #2)
> (In reply to Wei Wang from comment #1)
> > The new rhvm-appliance for 4.4 is coming, I will retest this bug later and
> > give the further update.
> 
> Thanks, please do.
> 
> That said, the failure is simply due to the fact that adding the host took
> too much time. We allow up to 10 minutes, and [1] (in the attached logs)
> starts with:
> 
> 2020-03-19 12:34:12 CST - TASK [Gathering Facts]
> *********************************************************
> 
> And ends (cut in the middle) with:
> 
> 2020-03-19 12:44:57 CST - TASK [ovirt-host-deploy-vnc-certificates : Setup
> VNC PKI] **********************
> 
> So:
> 
> 1. It might be wise to simply allow more time, for now. That's a trivial
> change:
> 
> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/commit/
> 11fd4397d44b56d7f696016b772791efad5b1aa3
> 
> 2. Perhaps try to understand why it took so long. E.g., can you try adding
> same host (perhaps with reprovisioned image of RHVH) to a standalone engine,
> and attach the host-deploy log? Skimming through the file, I see e.g.:
> 
Adding same host to a standalone engine, host is up successfully.
Attach the /var/log into attachment.

> 2020-03-19 12:37:50 CST - TASK [ovirt-host-deploy-vdsm-certificates : Add
> vdsm cacert files] *************
> 2020-03-19 12:38:29 CST - 
> 2020-03-19 12:38:29 CST - {
>   "status" : "OK",
> 
> Meaning, copying 3 small files took 40 seconds. I think that's too much.
> 
> [1]
> log/ovirt-hosted-engine-setup/engine-logs-2020-03-19T04\:23\:56Z/ovirt-
> engine/host-deploy/ovirt-host-deploy-ansible-20200319123407-hp-dl388g9-04.
> lab.eng.pek2.redhat.com-3eeecdb2.log

Comment 7 Wei Wang 2020-03-20 06:56:54 UTC
Created attachment 1671707 [details]
host_add_to_standalone_engine log files

Comment 8 Yedidyah Bar David 2020-03-22 08:32:47 UTC
(In reply to Wei Wang from comment #3)
> Test Version:
> RHVH-UNSIGNED-ISO-4.4-RHEL-8-20200318.0-RHVH-x86_64-dvd1
> cockpit-system-211.3-1.el8.noarch
> cockpit-ws-211.3-1.el8.x86_64
> cockpit-ovirt-dashboard-0.14.2-1.el8ev.noarch
> cockpit-211.3-1.el8.x86_64
> cockpit-bridge-211.3-1.el8.x86_64
> cockpit-dashboard-211.3-1.el8.noarch
> cockpit-storaged-211.3-1.el8.noarch
> ovirt-ansible-engine-setup-1.2.2-1.el8ev.noarch
> ovirt-ansible-hosted-engine-setup-1.1.0-1.el8ev.noarch
> rhvm-appliance-4.4-20200319.0.el8ev.x86_64

Thanks for testing.

> 
> Test Steps:
> 1. Clean install RHVH-UNSIGNED-ISO-4.4-RHEL-8-20200318.0-RHVH-x86_64-dvd1
> 2. Yum install rhvm-appliance-4.4-20200319.0.el8ev.x86_64
> 3. Deploy hosted engine via cockpit
> 
> Result:
> Failed at ansible task "ovirt.engine-setup : Run engine-setup with
> answerfile"
> 
> Please refer the picture "he_0320.png"
> [INFO ] TASK [ovirt.engine-setup : Run engine-setup with answerfile]
> [ ERROR ] fatal: [localhost ->
> rhevh-hostedengine-vm-06.lab.eng.pek2.redhat.com]: FAILED! => {"changed":
> true, "cmd": ["engine-setup", "--accept-defaults",
> "--config-append=/root/ovirt-engine-answers", "--offline"], "delta":
> "0:00:26.581864", "end": "2020-03-20 11:46:54.101298", "msg": "non-zero
> return code", "rc": 1, "start": "2020-03-20 11:46:27.519434", "stderr":
> "\nERROR: Python 2 is disabled in RHEL8.\n\n

I failed to find what emits this error. I am just guessing it's something
inside the appliance installed as "/usr/bin/python" that does that. Might
check the appliance content later on, but not sure it's that important.

[reset of error message is snipped]
> \n\nFatal Python error: Python 2 is
> disabled\n/usr/share/otopi/otopi-functions: line 47:

This line was introduced for fixing bug 1751324, to make otopi not use python3
on EL7. I see basically two options:

1. Revert the patch for that bug and declare otopi to not support el7 anymore.
This is reasonable, as it's not used anymore there anyway.

2. Rewrite that patch to not use python at all (any version) but source and
use content from /etc/os-release.

Moving this bug, for now.

Re comment 6 - if you still have the environment, please attach also /var/log/ovirt-engine
from the engine. Thanks.

Comment 9 Wei Wang 2020-03-24 07:58:01 UTC
(In reply to Yedidyah Bar David from comment #8)
> (In reply to Wei Wang from comment #3)
> > Test Version:
> > RHVH-UNSIGNED-ISO-4.4-RHEL-8-20200318.0-RHVH-x86_64-dvd1
> > cockpit-system-211.3-1.el8.noarch
> > cockpit-ws-211.3-1.el8.x86_64
> > cockpit-ovirt-dashboard-0.14.2-1.el8ev.noarch
> > cockpit-211.3-1.el8.x86_64
> > cockpit-bridge-211.3-1.el8.x86_64
> > cockpit-dashboard-211.3-1.el8.noarch
> > cockpit-storaged-211.3-1.el8.noarch
> > ovirt-ansible-engine-setup-1.2.2-1.el8ev.noarch
> > ovirt-ansible-hosted-engine-setup-1.1.0-1.el8ev.noarch
> > rhvm-appliance-4.4-20200319.0.el8ev.x86_64
> 
> Thanks for testing.
> 
> > 
> > Test Steps:
> > 1. Clean install RHVH-UNSIGNED-ISO-4.4-RHEL-8-20200318.0-RHVH-x86_64-dvd1
> > 2. Yum install rhvm-appliance-4.4-20200319.0.el8ev.x86_64
> > 3. Deploy hosted engine via cockpit
> > 
> > Result:
> > Failed at ansible task "ovirt.engine-setup : Run engine-setup with
> > answerfile"
> > 
> > Please refer the picture "he_0320.png"
> > [INFO ] TASK [ovirt.engine-setup : Run engine-setup with answerfile]
> > [ ERROR ] fatal: [localhost ->
> > rhevh-hostedengine-vm-06.lab.eng.pek2.redhat.com]: FAILED! => {"changed":
> > true, "cmd": ["engine-setup", "--accept-defaults",
> > "--config-append=/root/ovirt-engine-answers", "--offline"], "delta":
> > "0:00:26.581864", "end": "2020-03-20 11:46:54.101298", "msg": "non-zero
> > return code", "rc": 1, "start": "2020-03-20 11:46:27.519434", "stderr":
> > "\nERROR: Python 2 is disabled in RHEL8.\n\n
> 
> I failed to find what emits this error. I am just guessing it's something
> inside the appliance installed as "/usr/bin/python" that does that. Might
> check the appliance content later on, but not sure it's that important.
> 
> [reset of error message is snipped]
> > \n\nFatal Python error: Python 2 is
> > disabled\n/usr/share/otopi/otopi-functions: line 47:
> 
> This line was introduced for fixing bug 1751324, to make otopi not use
> python3
> on EL7. I see basically two options:
> 
> 1. Revert the patch for that bug and declare otopi to not support el7
> anymore.
> This is reasonable, as it's not used anymore there anyway.
> 
> 2. Rewrite that patch to not use python at all (any version) but source and
> use content from /etc/os-release.
> 
> Moving this bug, for now.
> 
> Re comment 6 - if you still have the environment, please attach also
> /var/log/ovirt-engine
> from the engine. Thanks.
The environment is destroyed, I will setup it again later.

Comment 10 Wei Wang 2020-03-24 08:44:40 UTC
Created attachment 1672967 [details]
engine log

Reply comment #8, attach the /var/log/ovirt-engine logs in engine.

Comment 11 Wei Wang 2020-03-26 10:03:38 UTC
Verify testing for comment #3
Version:
RHVH-4.4-20200325.0-RHVH-x86_64-dvd1.iso
cockpit-system-211.3-1.el8.noarch
cockpit-ws-211.3-1.el8.x86_64
cockpit-ovirt-dashboard-0.14.3-1.el8ev.noarch
cockpit-211.3-1.el8.x86_64
cockpit-bridge-211.3-1.el8.x86_64
cockpit-dashboard-211.3-1.el8.noarch
cockpit-storaged-211.3-1.el8.noarch
ovirt-hosted-engine-ha-2.4.2-1.el8ev.noarch
ovirt-hosted-engine-setup-2.4.3-2.el8ev.noarch
rhvm-appliance-4.4-20200325.0.el8ev.x86_64

Result:
Run ansible task "ovirt.engine-setup : Run engine-setup with answerfile" successfully during hosted engine deployment.


Hosted engine deployment still failed at comment #0 phenomenon. Report a new bug https://bugzilla.redhat.com/show_bug.cgi?id=1817402 to trace.
And QE will move this to "VERIFIED" until dev give the "ON_QA" status.

Comment 12 Yedidyah Bar David 2020-03-26 10:41:39 UTC
(In reply to Wei Wang from comment #11)
> Verify testing for comment #3
> Version:
> RHVH-4.4-20200325.0-RHVH-x86_64-dvd1.iso
> cockpit-system-211.3-1.el8.noarch
> cockpit-ws-211.3-1.el8.x86_64
> cockpit-ovirt-dashboard-0.14.3-1.el8ev.noarch
> cockpit-211.3-1.el8.x86_64
> cockpit-bridge-211.3-1.el8.x86_64
> cockpit-dashboard-211.3-1.el8.noarch
> cockpit-storaged-211.3-1.el8.noarch
> ovirt-hosted-engine-ha-2.4.2-1.el8ev.noarch
> ovirt-hosted-engine-setup-2.4.3-2.el8ev.noarch
> rhvm-appliance-4.4-20200325.0.el8ev.x86_64
> 
> Result:
> Run ansible task "ovirt.engine-setup : Run engine-setup with answerfile"
> successfully during hosted engine deployment.

That's very weird. We didn't yet build otopi for RHV.

So rhvm-appliance-4.4-20200319.0.el8ev.x86_64 was broken, and rhvm-appliance-4.4-20200325.0.el8ev.x86_64 is fine? It must be something else that broke/fixed, perhaps something in python.

> 
> 
> Hosted engine deployment still failed at comment #0 phenomenon. Report a new
> bug https://bugzilla.redhat.com/show_bug.cgi?id=1817402 to trace.
> And QE will move this to "VERIFIED" until dev give the "ON_QA" status.

Thanks.

Comment 13 Sandro Bonazzola 2020-05-20 20:00:20 UTC
This bugzilla is included in oVirt 4.4.0 release, published on May 20th 2020.

Since the problem described in this bug report should be
resolved in oVirt 4.4.0 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.