Bug 1541233

Summary: bad "before" or "after" parameters in _misc_configure_ovn_pki
Product: [oVirt] ovirt-engine Reporter: Yihui Zhao <yzhao>
Component: Setup.EngineAssignee: Sandro Bonazzola <sbonazzo>
Status: CLOSED CURRENTRELEASE QA Contact: Pavel Stehlik <pstehlik>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.2.1.4CC: bugs, cshao, dguo, didi, huzhao, nsednev, phbailey, qiyuan, rbarry, sbonazzo, weiwang, yaniwang, ycui, yisong
Target Milestone: ovirt-4.2.1Flags: rule-engine: ovirt-4.2+
rule-engine: blocker+
sbonazzo: devel_ack+
rule-engine: testing_ack+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-12 11:56:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1529119    
Bug Blocks: 1455169    
Attachments:
Description Flags
failed.png none

Description Yihui Zhao 2018-02-02 03:44:29 UTC
Created attachment 1389910 [details]
failed.png

Description of problem: 
Deploy SHE failed with appliance(rhvm-appliance-4.2-20180201.0.el7.noarch) via cockpit

Failed to execute stage "Environment Setup or Clean up" while deploy HE via cockpit. 
From the cockpit,
""""
hosted-engine --add-console-password
|- [ INFO ] Stage: Initializing
|- [ INFO ] Stage: Environment setup
|- Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/root/ovirt-engine-answers', '/root/heanswers.conf']
|- Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20180201215426-upqnxj.log
|- Version: otopi-1.7.7 (otopi-1.7.7-1.el7ev)
|- [ ERROR ] "before" parameter of method otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin._misc_configure_ovn_pki is a string, should probably be a tuple. Perhaps a missing comma?
|- methodinfo: {'priority': 5000, 'name': None, 'before': 'osetup.ovn.provider.service.restart', 'after': ('osetup.pki.ca.available', 'osetup.ovn.services.restart'), 'method': <bound method ?._misc_configure_ovn_pki of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x3432350>>, 'condition': <bound method ?.<lambda> of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x3432350>>, 'stage': 11}
|- [ ERROR ] "before" parameter of method otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin._misc_configure_provider is a string, should probably be a tuple. Perhaps a missing comma?
|- methodinfo: {'priority': 5000, 'name': None, 'before': 'osetup.ovn.provider.service.restart', 'after': ('osetup.pki.ca.available', 'osetup.ovn.services.restart'), 'method': <bound method ?._misc_configure_provider of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x3432350>>, 'condition': <bound method ?.<lambda> of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x3432350>>, 'stage': 11}
|- [ ERROR ] Failed to execute stage 'Environment setup': Found bad "before" or "after" parameters
|- [ INFO ] Stage: Clean up
|- Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20180201215426-upqnxj.log
|- [ ERROR ] Failed to execute stage 'Clean up': 'NoneType' object has no attribute 'cleanup'
|- [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20180201215427-setup.conf'
|- [ INFO ] Stage: Pre-termination
|- [ INFO ] Stage: Termination
|- [ ERROR ] Execution of setup failed
|- HE_APPLIANCE_ENGINE_SETUP_FAIL
Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180201214638-7scwu2.log
"""""

Failed errors:
"""
 Engine setup failed on the appliance
 Failed to execute stage 'Closing up': Engine setup failed on the appliance Please check its log on the appliance.
 Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.
"""""

Version-Release number of selected component (if applicable): 
cockpit-ws-157-1.el7.x86_64
cockpit-bridge-157-1.el7.x86_64
cockpit-storaged-157-1.el7.noarch
cockpit-dashboard-157-1.el7.x86_64
cockpit-157-1.el7.x86_64
cockpit-ovirt-dashboard-0.11.9-0.1.el7ev.noarch
cockpit-system-157-1.el7.noarch
ovirt-hosted-engine-setup-2.2.9-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.4-1.el7ev.noarch
rhvh-4.2.1.2-0.20180201.0+1
rhvm-appliance-4.2-20180201.0.el7.noarch

How reproducible: 
100% 


Steps to Reproduce: 
1. Clean install latest RHVH4.2.1 with ks(rhvh-4.2.1.2-0.20180201.0+1)
2. Deploy HE via cockpit with nfsv3 based otopi deployment


Actual results: 
1. After step2, deploy HE failed with these errors:
 Engine setup failed on the appliance
 Failed to execute stage 'Closing up': Engine setup failed on the appliance Please check its log on the appliance.
 Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.

Expected results: 
After step2, deploy HE via cockpit successfully based otopi deployment.

Additional info: 
Deploy SHE successfully with appliance(rhvm-appliance-4.2-20180125.0.el7.noarch) via cockpit

Comment 1 Simone Tiraboschi 2018-02-05 08:10:49 UTC
*** Bug 1541759 has been marked as a duplicate of this bug. ***

Comment 2 Yihui Zhao 2018-02-05 08:32:41 UTC
Deploy SHE successfully  with appliance(rhvm-appliance-4.2-20180202.0.el7.noarch) via cockpit

Test version:
cockpit-ws-157-1.el7.x86_64
cockpit-bridge-157-1.el7.x86_64
cockpit-storaged-157-1.el7.noarch
cockpit-dashboard-157-1.el7.x86_64
cockpit-157-1.el7.x86_64
cockpit-ovirt-dashboard-0.11.10-0.1.el7ev.noarch
cockpit-system-157-1.el7.noarch
ovirt-hosted-engine-setup-2.2.9-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.4-1.el7ev.noarch
rhvh-4.2.1.2-0.20180202.0+1
rhvm-appliance-4.2-20180202.0.el7.noarch


So, change the bug's status to verified!

Comment 3 Nikolai Sednev 2018-02-05 14:19:14 UTC
This bug still being reproduced on RHEL7.4:
[ ERROR ] fatal: [nsednev-he-4.scl.lab.tlv.redhat.com]: FAILED! => {"changed": true, "cmd": ["/usr/bin/engine-setup", "--offline", "--config-append=/root/ovirt-engine-answers", "--config-append=/root/heanswers.conf"], "delta": "0:00:01.653135", "end": "2018-02-05 16:00:08.791552", "msg": "non-zero return code", "rc": 1, "start": "2018-02-05 16:00:07.138417", "stderr": "", "stderr_lines": [], "stdout": "[ INFO  ] Stage: Initializing\n[ INFO  ] Stage: Environment setup\n          Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/root/ovirt-engine-answers', '/root/heanswers.conf']\n          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20180205160008-hor1y2.log\n          Version: otopi-1.7.7 (otopi-1.7.7-1.el7ev)\n[ ERROR ] \"before\" parameter of method otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin._misc_configure_ovn_pki is a string, should probably be a tuple. Perhaps a missing comma?\n          methodinfo: {'priority': 5000, 'name': None, 'before': 'osetup.ovn.provider.service.restart', 'after': ('osetup.pki.ca.available', 'osetup.ovn.services.restart'), 'method': <bound method ?._misc_configure_ovn_pki of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x26cd490>>, 'condition': <bound method ?.<lambda> of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x26cd490>>, 'stage': 11}\n[ ERROR ] \"before\" parameter of method otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin._misc_configure_provider is a string, should probably be a tuple. Perhaps a missing comma?\n          methodinfo: {'priority': 5000, 'name': None, 'before': 'osetup.ovn.provider.service.restart', 'after': ('osetup.pki.ca.available', 'osetup.ovn.services.restart'), 'method': <bound method ?._misc_configure_provider of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x26cd490>>, 'condition': <bound method ?.<lambda> of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x26cd490>>, 'stage': 11}\n[ ERROR ] Failed to execute stage 'Environment setup': Found bad \"before\" or \"after\" parameters\n[ INFO  ] Stage: Clean up\n          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20180205160008-hor1y2.log\n[ ERROR ] Failed to execute stage 'Clean up': 'NoneType' object has no attribute 'cleanup'\n[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20180205160008-setup.conf'\n[ INFO  ] Stage: Pre-termination\n[ INFO  ] Stage: Termination\n[ ERROR ] Execution of setup failed", "stdout_lines": ["[ INFO  ] Stage: Initializing", "[ INFO  ] Stage: Environment setup", "          Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/root/ovirt-engine-answers', '/root/heanswers.conf']", "          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20180205160008-hor1y2.log", "          Version: otopi-1.7.7 (otopi-1.7.7-1.el7ev)", "[ ERROR ] \"before\" parameter of method otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin._misc_configure_ovn_pki is a string, should probably be a tuple. Perhaps a missing comma?", "          methodinfo: {'priority': 5000, 'name': None, 'before': 'osetup.ovn.provider.service.restart', 'after': ('osetup.pki.ca.available', 'osetup.ovn.services.restart'), 'method': <bound method ?._misc_configure_ovn_pki of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x26cd490>>, 'condition': <bound method ?.<lambda> of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x26cd490>>, 'stage': 11}", "[ ERROR ] \"before\" parameter of method otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin._misc_configure_provider is a string, should probably be a tuple. Perhaps a missing comma?", "          methodinfo: {'priority': 5000, 'name': None, 'before': 'osetup.ovn.provider.service.restart', 'after': ('osetup.pki.ca.available', 'osetup.ovn.services.restart'), 'method': <bound method ?._misc_configure_provider of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x26cd490>>, 'condition': <bound method ?.<lambda> of <otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin object at 0x26cd490>>, 'stage': 11}", "[ ERROR ] Failed to execute stage 'Environment setup': Found bad \"before\" or \"after\" parameters", "[ INFO  ] Stage: Clean up", "          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20180205160008-hor1y2.log", "[ ERROR ] Failed to execute stage 'Clean up': 'NoneType' object has no attribute 'cleanup'", "[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20180205160008-setup.conf'", "[ INFO  ] Stage: Pre-termination", "[ INFO  ] Stage: Termination", "[ ERROR ] Execution of setup failed"]}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
[ INFO  ] TASK [Gathering Facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Remove local vm dir]
[ INFO  ] ok: [localhost]
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180205160016.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180205155050-9x79it.log


Tested using CLI on RHEL7.4 with these components:
ovirt-hosted-engine-setup-2.2.9-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.4-1.el7ev.noarch
rhvm-appliance-4.2-20180201.0.el7.noarch
cockpit-ovirt-dashboard-0.11.9-0.1.el7ev.noarch
Red Hat Enterprise Linux Server release 7.4 (Maipo)

Comment 4 Yedidyah Bar David 2018-02-05 14:27:58 UTC
(In reply to Nikolai Sednev from comment #3)
> This bug still being reproduced on RHEL7.4:
[snip]
> Tested using CLI on RHEL7.4 with these components:
> ovirt-hosted-engine-setup-2.2.9-1.el7ev.noarch
> ovirt-hosted-engine-ha-2.2.4-1.el7ev.noarch
> rhvm-appliance-4.2-20180201.0.el7.noarch

Comment 2 says it was verified with: 
rhvm-appliance-4.2-20180202.0.el7.noarch

> cockpit-ovirt-dashboard-0.11.9-0.1.el7ev.noarch
> Red Hat Enterprise Linux Server release 7.4 (Maipo)

Comment 5 Yedidyah Bar David 2018-02-05 14:38:58 UTC
BTW, if you see similar errors again, please open bugs with subject mentioning the method name, in this case:

otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin._misc_configure_ovn_pki

or some unique suffix if it's very long. Each such failure is a separate bug, unrelated to each other. Thanks.

Mentioning this also because sadly we had two different bugs in the same source file, one about _misc_configure_ovn_pki and one about _misc_configure_provider.

Comment 6 Sandro Bonazzola 2018-02-12 08:22:15 UTC
*** Bug 1541771 has been marked as a duplicate of this bug. ***

Comment 7 Sandro Bonazzola 2018-02-12 11:56:43 UTC
This bugzilla is included in oVirt 4.2.1 release, published on Feb 12th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.