Bug 1403854

Summary: Failed to "hosted-engine --upgrade-appliance" with Failed to execute stage 'Setup validation': 'OVEHOSTED_NETWORK/fqdnReverseValidation' error.
Product: [oVirt] ovirt-hosted-engine-setup Reporter: Nikolai Sednev <nsednev>
Component: ToolsAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED CURRENTRELEASE QA Contact: Nikolai Sednev <nsednev>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 2.1.0CC: bugs, mavital, stirabos
Target Milestone: ovirt-4.1.0-betaKeywords: Regression, Triaged
Target Release: 2.1.0Flags: rule-engine: ovirt-4.1+
gklein: blocker+
rule-engine: planning_ack+
rule-engine: devel_ack+
mavital: testing_ack+
Hardware: x86_64   
OS: Linux   
Whiteboard: integration
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-15 14:52:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1394740    
Attachments:
Description Flags
ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20161212152055-02ep5d.log none

Description Nikolai Sednev 2016-12-12 13:45:43 UTC
Description of problem:
Failed to "hosted-engine --upgrade-appliance" with Failed to execute stage 'Setup validation': 'OVEHOSTED_NETWORK/fqdnReverseValidation' error.

Version-Release number of selected component (if applicable):
Engine:
ovirt-engine-dwh-setup-4.1.0-0.0.master.20161129154019.el7.centos.noarch
ovirt-engine-lib-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-vmconsole-proxy-helper-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
python-ovirt-engine-sdk4-4.1.0-0.1.a0.20161208gitab2baf0.el7.centos.x86_64
ovirt-engine-dbscripts-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-wildfly-10.1.0-1.el7.x86_64
ovirt-engine-wildfly-overlay-10.0.0-1.el7.noarch
ovirt-engine-cli-3.6.9.2-1.el7.centos.noarch
ovirt-engine-hosts-ansible-inventory-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-tools-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-restapi-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-dwh-4.1.0-0.0.master.20161129154019.el7.centos.noarch
ovirt-engine-setup-plugin-ovirt-engine-common-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-setup-plugin-websocket-proxy-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-websocket-proxy-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-extensions-api-impl-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-sdk-python-3.6.9.2-0.1.20161204.gite99bbd1.el7.centos.noarch
ovirt-engine-backend-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-dashboard-1.1.0-0.4.20161128git5ed6f96.el7.centos.noarch
ovirt-engine-setup-base-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-setup-plugin-ovirt-engine-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-setup-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-tools-backup-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-webadmin-portal-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-userportal-4.1.0-0.2.master.20161210231201.git26a385e.el7.centos.noarch
ovirt-engine-extension-aaa-jdbc-1.1.3-0.0.master.20161118164738.gitd0ff686.el7.noarch
Linux version 3.10.0-327.36.3.el7.x86_64 (builder.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Mon Oct 24 16:09:20 UTC 2016
Linux 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Hosts:
ovirt-engine-appliance-4.1-20161202.1.el7.centos.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
ovirt-release41-pre-4.1.0-0.0.beta.20161201085255.git731841c.el7.centos.noarch
ovirt-hosted-engine-setup-2.1.0-0.0.master.20161130101611.gitb3ad261.el7.centos.noarch
libvirt-client-2.0.0-10.el7_3.2.x86_64
ovirt-engine-appliance-4.1-20161202.1.el7.centos.noarch
ovirt-host-deploy-1.6.0-0.0.master.20161107121647.gitfd7ddcd.el7.centos.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-imageio-daemon-0.5.0-0.201611201242.gitb02532b.el7.centos.noarch
ovirt-hosted-engine-ha-2.1.0-0.0.master.20161130135331.20161130135328.git3541725.el7.centos.noarch
ovirt-setup-lib-1.1.0-0.0.master.20161107100014.gitb73abeb.el7.centos.noarch
ovirt-imageio-common-0.5.0-0.201611201242.gitb02532b.el7.centos.noarch
vdsm-4.18.999-1020.git1ff41b1.el7.centos.x86_64
sanlock-3.4.0-1.el7.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.2.x86_64
ovirt-vmconsole-1.0.4-1.el7ev.noarch
mom-0.5.8-1.el7ev.noarch
Linux version 3.10.0-514.2.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Nov 16 13:15:13 EST 2016
Linux 3.10.0-514.2.2.el7.x86_64 #1 SMP Wed Nov 16 13:15:13 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)


How reproducible:
100%

Steps to Reproduce:
1.Get latest 4.1 HE installed over NFS on at least one RHEL7.3 host.
2.Add 2 data storage domains to your engine.
3.Make a backup db for the engine and copy it to none root folder on your host.
4.Make sure you have the appliance downloaded and installed on your host.
5.Run "hosted-engine --upgrade-appliance" on host, while its in global maintenance and provide the process with engine's db backup file, which is not in root folder.

Actual results:
I've got failed backup procedure with "[ ERROR ] Failed to execute stage 'Setup validation': 'OVEHOSTED_NETWORK/fqdnReverseValidation'"

Expected results:
Engine's upgrade should not fail.

Additional info:
Sosreport from host is attached.

Comment 1 Nikolai Sednev 2016-12-12 13:47:07 UTC
Created attachment 1230797 [details]
ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20161212152055-02ep5d.log

Comment 2 Nikolai Sednev 2016-12-12 13:51:02 UTC
Attaching link to host's sosreport here:
https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88SFNHXy1idmhKc2M/view?usp=sharing

Comment 3 Red Hat Bugzilla Rules Engine 2016-12-12 13:52:10 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 4 Red Hat Bugzilla Rules Engine 2016-12-12 13:54:39 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 5 Simone Tiraboschi 2016-12-13 09:59:15 UTC
The regression got introduced here: https://gerrit.ovirt.org/#/c/64472/
it affects just the appliance upgrade flow on 4.1

Comment 7 Nikolai Sednev 2017-02-07 19:06:34 UTC
Not being reproduced on these components on host:
rhvm-appliance-4.1.20170126.0-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-hosted-engine-ha-2.1.0.1-1.el7ev.noarch
ovirt-hosted-engine-setup-2.1.0.1-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-host-deploy-1.6.0-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-node-ng-nodectl-4.1.0-0.20170104.1.el7.noarch
libvirt-client-2.0.0-10.el7_3.4.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.3.x86_64
vdsm-4.19.4-1.el7ev.x86_64
sanlock-3.4.0-1.el7.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
mom-0.5.8-1.el7ev.noarch
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 7.3