Bug 1287136

Summary: Failed to upgrade DC compatibility version from 3.4 to 3.5 after upgrading hosts
Product: [oVirt] ovirt-engine Reporter: Kevin Alon Goldblatt <kgoldbla>
Component: BLL.InfraAssignee: Piotr Kliczewski <pkliczew>
Status: CLOSED CURRENTRELEASE QA Contact: Jiri Belka <jbelka>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.6.1CC: amureini, bugs, gklein, kgoldbla, nsednev, oourfali, pkliczew
Target Milestone: ovirt-3.6.1Keywords: Triaged
Target Release: 3.6.1.2Flags: gklein: ovirt-3.6.z?
gklein: blocker?
kgoldbla: planning_ack?
oourfali: devel_ack+
pstehlik: testing_ack+
Hardware: x86_64   
OS: Linux   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-13 14:35:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1285700    
Attachments:
Description Flags
engine, vdsm, server logs none

Description Kevin Alon Goldblatt 2015-12-01 15:18:41 UTC
Description of problem:
Failed to upgrade DC compatibility version from 3.4 to 3.5 after upgrading hosts from 3.4 to 3.5

Version-Release number of selected component (if applicable):
rhevm-3.6.1-0.2.el6.noarch
vdsm-4.16.30-1.el7ev.x86_64 - host1
vdsm-4.16.27-1.el7ev.x86_64 - host2
vdsm-4.16.30-1.el7ev.x86_64 - host3

How reproducible:
Happens every time 


Steps to Reproduce:
1. Installed Engine with RHEVM3.4 (3.4.5-0.3)
2. Upgrade engine to RHEVM3.5 (3.5.6.2-0.1)  
3. Install 3.4 on 2 rhel7.2 hosts and 3.5 on 1 rhevh host
4. Add hosts to the 3.5 Engine to a DC with compatibility ver 3.5 and a Cluster with compatibility version 3.4
5. Created several SD's including FC, iscsi, nfs and gluster
6. Upgrade engine to Rhevm3.6 (3.6.1)
6. Upgrade rhel7.2 hosts from 3.4 to 3.5
7. When attempting to upgrade the compatibility version of the DC fron 3.4 to 3.5 an error is generated and the upgrade fails >>>>> "Operation Canceled
Error while executing action: Some of the host still use xmlrpc protocol which is not supported by cluster 3.6 or higher "

Actual results:
The Cluster compatibility version upgrade from 3.4 to 3.5 fails

Expected results:
The Cluster compatibility version upgrade should succeed


Additional info:
Engine.log
----------------------------
First attempt:

2015-12-01 15:51:56,580 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) [53193265] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='ERROR', powerStatus='UNKNOWN'
, message='Failed: Unable to obtain correct plug status or plug is not available


'}, log id: 6aafc90a
2015-12-01 15:51:56,644 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [53193265] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Execution of powe
r management status on Host seal12.qa.lab.tlv.redhat.com using Proxy Host seal08.qa.lab.tlv.redhat.com and Fence Agent ipmilan:seal12.qa.lab.tlv.redhat.com failed.
2015-12-01 15:51:56,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-53) [53193265] START, SetMOMPolicyParametersVDSCommand(HostName = seal12.qa.lab.tlv.redhat.com, 
MomPolicyVDSParameters:{runAsync='true', hostId='ee97bdec-9ed8-4f10-afd1-912154d984b8'}), log id: 32796b57
2015-12-01 15:51:56,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-53) [53193265] FINISH, SetMOMPolicyParametersVDSCommand, log id: 32796b57
2015-12-01 15:51:56,796 INFO  [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Running command: RefreshHostDevicesCommand internal: true. Entities affected :  ID: ee97bdec-9ed
8-4f10-afd1-912154d984b8 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2015-12-01 15:51:56,861 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host se
al12.qa.lab.tlv.redhat.com was set to Up.
2015-12-01 15:51:56,916 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management 
test failed for Host seal12.qa.lab.tlv.redhat.com.Failed: Unable to obtain correct plug status or plug is not available



2015-12-01 15:51:57,008 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Correlation ID: 11c887e2, Call Stack: null, Custom Event ID: -1, Message: Failed to ver
ify Host seal12.qa.lab.tlv.redhat.com power management.
2015-12-01 15:51:57,167 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-53) [62375468] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: ee97bdec-9ed8-4f10-afd1-
912154d984b8 Type: VDS
2015-12-01 15:52:59,113 WARN  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-5) [bf0fee8] CanDoAction of action 'UpdateVdsGroup' failed for user vdcadmin.tlv.redhat.com. Reasons: VAR__TYPE__CLUSTER,VAR__
ACTION__UPDATE,VDS_GROUP_CANNOT_UPDATE_COMPATIBILITY_VERSION_WITH_LOWER_HOSTS
2015-12-01 15:57:18,910 WARN  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-5) [3fef8609] CanDoAction of action 'UpdateVdsGroup' failed for user vdcadmin.tlv.redhat.com. Reasons: VAR__TYPE__CLUSTER,VAR_
_ACTION__UPDATE,ACTION_TYPE_FAILED_WRONG_PROTOCOL_FOR_CLUSTER_VERSION
2015-12-01 16:16:56,739 INFO  [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) [] Running command: LoginAdminUserCommand internal: false.
2015-12-01 16:16:56,831 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in.
2015-12-01 16:18:26,793 INFO  [org.ovirt.engine.core.bll.hostdeploy.UpdateVdsCommand] (ajp-/127.0.0.1:8702-10) [7e5fa5bb] Running command: UpdateVdsCommand internal: false. Entities affected :  ID: dfd79527-b192-4324-b09c-2d63376040e5
 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN
2015-12-01 16:18:27,069 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management c
onfiguration for Host orchid-vds1.qa.lab.tlv.redhat.com.




----------------------------------------------------------------------------

2nd attempt:


2015-12-01 16:55:25,546 INFO  [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (org.ovirt.thread.pool-7-thread-29) [4e5dabbe] Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' refresh
2015-12-01 16:55:25,607 INFO  [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-29) [4e5dabbe] SSH execute 'root.128.15' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True'
2015-12-01 16:55:26,644 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Initializing.
2015-12-01 16:55:26,728 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Environment setup.
2015-12-01 16:55:26,803 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Environment packages setup.
2015-12-01 16:58:10,145 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Programs detection.
2015-12-01 16:58:10,186 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Environment customization.
2015-12-01 17:00:53,595 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Kdump supported.
2015-12-01 17:00:53,687 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Logs at host located at: '/tmp/ovirt-host-deploy-20151201165524-i10pig.log'.
2015-12-01 17:00:53,739 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployVdsmUnit] (VdsDeploy) [4e5dabbe] Host 10.35.128.15 reports unique id 0e17ce27-ae1c-401b-aea9-6ab451fc9bb9
2015-12-01 17:00:53,787 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployVdsmUnit] (VdsDeploy) [4e5dabbe] Assigning unique id 0e17ce27-ae1c-401b-aea9-6ab451fc9bb9 to Host 10.35.128.15
2015-12-01 17:00:53,904 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Disabling Kdump integration.
2015-12-01 17:00:54,029 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Setup validation.
2015-12-01 17:01:14,096 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Hardware supports virtualization.
2015-12-01 17:01:14,288 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Transaction setup.
2015-12-01 17:01:14,313 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Misc configuration.
2015-12-01 17:01:14,363 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Package installation.
2015-12-01 17:01:49,409 WARN  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-7) [32ca59bc] CanDoAction of action 'UpdateVdsGroup' failed for user vdcadmin.tlv.redhat.com. Reasons: VAR__TYPE__CLUSTER,VAR__ACTION__UPDATE,ACTION_TYPE_FAILED_WRONG_PROTOCOL_FOR_CLUSTER_VERSION
(END)

Comment 1 Kevin Alon Goldblatt 2015-12-01 15:28:00 UTC
Created attachment 1100940 [details]
engine, vdsm, server logs

added logs

Comment 2 Gil Klein 2015-12-01 16:21:02 UTC
*** Bug 1287134 has been marked as a duplicate of this bug. ***

Comment 12 Sandro Bonazzola 2015-12-10 12:42:14 UTC
This bug is referenced in ovirt-engine 3.6.1.2 git log.
Please check if the BZ is ready to be moved to QE and set target release accordingly.

Comment 13 Nikolai Sednev 2015-12-30 14:48:23 UTC
Works for me on these components:
Engine:
ovirt-host-deploy-java-1.4.1-1.el6ev.noarch
rhevm-3.6.1.3-0.1.el6.noarch
ovirt-vmconsole-proxy-1.0.0-1.el6ev.noarch
ovirt-engine-extension-aaa-jdbc-1.0.4-1.el6ev.noarch
ovirt-vmconsole-1.0.0-1.el6ev.noarch
ovirt-host-deploy-1.4.1-1.el6ev.noarch
Linux version 2.6.32-573.8.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Fri Sep 25 19:24:22 EDT 2015

Host:
vdsm-4.17.15-0.el7ev.noarch
sanlock-3.2.4-2.el7_2.x86_64
mom-0.5.1-1.el7ev.noarch
ovirt-vmconsole-1.0.0-1.el7ev.noarch
ovirt-vmconsole-host-1.0.0-1.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.2.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.5.x86_64
Linux version 3.10.0-327.4.4.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Dec 17 15:51:24 EST 2015

I was unable to reproduce this error and my host cluster compatibility version changed successfully 3.4->3.5->3.6, as well as DC compatibility version 3.4->3.5->3.6.

Comment 14 Sandro Bonazzola 2016-01-13 14:35:14 UTC
oVirt 3.6.1 has been released, closing current release