Bug 1287134 - Failed to upgrade DC compatibility version from 3.4 to 3.5 after upgrading hosts
Failed to upgrade DC compatibility version from 3.4 to 3.5 after upgrading hosts
Status: CLOSED DUPLICATE of bug 1287136
Product: ovirt-engine
Classification: oVirt
Component: Backend.Core (Show other bugs)
3.6.1
Unspecified Unspecified
unspecified Severity unspecified (vote)
: ---
: ---
Assigned To: nobody nobody
Aharon Canan
Integration
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-01 10:14 EST by Kevin Alon Goldblatt
Modified: 2015-12-01 11:21 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-01 11:21:02 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)

  None (edit)
Description Kevin Alon Goldblatt 2015-12-01 10:14:05 EST
Description of problem:
Failed to upgrade DC compatibility version from 3.4 to 3.5 after upgrading hosts from 3.4 to 3.5

Version-Release number of selected component (if applicable):
rhevm-3.6.1-0.2.el6.noarch
vdsm-4.16.30-1.el7ev.x86_64 - host1
vdsm-4.16.27-1.el7ev.x86_64 - host2
vdsm-4.16.30-1.el7ev.x86_64 - host3

How reproducible:
Happens every time 


Steps to Reproduce:
1. Installed Engine with RHEVM3.4 (3.4.5-0.3)
2. Upgrade engine to RHEVM3.5 (3.5.6.2-0.1)  
3. Install 3.4 on 2 rhel7.2 hosts and 3.5 on 1 rhevh host
4. Add hosts to the 3.5 Engine to a DC with compatibility ver 3.5 and a Cluster with compatibility version 3.4
5. Created several SD's including FC, iscsi, nfs and gluster
6. Upgrade engine to Rhevm3.6 (3.6.1)
6. Upgrade rhel7.2 hosts from 3.4 to 3.5
7. When attempting to upgrade the compatibility version of the DC fron 3.4 to 3.5 an error is generated and the upgrade fails >>>>> "Operation Canceled
Error while executing action: Some of the host still use xmlrpc protocol which is not supported by cluster 3.6 or higher "

Actual results:
The Cluster compatibility version upgrade from 3.4 to 3.5 fails

Expected results:
The Cluster compatibility version upgrade should succeed


Additional info:
Engine.log
----------------------------
First attempt:

2015-12-01 15:51:56,580 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) [53193265] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='ERROR', powerStatus='UNKNOWN'
, message='Failed: Unable to obtain correct plug status or plug is not available


'}, log id: 6aafc90a
2015-12-01 15:51:56,644 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [53193265] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Execution of powe
r management status on Host seal12.qa.lab.tlv.redhat.com using Proxy Host seal08.qa.lab.tlv.redhat.com and Fence Agent ipmilan:seal12.qa.lab.tlv.redhat.com failed.
2015-12-01 15:51:56,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-53) [53193265] START, SetMOMPolicyParametersVDSCommand(HostName = seal12.qa.lab.tlv.redhat.com, 
MomPolicyVDSParameters:{runAsync='true', hostId='ee97bdec-9ed8-4f10-afd1-912154d984b8'}), log id: 32796b57
2015-12-01 15:51:56,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-53) [53193265] FINISH, SetMOMPolicyParametersVDSCommand, log id: 32796b57
2015-12-01 15:51:56,796 INFO  [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Running command: RefreshHostDevicesCommand internal: true. Entities affected :  ID: ee97bdec-9ed
8-4f10-afd1-912154d984b8 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2015-12-01 15:51:56,861 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host se
al12.qa.lab.tlv.redhat.com was set to Up.
2015-12-01 15:51:56,916 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management 
test failed for Host seal12.qa.lab.tlv.redhat.com.Failed: Unable to obtain correct plug status or plug is not available



2015-12-01 15:51:57,008 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Correlation ID: 11c887e2, Call Stack: null, Custom Event ID: -1, Message: Failed to ver
ify Host seal12.qa.lab.tlv.redhat.com power management.
2015-12-01 15:51:57,167 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-53) [62375468] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: ee97bdec-9ed8-4f10-afd1-
912154d984b8 Type: VDS
2015-12-01 15:52:59,113 WARN  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-5) [bf0fee8] CanDoAction of action 'UpdateVdsGroup' failed for user vdcadmin@qa.lab.tlv.redhat.com. Reasons: VAR__TYPE__CLUSTER,VAR__
ACTION__UPDATE,VDS_GROUP_CANNOT_UPDATE_COMPATIBILITY_VERSION_WITH_LOWER_HOSTS
2015-12-01 15:57:18,910 WARN  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-5) [3fef8609] CanDoAction of action 'UpdateVdsGroup' failed for user vdcadmin@qa.lab.tlv.redhat.com. Reasons: VAR__TYPE__CLUSTER,VAR_
_ACTION__UPDATE,ACTION_TYPE_FAILED_WRONG_PROTOCOL_FOR_CLUSTER_VERSION
2015-12-01 16:16:56,739 INFO  [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) [] Running command: LoginAdminUserCommand internal: false.
2015-12-01 16:16:56,831 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in.
2015-12-01 16:18:26,793 INFO  [org.ovirt.engine.core.bll.hostdeploy.UpdateVdsCommand] (ajp-/127.0.0.1:8702-10) [7e5fa5bb] Running command: UpdateVdsCommand internal: false. Entities affected :  ID: dfd79527-b192-4324-b09c-2d63376040e5
 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN
2015-12-01 16:18:27,069 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management c
onfiguration for Host orchid-vds1.qa.lab.tlv.redhat.com.




----------------------------------------------------------------------------

2nd attempt:


2015-12-01 16:55:25,546 INFO  [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (org.ovirt.thread.pool-7-thread-29) [4e5dabbe] Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' refresh
2015-12-01 16:55:25,607 INFO  [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-29) [4e5dabbe] SSH execute 'root@10.35.128.15' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True'
2015-12-01 16:55:26,644 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Initializing.
2015-12-01 16:55:26,728 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Environment setup.
2015-12-01 16:55:26,803 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Environment packages setup.
2015-12-01 16:58:10,145 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Programs detection.
2015-12-01 16:58:10,186 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Environment customization.
2015-12-01 17:00:53,595 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Kdump supported.
2015-12-01 17:00:53,687 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Logs at host located at: '/tmp/ovirt-host-deploy-20151201165524-i10pig.log'.
2015-12-01 17:00:53,739 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployVdsmUnit] (VdsDeploy) [4e5dabbe] Host 10.35.128.15 reports unique id 0e17ce27-ae1c-401b-aea9-6ab451fc9bb9
2015-12-01 17:00:53,787 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployVdsmUnit] (VdsDeploy) [4e5dabbe] Assigning unique id 0e17ce27-ae1c-401b-aea9-6ab451fc9bb9 to Host 10.35.128.15
2015-12-01 17:00:53,904 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Disabling Kdump integration.
2015-12-01 17:00:54,029 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Setup validation.
2015-12-01 17:01:14,096 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Hardware supports virtualization.
2015-12-01 17:01:14,288 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Transaction setup.
2015-12-01 17:01:14,313 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Misc configuration.
2015-12-01 17:01:14,363 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Package installation.
2015-12-01 17:01:49,409 WARN  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-7) [32ca59bc] CanDoAction of action 'UpdateVdsGroup' failed for user vdcadmin@qa.lab.tlv.redhat.com. Reasons: VAR__TYPE__CLUSTER,VAR__ACTION__UPDATE,ACTION_TYPE_FAILED_WRONG_PROTOCOL_FOR_CLUSTER_VERSION
(END)
Comment 1 Gil Klein 2015-12-01 11:21:02 EST

*** This bug has been marked as a duplicate of bug 1287136 ***

Note You need to log in before you can comment on or make changes to this bug.