Bug 1287134
| Summary: | Failed to upgrade DC compatibility version from 3.4 to 3.5 after upgrading hosts | ||
|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Kevin Alon Goldblatt <kgoldbla> |
| Component: | Backend.Core | Assignee: | Nobody <nobody> |
| Status: | CLOSED DUPLICATE | QA Contact: | Aharon Canan <acanan> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.6.1 | CC: | bugs, gklein |
| Target Milestone: | --- | Flags: | rule-engine:
planning_ack?
rule-engine: devel_ack? rule-engine: testing_ack? |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | Integration | ||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-12-01 16:21:02 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** This bug has been marked as a duplicate of bug 1287136 *** |
Description of problem: Failed to upgrade DC compatibility version from 3.4 to 3.5 after upgrading hosts from 3.4 to 3.5 Version-Release number of selected component (if applicable): rhevm-3.6.1-0.2.el6.noarch vdsm-4.16.30-1.el7ev.x86_64 - host1 vdsm-4.16.27-1.el7ev.x86_64 - host2 vdsm-4.16.30-1.el7ev.x86_64 - host3 How reproducible: Happens every time Steps to Reproduce: 1. Installed Engine with RHEVM3.4 (3.4.5-0.3) 2. Upgrade engine to RHEVM3.5 (3.5.6.2-0.1) 3. Install 3.4 on 2 rhel7.2 hosts and 3.5 on 1 rhevh host 4. Add hosts to the 3.5 Engine to a DC with compatibility ver 3.5 and a Cluster with compatibility version 3.4 5. Created several SD's including FC, iscsi, nfs and gluster 6. Upgrade engine to Rhevm3.6 (3.6.1) 6. Upgrade rhel7.2 hosts from 3.4 to 3.5 7. When attempting to upgrade the compatibility version of the DC fron 3.4 to 3.5 an error is generated and the upgrade fails >>>>> "Operation Canceled Error while executing action: Some of the host still use xmlrpc protocol which is not supported by cluster 3.6 or higher " Actual results: The Cluster compatibility version upgrade from 3.4 to 3.5 fails Expected results: The Cluster compatibility version upgrade should succeed Additional info: Engine.log ---------------------------- First attempt: 2015-12-01 15:51:56,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) [53193265] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='ERROR', powerStatus='UNKNOWN' , message='Failed: Unable to obtain correct plug status or plug is not available '}, log id: 6aafc90a 2015-12-01 15:51:56,644 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [53193265] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Execution of powe r management status on Host seal12.qa.lab.tlv.redhat.com using Proxy Host seal08.qa.lab.tlv.redhat.com and Fence Agent ipmilan:seal12.qa.lab.tlv.redhat.com failed. 2015-12-01 15:51:56,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-53) [53193265] START, SetMOMPolicyParametersVDSCommand(HostName = seal12.qa.lab.tlv.redhat.com, MomPolicyVDSParameters:{runAsync='true', hostId='ee97bdec-9ed8-4f10-afd1-912154d984b8'}), log id: 32796b57 2015-12-01 15:51:56,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-53) [53193265] FINISH, SetMOMPolicyParametersVDSCommand, log id: 32796b57 2015-12-01 15:51:56,796 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: ee97bdec-9ed 8-4f10-afd1-912154d984b8 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2015-12-01 15:51:56,861 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host se al12.qa.lab.tlv.redhat.com was set to Up. 2015-12-01 15:51:56,916 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host seal12.qa.lab.tlv.redhat.com.Failed: Unable to obtain correct plug status or plug is not available 2015-12-01 15:51:57,008 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [5e5eef00] Correlation ID: 11c887e2, Call Stack: null, Custom Event ID: -1, Message: Failed to ver ify Host seal12.qa.lab.tlv.redhat.com power management. 2015-12-01 15:51:57,167 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-53) [62375468] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: ee97bdec-9ed8-4f10-afd1- 912154d984b8 Type: VDS 2015-12-01 15:52:59,113 WARN [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-5) [bf0fee8] CanDoAction of action 'UpdateVdsGroup' failed for user vdcadmin.tlv.redhat.com. Reasons: VAR__TYPE__CLUSTER,VAR__ ACTION__UPDATE,VDS_GROUP_CANNOT_UPDATE_COMPATIBILITY_VERSION_WITH_LOWER_HOSTS 2015-12-01 15:57:18,910 WARN [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-5) [3fef8609] CanDoAction of action 'UpdateVdsGroup' failed for user vdcadmin.tlv.redhat.com. Reasons: VAR__TYPE__CLUSTER,VAR_ _ACTION__UPDATE,ACTION_TYPE_FAILED_WRONG_PROTOCOL_FOR_CLUSTER_VERSION 2015-12-01 16:16:56,739 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) [] Running command: LoginAdminUserCommand internal: false. 2015-12-01 16:16:56,831 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. 2015-12-01 16:18:26,793 INFO [org.ovirt.engine.core.bll.hostdeploy.UpdateVdsCommand] (ajp-/127.0.0.1:8702-10) [7e5fa5bb] Running command: UpdateVdsCommand internal: false. Entities affected : ID: dfd79527-b192-4324-b09c-2d63376040e5 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2015-12-01 16:18:27,069 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management c onfiguration for Host orchid-vds1.qa.lab.tlv.redhat.com. ---------------------------------------------------------------------------- 2nd attempt: 2015-12-01 16:55:25,546 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (org.ovirt.thread.pool-7-thread-29) [4e5dabbe] Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' refresh 2015-12-01 16:55:25,607 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-29) [4e5dabbe] SSH execute 'root.128.15' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2015-12-01 16:55:26,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Initializing. 2015-12-01 16:55:26,728 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Environment setup. 2015-12-01 16:55:26,803 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Environment packages setup. 2015-12-01 16:58:10,145 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Programs detection. 2015-12-01 16:58:10,186 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Environment customization. 2015-12-01 17:00:53,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Kdump supported. 2015-12-01 17:00:53,687 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Logs at host located at: '/tmp/ovirt-host-deploy-20151201165524-i10pig.log'. 2015-12-01 17:00:53,739 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployVdsmUnit] (VdsDeploy) [4e5dabbe] Host 10.35.128.15 reports unique id 0e17ce27-ae1c-401b-aea9-6ab451fc9bb9 2015-12-01 17:00:53,787 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployVdsmUnit] (VdsDeploy) [4e5dabbe] Assigning unique id 0e17ce27-ae1c-401b-aea9-6ab451fc9bb9 to Host 10.35.128.15 2015-12-01 17:00:53,904 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Disabling Kdump integration. 2015-12-01 17:00:54,029 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Setup validation. 2015-12-01 17:01:14,096 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Hardware supports virtualization. 2015-12-01 17:01:14,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Transaction setup. 2015-12-01 17:01:14,313 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Misc configuration. 2015-12-01 17:01:14,363 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [4e5dabbe] Correlation ID: 4e5dabbe, Call Stack: null, Custom Event ID: -1, Message: Installing Host orchid-vds2.qa.lab.tlv.redhat.com. Stage: Package installation. 2015-12-01 17:01:49,409 WARN [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-7) [32ca59bc] CanDoAction of action 'UpdateVdsGroup' failed for user vdcadmin.tlv.redhat.com. Reasons: VAR__TYPE__CLUSTER,VAR__ACTION__UPDATE,ACTION_TYPE_FAILED_WRONG_PROTOCOL_FOR_CLUSTER_VERSION (END)