Bug 1160204

Summary: [PPC] Failed to attach NFS storage: Error while executing action Attach Storage Domain: AcquireHostIdFailure
Product: Red Hat Enterprise Virtualization Manager Reporter: Lukas Svaty <lsvaty>
Component: ovirt-engineAssignee: Liron Aravot <laravot>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: 3.5.0CC: amureini, ecohen, gklein, iheim, lpeer, lsurette, lsvaty, rbalakri, Rhev-m-bugs, scohen, tnisan, yeylon
Target Milestone: ---   
Target Release: 3.5.0   
Hardware: ppc64   
OS: Linux   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1170202 (view as bug list) Environment:
Last Closed: 2014-11-10 15:23:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1170202    

Description Lukas Svaty 2014-11-04 10:30:32 UTC
Description of problem:
On ppc p8 host and engine 3.5 in 3.4 compatibility version cluster. Host is unable to attach storage domain.

Version-Release number of selected component (if applicable):
vt8

How reproducible:
100%

Steps to Reproduce:
1. Create 3.4 DC in 3.5 engine
2. Add P8 ppc host
3. Add storage

Actual results:
Error while executing action Attach Storage Domain: AcquireHostIdFailure

Expected results:
should succeed

Additional info:
2014-11-04 11:24:20,997 ERROR [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-1) [1ed76d2c] Command org.ovirt.engine.core.bll.storage
.AddStoragePoolWithStoragesCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException
: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: ('faf29088-4b04-4c50-b113-bbb610e99126', SanlockException(19, 'Sanlock lockspace add failure', 'No such device')), code = 661 (Failed with error AcquireHostIdFailure and code 661)
2014-11-04 11:24:21,007 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [1ed76d2c] Correlation ID: 410884c5, Job ID: 4cd56f
45-b596-499f-bede-e9e4ca06ab71, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domains to Data Center PPC. (User: admin)
2014-11-04 11:24:21,012 INFO  [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-1) [1ed76d2c] Lock freed to object EngineLock [exclusiv
eLocks= key: faf29088-4b04-4c50-b113-bbb610e99126 value: STORAGE
, sharedLocks= ]
2014-11-04 11:24:21,014 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-1) [1ed76d2c] Command [id=9673910d-6cbd-4ff3-8c70-445f56
e5e2ac]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StoragePool; snapshot: id=97f43f09-7d3c-4c26-8566-ed0e743bcf77.
2014-11-04 11:24:21,016 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-1) [1ed76d2c] Command [id=9673910d-6cbd-4ff3-8c70-445f56
e5e2ac]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: storagePoolId = 97f43f09-7d3c-4c26-8566-ed0e743bcf77, storageI
d = faf29088-4b04-4c50-b113-bbb610e99126.
2014-11-04 11:24:21,017 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-1) [1ed76d2c] Command [id=9673910d-6cbd-4ff3-8c70-445f56
e5e2ac]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: id=faf29088-4b04-4c50-b113-bbb610e99126.
2014-11-04 11:24:21,026 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [1ed76d2c] Correlation ID: 46ec8c, Job ID: 4cd56f45
-b596-499f-bede-e9e4ca06ab71, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domain nfs02 to Data Center PPC. (User: admin)

Comment 3 Liron Aravot 2014-11-06 09:27:47 UTC
Hi Lukas,
it apparent that sanlock might have problem accessing the leases file located under the domain path (under dom_md/).

you can try to restart the sanlock service (be sure to put the host to maintenance first), if the problem persists please reproduce and attach the sanlock log as well and the exact sanlock/vdsm/engine versions and the output of the check of the leases file

thanks,
Liron.

Comment 4 Lukas Svaty 2014-11-10 15:23:15 UTC
Hi, thank

steps for fix:
1. `service sanlock restart` on hosts
2. clean path for nfs storage
3. Create and add new nfs storage

CLOSING this if I'll be able to reproduce it with I can re-open.