Bug 1465975 - Using proto=rdma (NFSoRDMA) fails on nfs storage data domains
Using proto=rdma (NFSoRDMA) fails on nfs storage data domains
Status: NEW
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: nfs-utils (Show other bugs)
7.4
x86_64 Linux
unspecified Severity high
: pre-dev-freeze
: ---
Assigned To: Steve Dickson
Filesystem QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-28 11:08 EDT by Sam Yangsao
Modified: 2017-12-19 14:21 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Sam Yangsao 2017-06-28 11:08:34 EDT
Description of problem:

Using proto=rdma (NFSoRDMA) fails on nfs storage data domains

Version-Release number of selected component (if applicable):

[manager]

# uname -a
Linux rhevm.lab.msp.redhat.com 3.10.0-514.21.2.el7.x86_64 #1 SMP Sun May 28 17:08:21 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa |grep ovirt
ovirt-web-ui-0.2.2-1.el7ev.x86_64
ovirt-engine-extension-aaa-jdbc-1.1.4-1.el7ev.noarch
ovirt-engine-websocket-proxy-4.1.2.3-0.1.el7.noarch
ovirt-engine-setup-4.1.2.3-0.1.el7.noarch
ovirt-engine-metrics-1.0.3-1.el7ev.noarch
ovirt-engine-setup-base-4.1.2.3-0.1.el7.noarch
ovirt-engine-dwh-setup-4.1.1-1.el7ev.noarch
ovirt-engine-restapi-4.1.2.3-0.1.el7.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.1.2.3-0.1.el7.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
python-ovirt-engine-sdk4-4.1.3-1.el7ev.x86_64
ovirt-engine-lib-4.1.2.3-0.1.el7.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-iso-uploader-4.0.2-1.el7ev.noarch
ovirt-imageio-proxy-setup-1.0.0-0.el7ev.noarch
ovirt-engine-tools-backup-4.1.2.3-0.1.el7.noarch
ovirt-host-deploy-java-1.6.5-1.el7ev.noarch
ovirt-engine-dwh-4.1.1-1.el7ev.noarch
ovirt-engine-backend-4.1.2.3-0.1.el7.noarch
ovirt-engine-webadmin-portal-4.1.2.3-0.1.el7.noarch
ovirt-engine-setup-plugin-ovirt-engine-4.1.2.3-0.1.el7.noarch
ovirt-log-collector-4.1.1-1.el7ev.noarch
ovirt-host-deploy-1.6.5-1.el7ev.noarch
ovirt-vmconsole-proxy-1.0.4-1.el7ev.noarch
ovirt-engine-setup-plugin-ovirt-engine-common-4.1.2.3-0.1.el7.noarch
ovirt-engine-extensions-api-impl-4.1.2.3-0.1.el7.noarch
ovirt-imageio-proxy-1.0.0-0.el7ev.noarch
ovirt-engine-setup-plugin-websocket-proxy-4.1.2.3-0.1.el7.noarch
ovirt-engine-userportal-4.1.2.3-0.1.el7.noarch
ovirt-engine-tools-4.1.2.3-0.1.el7.noarch
ovirt-engine-4.1.2.3-0.1.el7.noarch
ovirt-engine-cli-3.6.8.1-1.el7ev.noarch
ovirt-engine-dbscripts-4.1.2.3-0.1.el7.noarch
ovirt-engine-vmconsole-proxy-helper-4.1.2.3-0.1.el7.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
ovirt-engine-dashboard-1.1.2-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch

[hypervisor]

# uname -a
Linux rhev4.lab.msp.redhat.com 3.10.0-514.21.2.el7.x86_64 #1 SMP Sun May 28 17:08:21 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa |grep vdsm
vdsm-api-4.19.15-1.el7ev.noarch
vdsm-xmlrpc-4.19.15-1.el7ev.noarch
vdsm-hook-vmfex-dev-4.19.15-1.el7ev.noarch
vdsm-python-4.19.15-1.el7ev.noarch
vdsm-yajsonrpc-4.19.15-1.el7ev.noarch
vdsm-4.19.15-1.el7ev.x86_64
vdsm-cli-4.19.15-1.el7ev.noarch
vdsm-jsonrpc-4.19.15-1.el7ev.noarch

How reproducible:

Always

Steps to Reproduce:

1.  Install RHV 4.1
2.  Setup a RHEL-H host with an infiniband network using rdma proto
3.  Create new storage and attempt to mount via the rdma proto option in the "New Domain" section.

### Actual results:

See in BZ comments

### Expected results:

Adding nfs storage domains using the rdma protocol should continue to work.

### Additional info:

Manually mounts work fine on the hypervisor.

# mount -o proto=rdma,port=20049,vers=4.2 tatooine-ib0:/storage/export2 /mnt
# mount |grep mnt
tatooine-ib0:/storage/export2 on /mnt type nfs4 (rw,relatime,seclabel,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.14,local_lock=none,addr=192.168.1.17)

This used to work fine in v4.0.  Upgraded to v4.1 and it changes the mount proto to "tcp".  Attempting to remount the storage domain back to "rdma" continues to fail and puts the storage domain in an "unattached" state.  Attempts to "Attach Data" for this storage domain continue to fail.
Comment 1 Sam Yangsao 2017-06-28 11:14:28 EDT
Failure when trying to mount using the custom options 

New Domain > Custom Connection Parameters

Addional mount options:  proto=rdma,port=20049

# [manager] /var/log/ovirt-engine/engine.log

2017-06-28 09:23:14,634-05 INFO  [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-64) [639e261c-888c-4804-90ad-6c3312ac020a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[tatooine-ib0.lab.msp.redhat.com:/storage/export2=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2017-06-28 09:23:14,644-05 INFO  [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-64) [639e261c-888c-4804-90ad-6c3312ac020a] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-06-28 09:23:14,646-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-64) [639e261c-888c-4804-90ad-6c3312ac020a] START, ConnectStorageServerVDSCommand(HostName = rhev4.lab.msp.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='null', connection='tatooine-ib0.lab.msp.redhat.com:/storage/export2', iqn='null', vfsType='null', mountOptions='port=20049,proto=rdma', nfsVersion='V4_2', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4edb3f58
2017-06-28 09:23:15,421-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-64) [639e261c-888c-4804-90ad-6c3312ac020a] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 4edb3f58
2017-06-28 09:23:15,435-05 INFO  [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-64) [639e261c-888c-4804-90ad-6c3312ac020a] Lock freed to object 'EngineLock:{exclusiveLocks='[tatooine-ib0.lab.msp.redhat.com:/storage/export2=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2017-06-28 09:23:15,936-05 INFO  [org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand] (default task-7) [2af933a4-7d08-408e-bb92-4b3c565f609c] Running command: AddNFSStorageDomainCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-06-28 09:23:16,025-05 INFO  [org.ovirt.engine.core.bll.profiles.AddDiskProfileCommand] (default task-7) [8a4eec3] Running command: AddDiskProfileCommand internal: true. Entities affected :  ID: fdd281d6-4dbd-4261-a924-6cbf5713ed8a Type: StorageAction group CREATE_STORAGE_DISK_PROFILE with role type ADMIN
2017-06-28 09:23:16,063-05 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [8a4eec3] EVENT_ID: USER_ADDED_DISK_PROFILE(10,120), Correlation ID: 8a4eec3, Job ID: 5add7925-ddec-4c37-b52d-4945297d2b3c, Call Stack: null, Custom Event ID: -1, Message: Disk Profile tatooine-export2 was successfully added (User: admin@internal-authz).
2017-06-28 09:23:16,071-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-7) [8a4eec3] START, ConnectStorageServerVDSCommand(HostName = rhev4.lab.msp.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='0e247058-90eb-4e34-8e10-faed3a1da0fe', connection='tatooine-ib0.lab.msp.redhat.com:/storage/export2', iqn='null', vfsType='null', mountOptions='port=20049,proto=rdma', nfsVersion='V4_2', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3a27b093
2017-06-28 09:23:16,496-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-7) [8a4eec3] FINISH, ConnectStorageServerVDSCommand, return: {0e247058-90eb-4e34-8e10-faed3a1da0fe=0}, log id: 3a27b093
2017-06-28 09:23:16,501-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-7) [8a4eec3] START, CreateStorageDomainVDSCommand(HostName = rhev4.lab.msp.redhat.com, CreateStorageDomainVDSCommandParameters:{runAsync='true', hostId='f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5', storageDomain='StorageDomainStatic:{name='tatooine-export2', id='fdd281d6-4dbd-4261-a924-6cbf5713ed8a'}', args='tatooine-ib0.lab.msp.redhat.com:/storage/export2'}), log id: 330bc430
2017-06-28 09:23:17,224-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-7) [8a4eec3] FINISH, CreateStorageDomainVDSCommand, log id: 330bc430
2017-06-28 09:23:17,228-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (default task-7) [8a4eec3] START, GetStorageDomainStatsVDSCommand(HostName = rhev4.lab.msp.redhat.com, GetStorageDomainStatsVDSCommandParameters:{runAsync='true', hostId='f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5', storageDomainId='fdd281d6-4dbd-4261-a924-6cbf5713ed8a'}), log id: 146e2cb6
2017-06-28 09:23:18,052-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (default task-7) [8a4eec3] FINISH, GetStorageDomainStatsVDSCommand, return: StorageDomain:{domainName='', domainId='fdd281d6-4dbd-4261-a924-6cbf5713ed8a'}, log id: 146e2cb6
2017-06-28 09:23:18,083-05 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [8a4eec3] EVENT_ID: USER_ADD_STORAGE_DOMAIN(956), Correlation ID: 2af933a4-7d08-408e-bb92-4b3c565f609c, Job ID: 5add7925-ddec-4c37-b52d-4945297d2b3c, Call Stack: null, Custom Event ID: -1, Message: Storage Domain tatooine-export2 was added by admin@internal-authz
2017-06-28 09:23:18,315-05 INFO  [org.ovirt.engine.core.bll.storage.connection.DisconnectStorageServerConnectionCommand] (default task-12) [729e7d43-364a-4d89-9cda-d93b45a49f21] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-06-28 09:23:18,318-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-12) [729e7d43-364a-4d89-9cda-d93b45a49f21] START, DisconnectStorageServerVDSCommand(HostName = rhev4.lab.msp.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='0e247058-90eb-4e34-8e10-faed3a1da0fe', connection='tatooine-ib0.lab.msp.redhat.com:/storage/export2', iqn='null', vfsType='null', mountOptions='port=20049,proto=rdma', nfsVersion='V4_2', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 11e3bf30
2017-06-28 09:23:19,514-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-12) [729e7d43-364a-4d89-9cda-d93b45a49f21] FINISH, DisconnectStorageServerVDSCommand, return: {0e247058-90eb-4e34-8e10-faed3a1da0fe=0}, log id: 11e3bf30
2017-06-28 09:23:20,047-05 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [9ea4f91c-9fb6-417d-99f3-cdcebb160887] Lock Acquired to object 'EngineLock:{exclusiveLocks='[fdd281d6-4dbd-4261-a924-6cbf5713ed8a=<STORAGE, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2017-06-28 09:23:20,126-05 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [9ea4f91c-9fb6-417d-99f3-cdcebb160887] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: fdd281d6-4dbd-4261-a924-6cbf5713ed8a Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN,  ID: 00000001-0001-0001-0001-000000000311 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN
2017-06-28 09:23:20,281-05 INFO  [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (default task-1) [3167ca15] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-06-28 09:23:20,312-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-1) [3167ca15] START, ConnectStorageServerVDSCommand(HostName = rhev4.lab.msp.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='0e247058-90eb-4e34-8e10-faed3a1da0fe', connection='tatooine-ib0.lab.msp.redhat.com:/storage/export2', iqn='null', vfsType='null', mountOptions='port=20049,proto=rdma', nfsVersion='V4_2', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 253e51ca
2017-06-28 09:23:21,444-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-1) [3167ca15] FINISH, ConnectStorageServerVDSCommand, return: {0e247058-90eb-4e34-8e10-faed3a1da0fe=0}, log id: 253e51ca
2017-06-28 09:23:21,460-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (default task-1) [3167ca15] START, HSMGetStorageDomainInfoVDSCommand(HostName = rhev4.lab.msp.redhat.com, HSMGetStorageDomainInfoVDSCommandParameters:{runAsync='true', hostId='f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5', storageDomainId='fdd281d6-4dbd-4261-a924-6cbf5713ed8a'}), log id: 61eefbfc
2017-06-28 09:23:22,621-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (default task-1) [3167ca15] FINISH, HSMGetStorageDomainInfoVDSCommand, return: <StorageDomainStatic:{name='tatooine-export2', id='fdd281d6-4dbd-4261-a924-6cbf5713ed8a'}, null>, log id: 61eefbfc
2017-06-28 09:23:22,646-05 INFO  [org.ovirt.engine.core.bll.storage.connection.DisconnectStorageServerConnectionCommand] (default task-1) [1efae471] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-06-28 09:23:22,649-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-1) [1efae471] START, DisconnectStorageServerVDSCommand(HostName = rhev4.lab.msp.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='0e247058-90eb-4e34-8e10-faed3a1da0fe', connection='tatooine-ib0.lab.msp.redhat.com:/storage/export2', iqn='null', vfsType='null', mountOptions='port=20049,proto=rdma', nfsVersion='V4_2', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 26c22ea4
2017-06-28 09:23:23,836-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-1) [1efae471] FINISH, DisconnectStorageServerVDSCommand, return: {0e247058-90eb-4e34-8e10-faed3a1da0fe=0}, log id: 26c22ea4
2017-06-28 09:23:23,851-05 INFO  [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected :  ID: 00000001-0001-0001-0001-000000000311 Type: StoragePoolAction group CREATE_STORAGE_POOL with role type ADMIN
2017-06-28 09:23:23,991-05 INFO  [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (default task-1) [4ba0ef6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-06-28 09:23:24,022-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-1) [4ba0ef6] START, ConnectStorageServerVDSCommand(HostName = rhev5.lab.msp.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='867c53db-b77c-48a4-888a-1d9331b26be0', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='0e247058-90eb-4e34-8e10-faed3a1da0fe', connection='tatooine-ib0.lab.msp.redhat.com:/storage/export2', iqn='null', vfsType='null', mountOptions='port=20049,proto=rdma', nfsVersion='V4_2', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 345e5a94
2017-06-28 09:23:40,159-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [4ba0ef6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rhev5.lab.msp.redhat.com to the Storage Domains tatooine-export2.
2017-06-28 09:23:40,159-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-1) [4ba0ef6] FINISH, ConnectStorageServerVDSCommand, return: {0e247058-90eb-4e34-8e10-faed3a1da0fe=477}, log id: 345e5a94
2017-06-28 09:23:40,169-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [4ba0ef6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection tatooine-ib0.lab.msp.redhat.com:/storage/export2 returned by VDSM was: Problem while trying to mount target
2017-06-28 09:23:40,172-05 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (default task-1) [4ba0ef6] The connection with details 'tatooine-ib0.lab.msp.redhat.com:/storage/export2' failed because of error code '477' and error message is: problem while trying to mount target
2017-06-28 09:23:40,188-05 ERROR [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (default task-1) [4ba0ef6] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand'.
2017-06-28 09:23:40,193-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [4ba0ef6] START, CreateStoragePoolVDSCommand(HostName = rhev5.lab.msp.redhat.com, CreateStoragePoolVDSCommandParameters:{runAsync='true', hostId='867c53db-b77c-48a4-888a-1d9331b26be0', storagePoolId='00000001-0001-0001-0001-000000000311', storagePoolName='HP', masterDomainId='fdd281d6-4dbd-4261-a924-6cbf5713ed8a', domainsIdList='[fdd281d6-4dbd-4261-a924-6cbf5713ed8a]', masterVersion='1'}), log id: 335a7d50
2017-06-28 09:23:41,419-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [4ba0ef6] Failed in 'CreateStoragePoolVDS' method
2017-06-28 09:23:41,429-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [4ba0ef6] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM rhev5.lab.msp.redhat.com command CreateStoragePoolVDS failed: Storage domain does not exist: (u'fdd281d6-4dbd-4261-a924-6cbf5713ed8a',)
2017-06-28 09:23:41,430-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [4ba0ef6] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=358, message=Storage domain does not exist: (u'fdd281d6-4dbd-4261-a924-6cbf5713ed8a',)]]'
2017-06-28 09:23:41,430-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [4ba0ef6] HostName = rhev5.lab.msp.redhat.com
2017-06-28 09:23:41,430-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [4ba0ef6] Command 'CreateStoragePoolVDSCommand(HostName = rhev5.lab.msp.redhat.com, CreateStoragePoolVDSCommandParameters:{runAsync='true', hostId='867c53db-b77c-48a4-888a-1d9331b26be0', storagePoolId='00000001-0001-0001-0001-000000000311', storagePoolName='HP', masterDomainId='fdd281d6-4dbd-4261-a924-6cbf5713ed8a', domainsIdList='[fdd281d6-4dbd-4261-a924-6cbf5713ed8a]', masterVersion='1'})' execution failed: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Storage domain does not exist: (u'fdd281d6-4dbd-4261-a924-6cbf5713ed8a',), code = 358
2017-06-28 09:23:41,430-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [4ba0ef6] FINISH, CreateStoragePoolVDSCommand, log id: 335a7d50
2017-06-28 09:23:41,430-05 ERROR [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [4ba0ef6] Command 'org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Storage domain does not exist: (u'fdd281d6-4dbd-4261-a924-6cbf5713ed8a',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358)
2017-06-28 09:23:41,448-05 INFO  [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [4ba0ef6] Command [id=d71f5e0b-e56a-4ef9-86ba-b9009f93a97f]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StoragePool; snapshot: id=00000001-0001-0001-0001-000000000311.
2017-06-28 09:23:41,452-05 INFO  [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [4ba0ef6] Command [id=d71f5e0b-e56a-4ef9-86ba-b9009f93a97f]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{storagePoolId='00000001-0001-0001-0001-000000000311', storageId='fdd281d6-4dbd-4261-a924-6cbf5713ed8a'}.
2017-06-28 09:23:41,460-05 INFO  [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [4ba0ef6] Command [id=d71f5e0b-e56a-4ef9-86ba-b9009f93a97f]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: id=fdd281d6-4dbd-4261-a924-6cbf5713ed8a.
2017-06-28 09:23:41,488-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [4ba0ef6] EVENT_ID: USER_ATTACH_STORAGE_DOMAINS_TO_POOL_FAILED(1,003), Correlation ID: 779b60ab, Job ID: 210d3ec1-3581-4e62-a6d6-ae97fbce1660, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domains to Data Center HP. (User: admin@internal-authz)
2017-06-28 09:23:41,494-05 INFO  [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [4ba0ef6] Lock freed to object 'EngineLock:{exclusiveLocks='[fdd281d6-4dbd-4261-a924-6cbf5713ed8a=<STORAGE, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2017-06-28 09:23:41,504-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [4ba0ef6] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Correlation ID: 9ea4f91c-9fb6-417d-99f3-cdcebb160887, Job ID: 210d3ec1-3581-4e62-a6d6-ae97fbce1660, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domain tatooine-export2 to Data Center HP. (User: admin@internal-authz)
2017-06-28 09:23:41,508-05 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-1) [4ba0ef6] Trying to release exclusive lock which does not exist, lock key: 'fdd281d6-4dbd-4261-a924-6cbf5713ed8aSTORAGE'
2017-06-28 09:23:41,508-05 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [4ba0ef6] Lock freed to object 'EngineLock:{exclusiveLocks='[fdd281d6-4dbd-4261-a924-6cbf5713ed8a=<STORAGE, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
Comment 2 Sam Yangsao 2017-06-28 11:17:04 EDT
Attempt to "Attach Storage" since it's sitting in limbo in RHV now

Data Centers > HP > Storage tab > Attach Data

# [manager] /var/log/ovirt-engine/engine.log

2017-06-28 09:25:55,777-05 INFO  [org.ovirt.engine.core.bll.storage.domain.GetStorageDomainsWithAttachedStoragePoolGuidQuery] (default task-19) [69cf6824-f56d-4d4b-846a-1c0ff44ce3f8] vds id 'f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5' was chosen to fetch the Storage domain info
2017-06-28 09:25:55,779-05 INFO  [org.ovirt.engine.core.bll.storage.domain.GetStorageDomainsWithAttachedStoragePoolGuidQuery] (default task-19) [69cf6824-f56d-4d4b-846a-1c0ff44ce3f8] The Data Center is not in UP status.
2017-06-28 09:25:55,996-05 INFO  [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-16) [8b48ec64-9c2e-414f-ab77-b7b7772032d2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[00000001-0001-0001-0001-000000000311=<POOL, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2017-06-28 09:25:56,095-05 INFO  [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (default task-16) [74f8b1e4] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-06-28 09:25:56,126-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-16) [74f8b1e4] START, ConnectStorageServerVDSCommand(HostName = rhev4.lab.msp.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='0e247058-90eb-4e34-8e10-faed3a1da0fe', connection='tatooine-ib0.lab.msp.redhat.com:/storage/export2', iqn='null', vfsType='null', mountOptions='port=20049,proto=rdma', nfsVersion='V4_2', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 563bbb36
2017-06-28 09:25:56,579-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-16) [74f8b1e4] FINISH, ConnectStorageServerVDSCommand, return: {0e247058-90eb-4e34-8e10-faed3a1da0fe=0}, log id: 563bbb36
2017-06-28 09:25:56,588-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (default task-16) [74f8b1e4] START, HSMGetStorageDomainInfoVDSCommand(HostName = rhev4.lab.msp.redhat.com, HSMGetStorageDomainInfoVDSCommandParameters:{runAsync='true', hostId='f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5', storageDomainId='fdd281d6-4dbd-4261-a924-6cbf5713ed8a'}), log id: 26447dc7
2017-06-28 09:25:57,389-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (default task-16) [74f8b1e4] FINISH, HSMGetStorageDomainInfoVDSCommand, return: <StorageDomainStatic:{name='tatooine-export2', id='fdd281d6-4dbd-4261-a924-6cbf5713ed8a'}, null>, log id: 26447dc7
2017-06-28 09:25:57,407-05 INFO  [org.ovirt.engine.core.bll.storage.connection.DisconnectStorageServerConnectionCommand] (default task-16) [44d95eda] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-06-28 09:25:57,408-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-16) [44d95eda] START, DisconnectStorageServerVDSCommand(HostName = rhev4.lab.msp.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='f6ae8e72-4e87-49f9-8aa9-90e1a6952ad5', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='0e247058-90eb-4e34-8e10-faed3a1da0fe', connection='tatooine-ib0.lab.msp.redhat.com:/storage/export2', iqn='null', vfsType='null', mountOptions='port=20049,proto=rdma', nfsVersion='V4_2', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 10cfe7c3
2017-06-28 09:25:58,593-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-16) [44d95eda] FINISH, DisconnectStorageServerVDSCommand, return: {0e247058-90eb-4e34-8e10-faed3a1da0fe=0}, log id: 10cfe7c3
2017-06-28 09:25:58,616-05 INFO  [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (org.ovirt.thread.pool-7-thread-13) [8b48ec64-9c2e-414f-ab77-b7b7772032d2] Running command: AddStoragePoolWithStoragesCommand internal: false. Entities affected :  ID: 00000001-0001-0001-0001-000000000311 Type: StoragePoolAction group CREATE_STORAGE_POOL with role type ADMIN
2017-06-28 09:25:58,710-05 INFO  [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-06-28 09:25:58,740-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] START, ConnectStorageServerVDSCommand(HostName = rhev5.lab.msp.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='867c53db-b77c-48a4-888a-1d9331b26be0', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='0e247058-90eb-4e34-8e10-faed3a1da0fe', connection='tatooine-ib0.lab.msp.redhat.com:/storage/export2', iqn='null', vfsType='null', mountOptions='port=20049,proto=rdma', nfsVersion='V4_2', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 85b268
2017-06-28 09:26:14,828-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rhev5.lab.msp.redhat.com to the Storage Domains tatooine-export2.
2017-06-28 09:26:14,828-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] FINISH, ConnectStorageServerVDSCommand, return: {0e247058-90eb-4e34-8e10-faed3a1da0fe=477}, log id: 85b268
2017-06-28 09:26:14,835-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection tatooine-ib0.lab.msp.redhat.com:/storage/export2 returned by VDSM was: Problem while trying to mount target
2017-06-28 09:26:14,837-05 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] The connection with details 'tatooine-ib0.lab.msp.redhat.com:/storage/export2' failed because of error code '477' and error message is: problem while trying to mount target
2017-06-28 09:26:14,845-05 ERROR [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand'.
2017-06-28 09:26:14,847-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] START, CreateStoragePoolVDSCommand(HostName = rhev5.lab.msp.redhat.com, CreateStoragePoolVDSCommandParameters:{runAsync='true', hostId='867c53db-b77c-48a4-888a-1d9331b26be0', storagePoolId='00000001-0001-0001-0001-000000000311', storagePoolName='HP', masterDomainId='fdd281d6-4dbd-4261-a924-6cbf5713ed8a', domainsIdList='[fdd281d6-4dbd-4261-a924-6cbf5713ed8a]', masterVersion='2'}), log id: 7131709c
2017-06-28 09:26:16,033-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] Failed in 'CreateStoragePoolVDS' method
2017-06-28 09:26:16,040-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM rhev5.lab.msp.redhat.com command CreateStoragePoolVDS failed: Storage domain does not exist: (u'fdd281d6-4dbd-4261-a924-6cbf5713ed8a',)
2017-06-28 09:26:16,041-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=358, message=Storage domain does not exist: (u'fdd281d6-4dbd-4261-a924-6cbf5713ed8a',)]]'
2017-06-28 09:26:16,041-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] HostName = rhev5.lab.msp.redhat.com
2017-06-28 09:26:16,041-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] Command 'CreateStoragePoolVDSCommand(HostName = rhev5.lab.msp.redhat.com, CreateStoragePoolVDSCommandParameters:{runAsync='true', hostId='867c53db-b77c-48a4-888a-1d9331b26be0', storagePoolId='00000001-0001-0001-0001-000000000311', storagePoolName='HP', masterDomainId='fdd281d6-4dbd-4261-a924-6cbf5713ed8a', domainsIdList='[fdd281d6-4dbd-4261-a924-6cbf5713ed8a]', masterVersion='2'})' execution failed: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Storage domain does not exist: (u'fdd281d6-4dbd-4261-a924-6cbf5713ed8a',), code = 358
2017-06-28 09:26:16,041-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] FINISH, CreateStoragePoolVDSCommand, log id: 7131709c
2017-06-28 09:26:16,041-05 ERROR [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] Command 'org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Storage domain does not exist: (u'fdd281d6-4dbd-4261-a924-6cbf5713ed8a',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358)
2017-06-28 09:26:16,045-05 INFO  [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] Command [id=1908ae35-aec9-4a37-9c12-e7b96cb33a34]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StoragePool; snapshot: id=00000001-0001-0001-0001-000000000311.
2017-06-28 09:26:16,047-05 INFO  [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] Command [id=1908ae35-aec9-4a37-9c12-e7b96cb33a34]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{storagePoolId='00000001-0001-0001-0001-000000000311', storageId='fdd281d6-4dbd-4261-a924-6cbf5713ed8a'}.
2017-06-28 09:26:16,048-05 INFO  [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] Command [id=1908ae35-aec9-4a37-9c12-e7b96cb33a34]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: id=fdd281d6-4dbd-4261-a924-6cbf5713ed8a.
2017-06-28 09:26:16,070-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] EVENT_ID: USER_ATTACH_STORAGE_DOMAINS_TO_POOL_FAILED(1,003), Correlation ID: 8b48ec64-9c2e-414f-ab77-b7b7772032d2, Job ID: b2dba68f-4994-4d97-9291-689848a244c0, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domains to Data Center HP. (User: admin@internal-authz)
2017-06-28 09:26:16,075-05 INFO  [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (org.ovirt.thread.pool-7-thread-13) [53ba10a9] Lock freed to object 'EngineLock:{exclusiveLocks='[00000001-0001-0001-0001-000000000311=<POOL, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
Comment 5 Allon Mureinik 2017-06-29 04:43:16 EDT
Something's odd here.
In VDSM's log I can see

MountError: (32, ';mount.nfs: Failed to resolve server tatooine-ib0: Name or service not known\n')

Which both seems to work manually according to comment 4 (altghough the credentials seem wrong, so I can't check myself), and from the other host.

Need to give this a deeper look.
Comment 7 Allon Mureinik 2017-06-29 04:48:36 EDT
Daniel, can you take a look please?
Comment 8 Daniel Erez 2017-06-29 05:33:38 EDT
According to the logs [1], the same mounting is working well in host rhev4.lab.msp.redhat.com. Is there any difference between the two hosts (rhev4.lab.msp.redhat.com and rhev5.lab.msp.redhat.com)?
I.e. firewall/selinux/os/etc ?

[1]
2017-06-28 09:25:56,131-0500 INFO  (jsonrpc/6) [dispatcher] Run and protect: connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'mnt_options': u'port=20049,proto=rdma', u'id': u'0e247058-90eb-4e34-8e10-faed3a1da0fe', u'connection': u'tatooine-ib0.lab.msp.redhat.com:/storage/export2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'4.2', u'password': '********', u'port': u''}], options=None) (logUtils:51)
2017-06-28 09:25:56,133-0500 INFO  (jsonrpc/6) [storage.Mount] mounting tatooine-ib0.lab.msp.redhat.com:/storage/export2 at /rhev/data-center/mnt/tatooine-ib0.lab.msp.redhat.com:_storage_export2 (mount:187)
2017-06-28 09:25:56,236-0500 INFO  (jsonrpc/6) [IOProcessClient] Closing client ioprocess-1 (__init__:593)
2017-06-28 09:25:56,255-0500 INFO  (jsonrpc/6) [IOProcessClient] Closing client ioprocess-2 (__init__:593)
2017-06-28 09:25:56,256-0500 INFO  (jsonrpc/6) [dispatcher] Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': u'0e247058-90eb-4e34-8e10-faed3a1da0fe'}]} (logUtils:54)
Comment 9 Sam Yangsao 2017-06-29 10:03:39 EDT
(In reply to Allon Mureinik from comment #5)
> Something's odd here.
> In VDSM's log I can see
> 
> MountError: (32, ';mount.nfs: Failed to resolve server tatooine-ib0: Name or
> service not known\n')
> 
> Which both seems to work manually according to comment 4 (altghough the
> credentials seem wrong, so I can't check myself), and from the other host.
> 
> Need to give this a deeper look.

Yes, this dns was broken earlier on rhev5 since there was an incorrect entry, I have fixed this
Comment 10 Sam Yangsao 2017-06-29 10:06:36 EDT
(In reply to Daniel Erez from comment #8)
> According to the logs [1], the same mounting is working well in host
> rhev4.lab.msp.redhat.com. Is there any difference between the two hosts
> (rhev4.lab.msp.redhat.com and rhev5.lab.msp.redhat.com)?
> I.e. firewall/selinux/os/etc ?
> 
> [1]
> 2017-06-28 09:25:56,131-0500 INFO  (jsonrpc/6) [dispatcher] Run and protect:
> connectStorageServer(domType=1,
> spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'mnt_options':
> u'port=20049,proto=rdma', u'id': u'0e247058-90eb-4e34-8e10-faed3a1da0fe',
> u'connection': u'tatooine-ib0.lab.msp.redhat.com:/storage/export2', u'iqn':
> u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'4.2', u'password':
> '********', u'port': u''}], options=None) (logUtils:51)
> 2017-06-28 09:25:56,133-0500 INFO  (jsonrpc/6) [storage.Mount] mounting
> tatooine-ib0.lab.msp.redhat.com:/storage/export2 at
> /rhev/data-center/mnt/tatooine-ib0.lab.msp.redhat.com:_storage_export2
> (mount:187)
> 2017-06-28 09:25:56,236-0500 INFO  (jsonrpc/6) [IOProcessClient] Closing
> client ioprocess-1 (__init__:593)
> 2017-06-28 09:25:56,255-0500 INFO  (jsonrpc/6) [IOProcessClient] Closing
> client ioprocess-2 (__init__:593)
> 2017-06-28 09:25:56,256-0500 INFO  (jsonrpc/6) [dispatcher] Run and protect:
> connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id':
> u'0e247058-90eb-4e34-8e10-faed3a1da0fe'}]} (logUtils:54)

firewalld is actually turned off

SELINUX is turned on and the OS is built exactly the same and configured the same, with the exception of an incorrect dns entry on rhev5 which I fixed yesterday.

NOTE:  This mounts and works on both servers when the mount is done manually (proto=rdma,port=20049)
Comment 11 Daniel Erez 2017-06-29 10:15:55 EDT
(In reply to Sam Yangsao from comment #9)
> (In reply to Allon Mureinik from comment #5)
> > Something's odd here.
> > In VDSM's log I can see
> > 
> > MountError: (32, ';mount.nfs: Failed to resolve server tatooine-ib0: Name or
> > service not known\n')
> > 
> > Which both seems to work manually according to comment 4 (altghough the
> > credentials seem wrong, so I can't check myself), and from the other host.
> > 
> > Need to give this a deeper look.
> 
> Yes, this dns was broken earlier on rhev5 since there was an incorrect
> entry, I have fixed this

Not sure I follow. So currently it works fine on rhev4 but not on rhev5? 
Have you tried to restart vdsm service on rhev5?
Comment 12 Sam Yangsao 2017-07-05 17:08:04 EDT
(In reply to Daniel Erez from comment #11)
> (In reply to Sam Yangsao from comment #9)
> > (In reply to Allon Mureinik from comment #5)
> > > Something's odd here.
> > > In VDSM's log I can see
> > > 
> > > MountError: (32, ';mount.nfs: Failed to resolve server tatooine-ib0: Name or
> > > service not known\n')
> > > 
> > > Which both seems to work manually according to comment 4 (altghough the
> > > credentials seem wrong, so I can't check myself), and from the other host.
> > > 
> > > Need to give this a deeper look.
> > 
> > Yes, this dns was broken earlier on rhev5 since there was an incorrect
> > entry, I have fixed this
> 
> Not sure I follow. So currently it works fine on rhev4 but not on rhev5? 
> Have you tried to restart vdsm service on rhev5?

It fails on both when trying to mount on either node via the UI.

Mounting manually via command line works fine.

I've restarted both hosts - same results.
Comment 13 Daniel Erez 2017-07-05 17:42:44 EDT
(In reply to Sam Yangsao from comment #12)
> (In reply to Daniel Erez from comment #11)
> > (In reply to Sam Yangsao from comment #9)
> > > (In reply to Allon Mureinik from comment #5)
> > > > Something's odd here.
> > > > In VDSM's log I can see
> > > > 
> > > > MountError: (32, ';mount.nfs: Failed to resolve server tatooine-ib0: Name or
> > > > service not known\n')
> > > > 
> > > > Which both seems to work manually according to comment 4 (altghough the
> > > > credentials seem wrong, so I can't check myself), and from the other host.
> > > > 
> > > > Need to give this a deeper look.
> > > 
> > > Yes, this dns was broken earlier on rhev5 since there was an incorrect
> > > entry, I have fixed this
> > 
> > Not sure I follow. So currently it works fine on rhev4 but not on rhev5? 
> > Have you tried to restart vdsm service on rhev5?
> 
> It fails on both when trying to mount on either node via the UI.
> 
> Mounting manually via command line works fine.
> 
> I've restarted both hosts - same results.

Can you please upload failure logs from both hosts (because it seems to succeed according to rhev4 host log).
Comment 14 Sam Yangsao 2017-07-06 10:47:13 EDT
When it fails on both nodes via the UI, it gets in a stuck in the UI, there is no way to remove it at all to clean things up.

For example, I created the storage domain first with the NFS options:

proto=rdma,port=20049

The UI fails to mount this.  Now it's stuck in some sort of loop.  I can't remove the failure and the UI only allows me to Attach the data center (even though it does not attach)

Data Center > HP > Storage > Attach Data > tatooine-export2 

Test mounting via command line on *BOTH* nodes works abosolutely fine using the NFS options above.

The servers are all on the RH network, see comment#4 above.

I've rerun the ovirt log collector for this mornings attempt again.  Will update the next comments on where the file is.
Comment 17 Daniel Erez 2017-07-10 05:01:36 EDT
We're currently not supporting nfs 4.2 since it as issues with selinux and live migration (see https://bugzilla.redhat.com/show_bug.cgi?id=1464787).

Please try to follow the steps described in https://bugzilla.redhat.com/show_bug.cgi?id=1464787#c13 
to check whether it is a selinux issue.
Comment 18 Sam Yangsao 2017-07-17 12:16:23 EDT
(In reply to Daniel Erez from comment #17)
> We're currently not supporting nfs 4.2 since it as issues with selinux and
> live migration (see https://bugzilla.redhat.com/show_bug.cgi?id=1464787).
> 

I did not know this, thanks for confirming this.

> Please try to follow the steps described in
> https://bugzilla.redhat.com/show_bug.cgi?id=1464787#c13 
> to check whether it is a selinux issue.

Testing with selinux turned off works - this was not an issue with RHV 4.0, not sure what has changed between releases.

NOTE this is on a RHEL-H that's freshly built with RHV 4.1 (not an upgrade)

I'll test again with selinux turned on to see if I can dump out the selinux messages here for review.

Thanks!
Comment 19 Yaniv Kaul 2017-08-15 09:55:54 EDT
(In reply to Daniel Erez from comment #17)
> We're currently not supporting nfs 4.2 since it as issues with selinux and
> live migration (see https://bugzilla.redhat.com/show_bug.cgi?id=1464787).

That's only against Linux storage servers. There's no dependency here.
> 
> Please try to follow the steps described in
> https://bugzilla.redhat.com/show_bug.cgi?id=1464787#c13 
> to check whether it is a selinux issue.
Comment 20 Yaniv Kaul 2017-10-01 08:05:51 EDT
(In reply to Sam Yangsao from comment #18)
> (In reply to Daniel Erez from comment #17)
> > We're currently not supporting nfs 4.2 since it as issues with selinux and
> > live migration (see https://bugzilla.redhat.com/show_bug.cgi?id=1464787).
> > 
> 
> I did not know this, thanks for confirming this.
> 
> > Please try to follow the steps described in
> > https://bugzilla.redhat.com/show_bug.cgi?id=1464787#c13 
> > to check whether it is a selinux issue.
> 
> Testing with selinux turned off works - this was not an issue with RHV 4.0,
> not sure what has changed between releases.
> 
> NOTE this is on a RHEL-H that's freshly built with RHV 4.1 (not an upgrade)
> 
> I'll test again with selinux turned on to see if I can dump out the selinux
> messages here for review.
> 

Have you tested it selinux? Can you please share the results?

> Thanks!
Comment 21 Sam Yangsao 2017-10-02 19:16:22 EDT
(In reply to Yaniv Kaul from comment #20)
> (In reply to Sam Yangsao from comment #18)
> > (In reply to Daniel Erez from comment #17)
> > > We're currently not supporting nfs 4.2 since it as issues with selinux and
> > > live migration (see https://bugzilla.redhat.com/show_bug.cgi?id=1464787).
> > > 
> > 
> > I did not know this, thanks for confirming this.
> > 
> > > Please try to follow the steps described in
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1464787#c13 
> > > to check whether it is a selinux issue.
> > 
> > Testing with selinux turned off works - this was not an issue with RHV 4.0,
> > not sure what has changed between releases.
> > 
> > NOTE this is on a RHEL-H that's freshly built with RHV 4.1 (not an upgrade)
> > 
> > I'll test again with selinux turned on to see if I can dump out the selinux
> > messages here for review.
> > 
> 
> Have you tested it selinux? Can you please share the results?
> 
> > Thanks!

Sorry, I have not yet recently as my reproducer is currently in-use for other tasks.

One thing I did notice recently.

I moved my storage domain in between clusters.

1.  cluster 1 (HP-Cluster) had the following options to the nfs data store "vers=4.1,proto=rdma,port=20049"

2.  I put the nfs data store in maintenace mode and remove it from the cluster

3.  I go ahead and re-import this nfs data store back into a new cluster (Dell -Cluster) with the options "vers=4.1,proto=rdma,port=20049"

RHV mounts ths nfs data store via "proto=tcp" instead, and _not_ "proto=rdma" - keep in mind this cluster is built brand new with selinux turned _off_

I was assuming the proto issue was in between RHV versions, looks like it's not accepting the custom protocol when I choose to import.

This reproducer is on the network for review as well.

Thanks much.
Comment 22 Daniel Erez 2017-12-12 09:40:40 EST
Hi Sam,

Is the issue still reproducible when creating a new domain or is it affecting merely the import flow? Can you please attach full logs if reproducible? 

Thanks!

Note You need to log in before you can comment on or make changes to this bug.