Bug 1468911

Summary:

Failed to create NFS storage domain on ceph nfs backend on path "/"

Product:

[oVirt] ovirt-engine

Reporter:

Kevin Alon Goldblatt <kgoldbla>

Component:

BLL.Storage

Assignee:

Benny Zlotnik <bzlotnik>

Status:

CLOSED NOTABUG

QA Contact:

Raz Tamir <ratamir>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

4.1.3.4

CC:

bugs, kgoldbla, tnisan

Target Milestone:

ovirt-4.2.0

Keywords:

Reopened

Target Release:

---

Flags:

rule-engine: ovirt-4.2+

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

storage

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2017-08-28 14:27:33 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

Storage

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
vdsm, server, engine and screen shots	none

Description Kevin Alon Goldblatt 2017-07-09 20:03:50 UTC

Description of problem:
Creating an nfs domain on the ceph backend fails with on path "/"

Version-Release number of selected component (if applicable):
ovirt-engine-4.1.3.4-0.1.el7.noarch
rhevm-4.1.3.4-0.1.el7.noarch
vdsm-4.19.19-1.el7ev.x86_64

How reproducible:
100%



Steps to Reproduce:
1. Create an ganesha-nfs instant on a ganesha cluster node. The ganesha name space is on "/"
2.From my vdsm host (blond-vdsf) I am able to manually mount the ganesha instance on green as follows:
a) mount -t nfs -o nfsvers=4.1,noauto,soft,sync,proto=tcp green-vdsc.qa.lab.tlv.redhat.com:/ /mnt/nfs
b) I can create directories through the mount
c) I am also able to mount a subdirectory with chown 36:36 permissions from the host as follows:
d) mount -t nfs -o nfsvers=4.1,noauto,soft,sync,proto=tcp green-vdsc.qa.lab.tlv.redhat.com:/ceph_nfs3 /mnt/nfs

When I try and create the nfs storage via the Rhev admin interface on the same paths it fails as follows:
e) When I use path "green-vdsc.qa.lab.tlv.redhat.com:/ " as in a) the GUI does not accept the path as being semantically correct

f) On the ganesh cluster member the contents of the /ceph_nfs3 directory is as follows:

[root@green-vdsc ceph_nfs3]# ll
total 0
drwxr-xr-x. 1 vdsm kvm 0 Jun 28 15:52 d2d4ce1b-558a-442a-ba6b-e9bf94b9ad1f
-rwxr-xr-x. 1 vdsm kvm 0 Jun 28 15:52 __DIRECT_IO_TEST__

Actual results:
The create nfs domain fails 

Expected results:
The nfs domain should create successfully on the ganesha-nfs mount point

Additional info:

vdsm.log:
-----------
2017-07-02 12:38:51,822+0300 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.00 seconds (__init__:539)
2017-07-02 12:38:52,276+0300 INFO  (jsonrpc/6) [vdsm.api] START createStorageDomain(storageType=1, sdUUID='02ede875-f7a4-4555-9d2a-aea93e25b9d5', domainName='ceph_nfs', typeSpecificArg='green-vdsc.qa.lab.tlv.red
hat.com:/ceph_nfs3', domClass=1, domVersion='4', options=None) from=::ffff:10.35.161.23,51616, flow_id=1fd014db (api:46)
2017-07-02 12:38:52,586+0300 INFO  (itmap/0) [IOProcessClient] Starting client ioprocess-14 (__init__:325)
2017-07-02 12:38:52,610+0300 INFO  (ioprocess communication (25942)) [IOProcess] Starting ioprocess (__init__:447)
2017-07-02 12:38:52,612+0300 INFO  (jsonrpc/6) [storage.StorageDomain] sdUUID=02ede875-f7a4-4555-9d2a-aea93e25b9d5 domainName=ceph_nfs remotePath=green-vdsc.qa.lab.tlv.redhat.com:/ceph_nfs3 domClass=1 (nfsSD:70)
2017-07-02 12:38:52,644+0300 INFO  (jsonrpc/6) [IOProcessClient] Starting client ioprocess-15 (__init__:325)
2017-07-02 12:38:52,668+0300 INFO  (ioprocess communication (25952)) [IOProcess] Starting ioprocess (__init__:447)
2017-07-02 12:38:54,221+0300 INFO  (jsonrpc/6) [storage.xlease] Formatting index for lockspace '02ede875-f7a4-4555-9d2a-aea93e25b9d5' (version=1) (xlease:647)
2017-07-02 12:38:54,255+0300 INFO  (jsonrpc/6) [vdsm.api] FINISH createStorageDomain error=[Errno 5] Input/output error from=::ffff:10.35.161.23,51616, flow_id=1fd014db (api:50)
2017-07-02 12:38:54,256+0300 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='b74eadfb-8a8f-4bd8-a004-903e05bc7139') Unexpected error (task:870)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 877, in _run
    return fn(*args, **kargs)
  File "<string>", line 2, in createStorageDomain
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
    ret = func(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2573, in createStorageDomain
    storageType, domVersion)
  File "/usr/share/vdsm/storage/nfsSD.py", line 87, in create
    remotePath, storageType, version)
  File "/usr/share/vdsm/storage/fileSD.py", line 436, in _prepareMetadata
    cls.format_external_leases(sdUUID, xleases_path)
  File "/usr/share/vdsm/storage/sd.py", line 1140, in format_external_leases
    xlease.format_index(lockspace, backend)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 659, in format_index
    index.dump(file)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 765, in dump
    file.write(self._buf)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 881, in write
    pos += uninterruptible(self._file.write, wbuf)
  File "/usr/lib/python2.7/site-packages/vdsm/common/osutils.py", line 56, in uninterruptible
    return func(*args, **kwargs)
IOError: [Errno 5] Input/output error
2017-07-02 12:38:54,257+0300 INFO  (jsonrpc/6) [storage.TaskManager.Task] (Task='b74eadfb-8a8f-4bd8-a004-903e05bc7139') aborting: Task is aborted: u'[Errno 5] Input/output error' - code 100 (task:1175)
2017-07-02 12:38:54,258+0300 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH createStorageDomain error=[Errno 5] Input/output error (dispatcher:82)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper
    result = ctask.prepare(func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 105, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1183, in prepare
    raise self.error
IOError: [Errno 5] Input/output error


ENGINE.LOG
--------------------------------------------------
2017-07-02 12:38:54,638+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-3) [1fd014db] Failed in 'CreateStorageDomainVDS' method
2017-07-02 12:38:54,652+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-3) [1fd014db] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM blond-vdsf.qa.lab.tlv.redhat.com command CreateStorageDomainVDS failed: Error creating a storage domain: ('storageType=1, sdUUID=02ede875-f7a4-4555-9d2a-aea93e25b9d5, domainName=ceph_nfs, domClass=1, typeSpecificArg=green-vdsc.qa.lab.tlv.redhat.com:/ceph_nfs3 domVersion=4',)
2017-07-02 12:38:54,652+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-3) [1fd014db] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand' return value 'StatusOnlyReturn [status=Status [code=351, message=Error creating a storage domain: ('storageType=1, sdUUID=02ede875-f7a4-4555-9d2a-aea93e25b9d5, domainName=ceph_nfs, domClass=1, typeSpecificArg=green-vdsc.qa.lab.tlv.redhat.com:/ceph_nfs3 domVersion=4',)]]'
2017-07-02 12:38:54,652+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-3) [1fd014db] HostName = blond-vdsf.qa.lab.tlv.redhat.com
2017-07-02 12:38:54,652+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-3) [1fd014db] Command 'CreateStorageDomainVDSCommand(HostName = blond-vdsf.qa.lab.tlv.redhat.com, CreateStorageDomainVDSCommandParameters:{runAsync='true', hostId='e4d1e9e3-68d6-437f-9f91-87af5ef98099', storageDomain='StorageDomainStatic:{name='ceph_nfs', id='02ede875-f7a4-4555-9d2a-aea93e25b9d5'}', args='green-vdsc.qa.lab.tlv.redhat.com:/ceph_nfs3'})' execution failed: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: ('storageType=1, sdUUID=02ede875-f7a4-4555-9d2a-aea93e25b9d5, domainName=ceph_nfs, domClass=1, typeSpecificArg=green-vdsc.qa.lab.tlv.redhat.com:/ceph_nfs3 domVersion=4',), code = 351
2017-07-02 12:38:54,652+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-3) [1fd014db] FINISH, CreateStorageDomainVDSCommand, log id: 3466e46b
2017-07-02 12:38:54,653+03 ERROR [org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand] (default task-3) [1fd014db] Command 'org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: ('storageType=1, sdUUID=02ede875-f7a4-4555-9d2a-aea93e25b9d5, domainName=ceph_nfs, domClass=1, typeSpecificArg=green-vdsc.qa.lab.tlv.redhat.com:/ceph_nfs3 domVersion=4',), code = 351 (Failed with error StorageDomainCreationError and code 351)
2017-07-02 12:38:54,657+03 INFO  [org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand] (default task-3) [1fd014db] Command [id=56dc010f-2795-4fc6-b644-0954ed37b6cc]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.profiles.DiskProfile; snapshot: c868e2b8-5f28-49d6-a6af-d962843060d0.
2017-07-02 12:38:54,659+03 INFO  [org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand] (default task-3) [1fd014db] Command [id=56dc010f-2795-4fc6-b644-0954ed37b6cc]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; snapshot: 02ede875-f7a4-4555-9d2a-aea93e25b9d5.
2017-07-02 12:38:54,659+03 INFO  [org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand] (default task-3) [1fd014db] Command [id=56dc010f-2795-4fc6-b644-0954ed37b6cc]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: 02ede875-f7a4-4555-9d2a-aea93e25b9d5.
2017-07-02 12:38:54,671+03 ERROR [org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand] (default task-3) [1fd014db] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand'.
2017-07-02 12:38:54,681+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-3) [1fd014db] EVENT_ID: USER_ADD_STORAGE_DOMAIN_FAILED(957), Correlation ID: 974b5129-ed20-419e-8547-26e28e491cca, Job ID: 998d279f-49b0-415e-89fe-3f8496d34fc6, Call Stack: null, Custom Event ID: -1, Message: Failed to add Storage Domain ceph_nfs. (User: admin@internal-authz)
2017-07-02 12:38:55,206+03 INFO  [org.ovirt.engine.core.bll.storage.connection.RemoveStorageServerConnectionCommand] (default task-4) [6cd42798-9eb6-42f7-8856-96da1c525ea1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[ffe47d0c-9ec2-4d4f-9358-6d6960cbe8e8=STORAGE_CONNECTION, green-vdsc.qa.lab.tlv.redhat.com:/ceph_nfs3=STORAGE_CONNECTION]', sharedLocks=''}'
2017-07-02 12:38:55,290+03 INFO  [org.ovirt.engine.core.bll.storage.connection.RemoveStorageServerConnectionCommand] (default task-4) [6cd42798-9eb6-42f7-8856-96da1c525ea1] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-07-02 12:38:55,311+03 INFO  [org.ovirt.engine.core.bll.storage.connection.RemoveStorageServerConnectionCommand] (default task-4) [6cd42798-9eb6-42f7-8856-96da1c525ea1] Removing connection 'ffe47d0c-9ec2-4d4f-9358-6d6960cbe8e8' from database 
2017-07-02 12:38:55,312+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-4) [6cd42798-9eb6-42f7-8856-96da1c525ea1] START, DisconnectStorageServerVDSCommand(HostName = blond-vdsf.qa.lab.tlv.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e4d1e9e3-68d6-437f-9f91-87af5ef98099', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='ffe47d0c-9ec2-4d4f-9358-6d6960cbe8e8', connection='green-vdsc.qa.lab.tlv.redhat.com:/ceph_nfs3', iqn='null', vfsType='null', mountOptions='null', nfsVersion='AUTO', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3790a61b
2017-07-02 12:38:56,628+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-4) [6cd42798-9eb6-42f7-8856-96da1c525ea1] FINISH, DisconnectStorageServerVDSCommand, return: {ffe47d0c-9ec2-4d4f-9358-6d6960cbe8e8=0}, log id: 3790a61b
2017-07-02 12:38:56,638+03 INFO  [org.ovirt.engine.core.bll.storage.connection.RemoveStorageServerConnectionCommand] (default task-4) [6cd42798-9eb6-42f7-8856-96da1c525ea1] Lock freed to object 'EngineLock:{exclusiveLocks='[ffe47d0c-9ec2-4d4f-9358-6d6960cbe8e8=STORAGE_CONNECTION, green-vdsc.qa.lab.tlv.redhat.com:/ceph_nfs3=STORAGE_CONNECTION]', sharedLocks=''}'
(END)

Comment 1 Kevin Alon Goldblatt 2017-07-09 20:04:45 UTC

Created attachment 1295642 [details]
vdsm, server, engine and screen shots

Adding logs

Comment 2 Tal Nisan 2017-07-10 10:05:35 UTC


*** This bug has been marked as a duplicate of bug 1425879 ***

Comment 3 Kevin Alon Goldblatt 2017-07-11 15:52:15 UTC

After making implementing the changes to the ganesha.conf file to support the pseudo path, the create domain operation now fails with 


vdsm.log:
---------------------------------
2017-07-11 18:31:39,138+0300 INFO  (itmap/0) [IOProcessClient] Starting client ioprocess-1 (__init__:325)
2017-07-11 18:31:39,160+0300 INFO  (ioprocess communication (10901)) [IOProcess] Starting ioprocess (__init__:447)
2017-07-11 18:31:39,162+0300 INFO  (jsonrpc/0) [storage.StorageDomain] sdUUID=6968c23b-ac37-4783-8098-0a435bc299f2 domainName=ceph5_nfs1 remotePath=green-vdsc.qa.lab.tlv.redhat.com:/ceph5/nfs1 domClass=1 (nfsSD:
70)
2017-07-11 18:31:39,184+0300 INFO  (jsonrpc/0) [IOProcessClient] Starting client ioprocess-2 (__init__:325)
2017-07-11 18:31:39,207+0300 INFO  (ioprocess communication (10911)) [IOProcess] Starting ioprocess (__init__:447)
2017-07-11 18:31:39,211+0300 ERROR (jsonrpc/0) [storage.fileSD] Underlying file system doesn't supportdirect IO (fileSD:102)
2017-07-11 18:31:39,211+0300 INFO  (jsonrpc/0) [vdsm.api] FINISH createStorageDomain error=Storage Domain target is unsupported: () from=::ffff:10.35.161.23,56492, flow_id=3072551, task_id=4571d4d5-4a86-43ed-858e-6f0ef6180e7a (api:50)
2017-07-11 18:31:39,212+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='4571d4d5-4a86-43ed-858e-6f0ef6180e7a') Unexpected error (task:870)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 877, in _run
    return fn(*args, **kargs)
  File "<string>", line 2, in createStorageDomain
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
    ret = func(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2573, in createStorageDomain
    storageType, domVersion)
  File "/usr/share/vdsm/storage/nfsSD.py", line 83, in create
    version)
  File "/usr/share/vdsm/storage/nfsSD.py", line 50, in _preCreateValidation
    fileSD.validateFileSystemFeatures(sdUUID, domPath)
  File "/usr/share/vdsm/storage/fileSD.py", line 104, in validateFileSystemFeatures
    raise se.StorageDomainTargetUnsupported()
StorageDomainTargetUnsupported: Storage Domain target is unsupported: ()
2017-07-11 18:31:39,214+0300 INFO  (jsonrpc/0) [storage.TaskManager.Task] (Task='4571d4d5-4a86-43ed-858e-6f0ef6180e7a') aborting: Task is aborted: 'Storage Domain target is unsupported' - code 399 (task:1175)
2017-07-11 18:31:39,214+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH createStorageDomain error=Storage Domain target is unsupported: () (dispatcher:78)
2017-07-11 18:31:39,215+0300 INFO  (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StorageDomain.create failed (error 399) in 0.38 seconds (__init__:539)
2017-07-11 18:31:40,429+0300 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)
2017-07-11 18:31:41,941+0300 INFO  (jsonrpc/6) [vdsm.api] START disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'id': '52ad14df-a937-4213-80f2-0a795df3e11b', 'connection': 'green-vdsc.qa.lab.tlv.redhat.com:/ceph5/nfs1', 'iqn': u'', 'user': u'', 'tpgt': '1', 'protocol_version': 'auto', 'password': '********', 'port': u''}], options=None) from=::ffff:10.35.161.23,56492, flow_id=42683dd8-d6f7-4e7a-b540-ba8d9cf46064, task_id=7197e220-519e-4d99-a235-29b7ef2623b2 (api:46)
2017-07-11 18:31:41,942+0300 INFO  (jsonrpc/6) [storage.Mount] unmounting /rhev/data-center/mnt/green-vdsc.qa.lab.tlv.redhat.com:_ceph5_nfs1 (mount:195)
2017-07-11 18:31:42,236+0300 INFO  (jsonrpc/6) [vdsm.api] FINISH disconnectStorageServer return={'statuslist': [{'status': 0, 'id': '52ad14df-a937-4213-80f2-0a795df3e11b'}]} from=::ffff:10.35.161.23,56492, flow_id=42683dd8-d6f7-4e7a-b540-ba8d9cf46064, task_id=7197e220-519e-4d99-a235-29b7ef2623b2 (api:52)
2017-07-11 18:31:42,237+0300 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.disconnectStorageServer succeeded in 0.30 seconds (__init__:539)
2017-07-11 18:31:48,138+0300 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)
2017-07-11 18:31:49,227+0300 INFO  (periodic/1) [vdsm.api] START repoStats(options=None) from=internal, task_id=f684d017-c7f2-4f29-ae52-0b740200762c (api:46)
2017-07-11 18:31:49,228+0300 INFO  (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=f684d017-c7f2-4f29-ae52-0b740200762c (api:52)
2017-07-11 18:31:54,326+0300 INFO  (jsonrpc/3) [vdsm.api] START repoStats(options=None) from=::ffff:10.35.161.23,56492, flow_id=5437a9e2, task_id=170b0312-a777-489c-bdab-f68997bf430f (api:46)
2017-07-11 18:31:54,327+0300 INFO  (jsonrpc/3) [vdsm.api] FINISH repoStats return={} from=::ffff:10.35.161.23,56492, flow_id=5437a9e2, task_id=170b0312-a777-489c-bdab-f68997bf430f (api:52)
2017-07-11 18:31:54,336+0300 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:539)
2017-07-11 18:31:56,445+0300 INFO  (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:539)
2017-07-11 18:32:03,157+0300 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] 696 requests processed during 3600 seconds (__init__:526)






engine.log:
---------------------------------
2017-07-11 18:31:38,042+03 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [3072551] EVENT_ID: USER_ADDED_DISK_PROFILE(10,120), Correlation ID: 3072551, Job ID: 5cd2
a270-ca76-4d2e-bc01-b0be93d57687, Call Stack: null, Custom Event ID: -1, Message: Disk Profile ceph5_nfs1 was successfully added (User: admin@internal-authz).
2017-07-11 18:31:38,057+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-1) [3072551] START, ConnectStorageServerVDSCommand(HostName = blond-vdsf.qa.lab.tlv.redha
t.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='2cc96c60-9041-4174-9708-d0e9fee3a948', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[St
orageServerConnections:{id='52ad14df-a937-4213-80f2-0a795df3e11b', connection='green-vdsc.qa.lab.tlv.redhat.com:/ceph5/nfs1', iqn='null', vfsType='null', mountOptions='null', nfsVersion='AUTO', nfsRetrans='null'
, nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 69dca48b
2017-07-11 18:31:38,820+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-1) [3072551] FINISH, ConnectStorageServerVDSCommand, return: {52ad14df-a937-4213-80f2-0a7
95df3e11b=0}, log id: 69dca48b
2017-07-11 18:31:38,826+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [3072551] START, CreateStorageDomainVDSCommand(HostName = blond-vdsf.qa.lab.tlv.redhat.
com, CreateStorageDomainVDSCommandParameters:{runAsync='true', hostId='2cc96c60-9041-4174-9708-d0e9fee3a948', storageDomain='StorageDomainStatic:{name='ceph5_nfs1', id='6968c23b-ac37-4783-8098-0a435bc299f2'}', a
rgs='green-vdsc.qa.lab.tlv.redhat.com:/ceph5/nfs1'}), log id: 78caf416
2017-07-11 18:31:39,319+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [3072551] Failed in 'CreateStorageDomainVDS' method
2017-07-11 18:31:39,362+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [3072551] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: 
null, Custom Event ID: -1, Message: VDSM blond-vdsf.qa.lab.tlv.redhat.com command CreateStorageDomainVDS failed: Storage Domain target is unsupported: ()
2017-07-11 18:31:39,363+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [3072551] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSComm
and' return value 'StatusOnlyReturn [status=Status [code=399, message=Storage Domain target is unsupported: ()]]'
2017-07-11 18:31:39,363+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [3072551] HostName = blond-vdsf.qa.lab.tlv.redhat.com
2017-07-11 18:31:39,363+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [3072551] Command 'CreateStorageDomainVDSCommand(HostName = blond-vdsf.qa.lab.tlv.redha
t.com, CreateStorageDomainVDSCommandParameters:{runAsync='true', hostId='2cc96c60-9041-4174-9708-d0e9fee3a948', storageDomain='StorageDomainStatic:{name='ceph5_nfs1', id='6968c23b-ac37-4783-8098-0a435bc299f2'}',
 args='green-vdsc.qa.lab.tlv.redhat.com:/ceph5/nfs1'})' execution failed: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Storage Domain target is unsupported: (), code = 399
2017-07-11 18:31:39,363+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [3072551] FINISH, CreateStorageDomainVDSCommand, log id: 78caf416
2017-07-11 18:31:39,363+03 ERROR [org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand] (default task-1) [3072551] Command 'org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand' fai
led: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Storage Domain target is unsupported: (), code
 = 399 (Failed with error StorageDomainTargetUnsupported and code 399)
2017-07-11 18:31:39,402+03 INFO  [org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand] (default task-1) [3072551] Command [id=3ab3e9a1-2114-436b-a0f2-1a8933aa3afc]: Compensating NEW_ENTITY_ID of 
org.ovirt.engine.core.common.businessentities.profiles.DiskProfile; snapshot: 9a3d7446-9901-4143-a437-d2d0b4d4b0ab.
2017-07-11 18:31:39,430+03 INFO  [org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand] (default task-1) [3072551] Command [id=3ab3e9a1-2114-436b-a0f2-1a8933aa3afc]: Compensating NEW_ENTITY_ID of 
org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; snapshot: 6968c23b-ac37-4783-8098-0a435bc299f2.
2017-07-11 18:31:39,451+03 INFO  [org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand] (default task-1) [3072551] Command [id=3ab3e9a1-2114-436b-a0f2-1a8933aa3afc]: Compensating NEW_ENTITY_ID of 
org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: 6968c23b-ac37-4783-8098-0a435bc299f2.
2017-07-11 18:31:39,493+03 ERROR [org.ovirt.engine.core.bll.storage.domain.AddNFSStorageDomainCommand] (default task-1) [3072551] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.domain.Add
NFSStorageDomainCommand'.
2017-07-11 18:31:39,637+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [3072551] EVENT_ID: USER_ADD_STORAGE_DOMAIN_FAILED(957), Correlation ID: daff7990-e4be-46be-a5b1-849ed2dca864, Job ID: 5cd2a270-ca76-4d2e-bc01-b0be93d57687, Call Stack: null, Custom Event ID: -1, Message: Failed to add Storage Domain ceph5_nfs1. (User: admin@internal-authz)
2017-07-11 18:31:41,844+03 INFO  [org.ovirt.engine.core.bll.storage.connection.RemoveStorageServerConnectionCommand] (default task-4) [42683dd8-d6f7-4e7a-b540-ba8d9cf46064] Lock Acquired to object 'EngineLock:{exclusiveLocks='[52ad14df-a937-4213-80f2-0a795df3e11b=STORAGE_CONNECTION, green-vdsc.qa.lab.tlv.redhat.com:/ceph5/nfs1=STORAGE_CONNECTION]', sharedLocks=''}'
2017-07-11 18:31:41,906+03 INFO  [org.ovirt.engine.core.bll.storage.connection.RemoveStorageServerConnectionCommand] (default task-4) [42683dd8-d6f7-4e7a-b540-ba8d9cf46064] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-07-11 18:31:41,936+03 INFO  [org.ovirt.engine.core.bll.storage.connection.RemoveStorageServerConnectionCommand] (default task-4) [42683dd8-d6f7-4e7a-b540-ba8d9cf46064] Removing connection '52ad14df-a937-4213-80f2-0a795df3e11b' from database

Comment 4 Yaniv Kaul 2017-07-15 01:58:46 UTC

Did you see the error:
"Underlying file system doesn't supportdirect IO" ?

Can you check if your NFS implementation supports Direct IO?