Bug 1245147

Created attachment 1054288 [details]
vdsm log

New vdsm log attached

From the logs:

hread-61097::DEBUG::2015-07-21 08:05:00,255::mount::227::Storage.Misc.excCmd::(_runcmd) /usr/bin/sudo -n /usr/bin/mount -t glusterfs 10.35.160.6:/compute_volume01 /rhev/data-center/mnt/glusterSD/10.35.160.6:_compute__volume01 (cwd None)
Thread-61097::ERROR::2015-07-21 08:05:00,383::storageServer::213::Storage.StorageServer.MountConnection::(connect) Mount failed: (1, ';Mount failed. Please check the log file for more details.\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 211, in connect
  File "/usr/share/vdsm/storage/mount.py", line 223, in mount
  File "/usr/share/vdsm/storage/mount.py", line 239, in _runcmd
MountError: (1, ';Mount failed. Please check the log file for more details.\n')

Raz - please attach the relevant gluster log too.

Anyway, from trying to mount the same volume on my machine, I received the following error:

2015-07-21 11:01:06.728960] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-compute_volume01-client-0: changing port to 49248 (from 0)
[2015-07-21 11:01:06.733607] E [socket.c:2332:socket_connect_finish] 0-compute_volume01-client-0: connection to 10.35.160.6:49248 failed (Connection refused)
[2015-07-21 11:01:06.737247] I [fuse-bridge.c:5086:fuse_graph_setup] 0-fuse: switched to graph 0
[2015-07-21 11:01:06.737779] I [fuse-bridge.c:4012:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.23
[2015-07-21 11:01:06.738520] W [fuse-bridge.c:780:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Transport endpoint is not connected)
[2015-07-21 11:01:06.760107] I [fuse-bridge.c:4933:fuse_thread_proc] 0-fuse: unmounting /root/raz/g
[2015-07-21 11:01:06.760611] W [glusterfsd.c:1219:cleanup_and_exit] (--> 0-: received signum (15), shutting down

This seems either like a configuration with the machine's firewall or a change in gluster's packages.

gluster log attached.
From quick look it seems that the operation didn't reach to gluster

Created attachment 1054305 [details]
gluster log

Are both the volumes on the same host, i.e the first storage domain and the second storage domain - are they connecting to gluster volumes on the same hosts?
If not, are the ports open for the second storage domain's volume (on 10.35.160.6)?

Can you provide "gluster volume status compute_volume01" output?

After fixing gluster configuration issues we do not see it happens anymore, in case we will, we will change severity back.

For now we are not closing as ovirt community member face that issue as well.

(In reply to Aharon Canan from comment #6)
> For now we are not closing as ovirt community member face that issue as well.
The attached logs are from the gluster setup with the aforementioned configuration issues, iiuc.
Can we get the logs from teh issue you're referencing here please?

Created attachment 1057050 [details]
user logs

user logs attached

This seems like the brunt of it:

Thread-205::INFO::2015-07-20 16:23:30,644::nfsSD::69::Storage.StorageDomain::(create) sdUUID=d6df7930-342a-493a-b70b-fb1c52b0828c domainName=ovirtprd01 remotePath=superstore001-stor.cs.example.com:/ovirtprd01 domClass=1
Thread-205::ERROR::2015-07-20 16:23:30,659::task::866::Storage.TaskManager.Task::(_setError) Task=`050e9378-ba78-4e6f-b986-0dda7bb09aa7`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
  File "/usr/share/vdsm/storage/hsm.py", line 2670, in createStorageDomain
  File "/usr/share/vdsm/storage/nfsSD.py", line 80, in create
  File "/usr/share/vdsm/storage/nfsSD.py", line 49, in _preCreateValidation
  File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemFeatures
  File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 507, in touch
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 391, in _sendCommand
OSError: [Errno 2] No such file or directory

Offhand, seems like sending a wrong path, not sure what we can do here.

What was the issue you were having?
What fix do you still need on this?
Can you attach the config that caused this issue?

Can not reproduce

Will reopen if we will face again.