Bug 857632 - When removing gluster storage, it always fails on timeout (POSIXFS)
When removing gluster storage, it always fails on timeout (POSIXFS)
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.1.0
x86_64 Linux
urgent Severity urgent
: ---
: 3.1.0
Assigned To: Tal Nisan
RedHat Israel QE
storage
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-15 06:29 EDT by Petr Dufek
Modified: 2016-02-10 14:01 EST (History)
14 users (show)

See Also:
Fixed In Version: SI21
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-04 15:04:48 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdsm log (238.01 KB, application/x-gzip)
2012-09-15 06:29 EDT, Petr Dufek
no flags Details
logs (1.33 MB, application/x-gzip)
2012-09-25 07:51 EDT, Petr Dufek
no flags Details

  None (edit)
Description Petr Dufek 2012-09-15 06:29:50 EDT
Created attachment 613229 [details]
vdsm log

Description of problem:
When removing gluster storage, it always fails on timeout.


How reproducible:
- attach gluster storage (storage type: posix compliant FS, VFStype: glusterfs)
- creation of VM with adding disk from this storage is possible (can be seen in log)
- detach storage
- when removing storage, timeout always happens


Additional info:

Host:
-----
vdsm-4.9.6-34.0.el6_3.x86_64 (or tested also with: vdsm-4.9.6-31.0.el6_3.x86_64)
glusterfs-debuginfo-3.3.0qa45-1.el6.x86_64
glusterfs-3.3.0qa45-1.el6.x86_64
glusterfs-devel-3.3.0qa45-1.el6.x86_64
glusterfs-fuse-3.3.0qa45-1.el6.x86_64

- vdsm log is attached.
Comment 3 Petr Dufek 2012-09-17 06:46:52 EDT
command which fails: connectStorageServer (Could not connect to storageServer, MountError: (32, ';mount.nfs: Connection timed out\n'))

workflow of commands is as follows:
- validateStorageServerConnection
- connectStorageServer
- createStorageDomain
- getStorageDomainStats
- validateStorageServerConnection
- connectStorageServer
- createStoragePool
- validateStorageServerConnection
- connectStorageServer
- connectStoragePool
- activateStorageDomain
- connectStoragePool
- createVolume
- updateVM
- removeVM
- deleteImage
- deactivateStorageDomain
- disconnectStoragePool
- disconnectStorageServer
- validateStorageServerConnection
- connectStorageServer
- connectStoragePool
- destroyStoragePool
- disconnectStorageServer
- disconnectStoragePool
- validateStorageServerConnection
- connectStorageServer

the last 'connectStorageServer' always returns:
Thread-281661::ERROR::2012-09-17 12:09:26,194::hsm::1971::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 1968, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 179, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Comment 4 Ayal Baron 2012-09-23 03:20:35 EDT
(In reply to comment #0)
> Created attachment 613229 [details]
> vdsm log
> 
> Description of problem:
> When removing gluster storage, it always fails on timeout.

What do you mean removing?
What do you actually do? remove how?

Please attach engine log as well (always)
Comment 5 Petr Dufek 2012-09-25 07:50:48 EDT
Testing scenario:
- adding gluster storage in webadmin (storage type: posix compliant FS, VFStype: glusterfs) & attaching to data center
- creating of VM with adding disk from this storage in webadmin (can be seen in log)
- moving storage to maintenance in webadmin
- removing data center in webadmin
- removing storage in webadmin - timeout always happens

vdsm & engine logs are attached
Comment 6 Petr Dufek 2012-09-25 07:51:22 EDT
Created attachment 616994 [details]
logs
Comment 7 Attila Darazs 2012-09-27 08:15:01 EDT
This issue blocks us from adding "Posix FS over Gluster" run mode to our basic Storage Sanity test.
Comment 9 Ayal Baron 2012-09-29 10:22:03 EDT
After detach engine did not pass the vfs_type to vdsm so connect failed:

Thread-1304::INFO::2012-09-25 13:30:05,137::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=6, spUUID='00000000-
0000-0000-0000-000000000000', conList=[{'connection': 'filer01.qa.lab.tlv.redhat.com:/pdufekdd', 'iqn': '', 'portal': '', 'user': '', 'password': '******', '
id': '00928e25-79ca-4ca2-bc39-7e398c01b47f', 'port': ''}], options=None)

Thread-1305::ERROR::2012-09-25 13:32:17,212::hsm::1971::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 1968, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 179, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Comment 11 Tal Nisan 2012-10-15 13:36:26 EDT
http://gerrit.ovirt.org/#/c/8576/
Comment 13 Petr Dufek 2012-10-19 05:50:30 EDT
verified in si21

Note You need to log in before you can comment on or make changes to this bug.