Bug 1315306

Created attachment 1133729 [details]
vdsm.log

Liron, haven't we seen a similar issue before?

Allon, not that i remember.


Thanks for reporting Nicolas

In the provided logs we can see that the deactivation fails because of failed master migration - we fail when copying the master fs (see [1]).

I can see that this issue happens from time to time when having a Gluster master storage domain (see https://bugzilla.redhat.com/show_bug.cgi?id=1248453) as in your usecase.


Just to verify, can you please provide the following info?
1. have you tried to deactivate the domain again? was the issue consistent? 
2. Have you performed any operations on the the source/target domains storage that could cause to the copy failure? 


We need to dig into further into that issue (possibly with the Gluster team).


[1]-
-9bec08957c5f/master/vms
jsonrpc.Executor/0::DEBUG::2016-03-02 13:01:50,575::fileUtils::124::Storage.fileUtils::(cleanupdir) Removing directory: /rhev/data-center/mnt/blockSD/9339780c-3667-4fef-aa13
-9bec08957c5f/master/tasks
jsonrpc.Executor/0::ERROR::2016-03-02 13:01:50,713::sp::820::Storage.StoragePool::(masterMigrate) migration to new master failed
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 809, in masterMigrate
    exclude=('./lost+found',))
  File "/usr/share/vdsm/storage/fileUtils.py", line 68, in tarCopy
    raise TarCopyFailed(tsrc.returncode, tdst.returncode, out, err)
TarCopyFailed: (1, 0, '', '')
jsonrpc.Executor/0::DEBUG::2016-03-02 13:01:50,721::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction

Sahina, can you take a look in this bug (and possibly in BZ 1248453 which includes system logs as well) and let us know if this issue or its cause are known?

thanks,
Liron.

Liron,

1. Yes, several colleagues tried the same operation and all of them failed. In total, 5-7 attempts.
2. No that I know, we basically moved everything from glusterfs to iSCSI, once we were sure there's not anything left on glusterfs and after a few days we wanted to detach it, so we tried putting it on maintenance and it crashed all times. What I don't remember is whether after each of these attempts the SPM changed, but I remember that at least on two of these attempts it did.

(In reply to Liron Aravot from comment #4)
> Sahina, can you take a look in this bug (and possibly in BZ 1248453 which
> includes system logs as well) and let us know if this issue or its cause are
> known?
> 
> thanks,
> Liron.

As per gluster dev, for the bug 1248453 - the error was "TarCopyFailed: (1, 0, '', '/usr/bin/tar: ./vms: time stamp 2015-07-30 14:59:44 is 23312.863365733 s in the future" and is possibly related to Bug 1312721 in glusterfs (fixed in 3.7.9)

This bug has no gluster logs and the error in vdsm.log has no error message to indicate the cause.

Is it possible to set below option on your gluster volume and try to migrate again?
# gluster volume set <volname> cluster.consistent-metadata on

(This option has performance impact - but should be ok in your case as you're trying to detach the gluster domain)

At some point within the next 2 weeks we're removing the gluster storage domain from our second oVirt infrastructure too, so I'll make that change and provide feedback then.

Today we've removed the glusterfs-based backend. As the first approach we didn't touch configuration so we could reproduce the problem first and confirm it happens on both infrastructures, but unfortunately we weren't able.

We put the storage domain backend on maintenance and it went smoothly - we're using the same versions of engine & VDSM on both infrastructures, which makes it even stranger.

FWIW, although both infrastructures used glusterfs backends, they were different servers and volumes for both oVirt installations.

Nicolas, can you please specify what version of gluster are you using? And possibly attach the gluster logs?

(In reply to Sahina Bose from comment #6)
> As per gluster dev, for the bug 1248453 - the error was "TarCopyFailed: (1,
> 0, '', '/usr/bin/tar: ./vms: time stamp 2015-07-30 14:59:44 is
> 23312.863365733 s in the future" and is possibly related to Bug 1312721 in
> glusterfs (fixed in 3.7.9)
> 

The bug isn't caused/solved by the gluster BZ, but to general copy using tar when the clocks aren't synced (let's continue the discussion about it in the BZ) - so its not related to our bug here.

> This bug has no gluster logs and the error in vdsm.log has no error message
> to indicate the cause.
> 
> Is it possible to set below option on your gluster volume and try to migrate
> again?
> # gluster volume set <volname> cluster.consistent-metadata on
> 
> (This option has performance impact - but should be ok in your case as
> you're trying to detach the gluster domain)

In comment 9 I've asked Nicolas to provide the logs if possible.

thanks,
Liron.

(In reply to Sahina Bose from comment #6)
> (In reply to Liron Aravot from comment #4)
> > Sahina, can you take a look in this bug (and possibly in BZ 1248453 which
> > includes system logs as well) and let us know if this issue or its cause are
> > known?
> > 
> > thanks,
> > Liron.
> 
> As per gluster dev, for the bug 1248453 - the error was "TarCopyFailed: (1,
> 0, '', '/usr/bin/tar: ./vms: time stamp 2015-07-30 14:59:44 is
> 23312.863365733 s in the future" and is possibly related to Bug 1312721 in
> glusterfs (fixed in 3.7.9)
> 
> This bug has no gluster logs and the error in vdsm.log has no error message
> to indicate the cause.
> 

Sahina- the tar warnings in bug 1248453 on the timestamp aren't the cause for the failure (but just warnings on stderr), so if needed you can use the provided info there to diagnose the gluster copy failure (till the info will be provided here as well). Thanks.

glusterfs versions are:

3.7.2-3 for client (oVirt nodes)
3.6.3 for server

We needed to use this version for client because any further version wouldn't work with this server version (we tried mounting the volume manually and it failed).

Unfortunately, at this time we have no logs for the time this happened as they rotated.

Apologies - haven't been able to spend time on this bug. From attached logs in bug 1248453  I could only see tar copy errors on migration - no gluster specific errors.
Unless we have a reproducer with newer versions, we can close this bug?

Can you reproduce with latest gluster\ovirt?

We don't use gluster anymore since it caused a lot of problems, so we don't have a gluster infrastructure to try to reproduce it.

I think it is ok to close this BZ since no one seems to have complained about this issue and I cannot provide additional info at this time.