Red Hat Bugzilla – Bug 226736
The DLM's droplocks callback clobbers GFS2 performance
Last modified: 2007-11-30 17:11:56 EST
I've been a little worried about the DLM's droplocks callback and its effects on
the GFS2's caching of locks & inodes for a while. While doing testing recently
(on what is a rather unusual set up I'll agree) I realised that it was causing a
big performance penalty.
My set up was as follows: one node running gfs2 and dlm. One filesystem mounted
running postmark (set number 100000, set transactions 100000). On my test
machine which has 4G of RAM, the dlm exceeded the lock limit and started to call
droplocks. After a one liner bug fix to this code (it was trying to drop locks
which were still attached to transactions) I noticed that the performance was
really very poor compared with when there was no limit on locks (tested by
commenting out the droplocks callback in the gfs2 code).
Now I could raise the limit so that these callbacks don't happen, but I don't
think that really gets to the root of the problem.
If I comment out the callback, then for my single node set up, all works fine.
The reason is that RAM is a finite resource and the amount of RAM determines the
maximum amount of locks that its reasonable to cache. As the RAM fills up, the
VM system pushes inodes out of cache and as a result the glocks get demoted, so
there is already a feedback mechanism in place that works perfectly well in the
single node case.
Obviously in the cluster case, the same mechanism isn't going to work as the
lock manager will often be mastering locks for other nodes. If we assume for a
moment that the locks are roughly evenly distributed across the cluster then
problems are only likely to occur when there are machines with differing amounts
of RAM available (which may mean differing amounts of physical RAM or just that
one node is virtually idle whilst another has its RAM more or less full with
page cache pages) or where the distribution of locks across the cluster has
become uneven (I think thats then a failure of the hashing).
It seems to me that one solution to this problem would therefore be to migrate
locks away from nodes which have "too many" locks on them by moving the locks
which are held by other nodes to the nodes in question. The problem in this case
is in defining "too many" and probably that needs either an input from the VM or
some way for a node to determine whether its holding an excessive number of
locks compared to the other nodes with which its in contact (probably a better
I discussed this idea briefly with Patrick and he suggested that it would be
possible to do this, but that it would take a fair amount of work to do it.
The above might not be the right answer, but I'd like to kick off some
discussion on this point as I'd like to have a solution which keeps a reasonable
balance of lock numbers without needing any user adjustments if at all possible.
There are probably a couple of more basic things we could do to
minimize the problem:
First is the way the dlm implements the resource directory which is
very inefficient. Back when I was seeing the memory outages during
recovery, it would often occur during the initial rebuilding of the
resource directory. The directory is a hash table of structs
(name/nodeid pairs) that's completely independent of the rsb hash
table which already contains much of the same info. On my todo list
forever has been to integrate the directory into the rsb hash table
(which is the way the VMS docs imply it's done.)
Second is probably minor, but I've also wanted to have an amount
of space in the rsb itself for the name such that most names will
fit inside and not require a separate malloc for just the name.
i.e. copy how the struct dentry is done.
Third, during remastering, we could have a node be more agressive about
becoming the new master of resources if it had more local memory
available and less agressive (or refuse) to become the new master of
a resource if it was low on memory.
The first two things would improve the dlm's memory usage in general
and also reduce the likelihood of problems during recovery. The third
would be a way to avoiding a real OOM condition when the situation is
ripe for it.
Migrating or remastering locks during normal operation is something
we may also want to do sometime; that also sounds a bit like what
Yes, thats pretty much what I'm suggesting. Really its an algorithm to decide
when locks should be remastered (or in fact, two possible algorithms as I can't
decide which of these is better at the moment) which look like this in pseudo code:
Remaster locks when there is memory pressure (can we get the VM to tell us?) and
select the locks to be remastered on the basis of:
1. Locks which are held by a node other than the current node
2. Locks which have been locked for a long time and don't change state very much
Locks would be "pushed" from the loaded DLM towards the node where the lock was
currently being held.
Assume that each time one DLM sends a message to another DLM it includes with
the message a total number of locks which its currently mastering. This allows
any particular DLM instance to know the approx load on its neighbours. Its then
possible to create an algorithm along the lines of:
Migrate locks in case number of local locks > X% above the average number of
locks held on a node, the selection would be the same as Algo 1. The X in this
case could be selected so as to produce a fairly even distribution over time,
but without causing too much migration.
The big question is whether its worth pushing locally held locks to other nodes
in case the other nodes are lightly loaded and the local node is under pressure.
I'm not sure this makes any sense unless we are certain that lock mastering will
take a lot more memory than is used to keep track of the local state of the
lock. I don't know whether thats true or not, and I dare say you have a better
idea than I do. In other words we'd need to compare the memory useage on the
lock holding node in the case where it mastered the lock and the case where it
was mastered remotely and see how much difference there was.
In addition, since the latency of the locking operations will increase with the
lock being remote, it would seem that a certain amount of the potential
advantage of migration away from a local node would be lost in case that
activity on the lock was greater than a certain threshold.
So I guess in summary I'm not that keen on trying to remaster locks which have a
local holder at all. As to which of algo 1 or 2 above is better, is left as an
exercise for the reader :-)
Something else I noticed recently is that the two lock_dlm threads spend a lot
of time waiting for I/O. It might be an idea to try and push the drop locks
stuff into the existing code which scans locks in order to not hold up other
lock requests behind the drop locks request.
There is also another reason that the lock_dlm threads wait for I/O but I'll
save that for bz 221152.
You shouldn't be seeing drop-locks callbacks regularly, it should only
be a rare condition when gfs gets that callback. Drop-locks was never
intended to be a part of the normal operation of the fs, it was intended
to be an "emergency" button that the dlm could press when in danger of
running out of resources and crashing. If you're getting drop-locks
callbacks during your tests, then you should probably disable them
altogether by echoing 0 into the /sys/fs/gfs/... file. I've also
mentioned before that we might disable drop-locks by default.
I'd be happy to take a patch to disable the drop locks stuff by default. With
postmark it seems to make an approx 10x reduction in performance once it
triggers. With drop locks off, postmark runs at a pretty similar speed as it
does with lock_nolock (this is a single node dlm set up).
or just echo 0 >> /sys/fs/gfs2/locking/drop_count
An alternative to setting it to 0 by default is to set it high
enough that most people/workloads don't hit it. That's what I'd
hoped to do recently when I increased the default, but maybe I
didn't increase it enough. It doesn't matter to me which we do.
Fixed in upstream by making 0 the default value of drop_count.