Hi Stefan,

Can you please open a support ticket via your Red Hat official channels? That helps us a lot as we gather data and try to look at issues.

Thanks for the report!

Ric

Looks like cifs_writepages is stuck trying to lock a page. That likely means that something else is holding the page lock on that page. The question is what's holding it...

What would probably be most helpful is a full dump of the task status (via sysrq-t).

I have opened case 00584861 at support.redhat.com and attached the
content of /var/log/messages after a sysrq-t when login hangs.

Thanks. I don't see any process that's obviously holding the page lock and not releasing it. The most likely culprit here is the changes that added async writepages capability to cifs, but I've looked over that code and don't see any way that we could end up with pages being locked after cifs_writepages does its send. Perhaps this is just exposing another existing bug? Would it be possible to get a vmcore from this host? With that I might be able to tell more about the page(s) that everything is waiting on.

> Would it be possible
> to get a vmcore from this host? With that I might be able to tell more about
> the page(s) that everything is waiting on.

I have posted download links for two vmcore.bz2 files to case 00584861.
The files are too large to attach.

vmcore file: vmcore.bz2

Queued
Requestor : pbandark
Corefile(s) : vmcore.bz2
How do I check status?:
Simply get on irc in a channel with tambot (normally #gss on rhirc.redhat.com) and type the following:
tambot: cas_status 20120111100830

The problem described in this case is a regression caused by changes introduced by the patch


* Tue Jul 12 2011 Kyle McMartin <kmcmarti> [2.6.32-168.el6]
- [fs] cifs: convert async write callback to slow_work (Jeff Layton) [708000]

We are considering various options to avoid the deadlock caused by this patch. If you do encounter this issue, please use an older version of the kernel which doesn't contain the patch above until we have a fix for this issue.

The problem is that you have (a) a single pool of threads with a fixed finite limit on it and (b) tasks of two types with a dependency.  You are always at risk of deadlocking the pool by having the running threads all taken up with the dependent type of tasks and no running dependee tasks.

This is true even if the number of threads currently in the pool can be increased - provided there's a hard ceiling.

This cannot be solved without making a second pool whereby each type of task is segregated into its own pool.

That said, if the dependent task is marked as SLOW_WORK_VERY_SLOW it should provide this effect.  The slow-work facility guarantees to keep at least one thread nominally earmarked for ordinary slow work free from very-slow-work tasks, even through it will usually let ordinary threads process very-slow-work tasks if there's nothing better to do.

Summary: 

We see a deadlock caused by the following threads.

1) cifs_writepages waiting for the page writeback bit on a page to be cleared. It does this while holding the page lock.
2) There are 2 threads in the slow-work queue which are waiting for the pagelock held in 1. These occupy all running threads in the slow-work mechanism which blocks other slow-work tasks from running.
3) A task meant to clear the page writeback bit required by 1 is enqueued as a slow work task  behind the tasks in 2.

This leads to a deadlock.

The problem was introduced by the patch

commit 8dded88b7831e98dfc65ff15b6f53c1365117545
Author: Jeff Layton <jlayton>
Date:   Wed Jul 6 12:45:06 2011 -0400

    [fs] cifs: convert async write callback to slow_work
    
   
    RHEL6 doesn't have concurrency managed workqueues. Convert the async
    write callback code to use slow_work instead.
    
    Signed-off-by: Jeff Layton <jlayton>
    Signed-off-by: Kyle McMartin <kmcmarti>

which went into version 2.6.32-168.el6 ie. RHEL 6.2. This patch was part of a series of patches which introduced the async page write back performance for CIFS. 

The upstream version of the patch relies on the Concurrency managed workqueue mechanism available upstream. Since this mechanism is not available in RHEL 6.2, we decided to instead use the slow-work mechanism which can concurrently execute the tasks enqueued. However, in certain conditions such as the one reported here, this doesn't work. 

When slow-work detects that a new thread is required, it enqueues the task to create a new thread onto the same slow-work queue. However this can only be executed once atleast one of the threads executing exits. Since the 2 tasks currently running in the slow-work queue block all other tasks on the slow-work queue, we encounter this deadlock.

Created attachment 557020 [details]
patch -- convert oplock break job to very_slow_work

(In reply to comment #15)

> That said, if the dependent task is marked as SLOW_WORK_VERY_SLOW it should
> provide this effect.  The slow-work facility guarantees to keep at least one
> thread nominally earmarked for ordinary slow work free from very-slow-work
> tasks, even through it will usually let ordinary threads process very-slow-work
> tasks if there's nothing better to do.

Ok, so converting the oplock break job to very_slow_work should do the right thing. Something like this patch, perhaps?

In the light of the latest local root exploit fixed in kernel-2.6.32-131.4.1
we cannot stick with an old kernel any more (we use CIFS home directories
in public student labs).

I have rebuilt the latest kernel with the patch from Jeff's last comment. We
tested it and it seems to work reliably. We have now deployed this kernel in
our student labs.

This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.

Hi, could you give me some tips about how to reproduce this bug? thanks

Jian,

This was only reproduced on the user end. They have tested the new patch and haven't seen this issue reproduced.

Sachin Prabhu

Stefan,

Would you be able to test the official fix for this kernel once it is available?

Sachin Prabhu

(In reply to comment #24)
> Jian,
> 
> This was only reproduced on the user end. They have tested the new patch and
> haven't seen this issue reproduced.
> 
> Sachin Prabhu

thanks sachin, qa_ack+

Sure, I can easily test any kernel you give me. 

BTW, the 2.6.32-220.4.1 kernel we built ourselves with the one change from
comment 17 has worked fine ever since we deployed it on all student lab
machines.

Patch(es) available on kernel-2.6.32-232.el6

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
In the Common Internet File System (CIFS), the oplock break jobs and async callback handlers both use the SLOW-WORK workqueue, which has a finite pool of threads. Previously, these oplock break jobs could end up taking all the running queues waiting for a page lock which blocks the callback required to free this page lock from being completed. This update separates the oplock break jobs into a separate workqueue VERY-SLOW-WORK, allowing the callbacks to be completed successfully and preventing the deadlock.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0862.html