Bug 848706

also can reference Bug 836095 (I originally sent, before one month). It is the same.

Hi Chen,

Have you described these issues upstream?  That is the best place to work to resolve issues unless you have a support contract with Red Hat or formal partner relationship.

Best regards,

Ric

IIUC, the document spells out a deadlock that is possible if you mount via nfs a filesystem being exported from knfsd on the same host. That's an easily deadlockable situation for many reasons and is quite explicitly unsupported for that reason.

It's quite easy for the server to get into a situation where it needs memory in order to process a request from the client, so it tries to flush data to the server which can't do anything until that flush completes.

This is not just a deadlockable configuration for RHEL5, but for *any* version of the kernel.

I'm going to go ahead and close this WONTFIX. Please reopen if you wish to discuss it further.

(In reply to comment #2)
> Hi Chen,
> 
> Have you described these issues upstream?  That is the best place to work to
> resolve issues unless you have a support contract with Red Hat or formal
> partner relationship.
> 
> Best regards,
> 
> Ric

It is running under Asianux test envrionments.

Asianux kernel is based on Red Hat kernel.

For RHEL5 (Asianux Server 3), we need merge our patches into kernel.

The test environments is described in the report document (please reference it).

(for stress test, we use ltpstress.sh, and it will do it under local machine).


hope these information is helpful for you.

(In reply to comment #3)
> IIUC, the document spells out a deadlock that is possible if you mount via
> nfs a filesystem being exported from knfsd on the same host. That's an
> easily deadlockable situation for many reasons and is quite explicitly
> unsupported for that reason.
> 
> It's quite easy for the server to get into a situation where it needs memory
> in order to process a request from the client, so it tries to flush data to
> the server which can't do anything until that flush completes.
> 
> This is not just a deadlockable configuration for RHEL5, but for *any*
> version of the kernel.
> 
> I'm going to go ahead and close this WONTFIX. Please reopen if you wish to
> discuss it further.



Firtly,

sorry for my english not quite well (I do not know the meaning of "IIUC").



Secondly, 

1) I do not think that it can happen for *any* version of the kernel.

2) I have tested some versions of RHEL6 kernel (at least 5 versions) using the same test environments (almost same, at least), and not found this issue ("maybe" need spend longer time for stress testing ??).

3) As far as I know (from read the source code of nfs version 3 and nfsd version 3), I think the origianl nfs module "seems" (only seems) that it has considered the situation that knfsd mounted on local machine.

4) For RHEL5, all issues (although they are diferent) occured around of the commit features of nfs sub-system, not around of any other features of nfs sub-system.


At last:

Sorry, that I do not think it is suitable to mark this bug as CLOSED.

(In reply to comment #5)
> 
> Firtly,
> 
> sorry for my english not quite well (I do not know the meaning of "IIUC").
> 

IIUC means "If I understand correctly".

> 
> 
> Secondly, 
> 
> 1) I do not think that it can happen for *any* version of the kernel.
> 

I've looked over all of the stack traces in your MS-Word document, and they all point to same root cause. You have configuration where the same host is acting as both NFS client and server. That's a configuration known to cause deadlocks.


> 2) I have tested some versions of RHEL6 kernel (at least 5 versions) using
> the same test environments (almost same, at least), and not found this issue
> ("maybe" need spend longer time for stress testing ??).
> 

Then you've been lucky (or unlucky). These sorts of deadlocks are just as possible there. You can minimize the chances of them by tuning the virtual memory settings in such a way as to keep knfsd out of direct reclaim, but it's still possible to deadlock.

> 3) As far as I know (from read the source code of nfs version 3 and nfsd
> version 3), I think the origianl nfs module "seems" (only seems) that it has
> considered the situation that knfsd mounted on local machine.
> 

I may have considered it, but it certainly isn't engineered in such a way as to do so under heavy stress. We occasionally use mounts over loopback for certain sorts of testing, but anything that stresses memory utilization is generally problematic for this reason.

> 4) For RHEL5, all issues (although they are diferent) occured around of the
> commit features of nfs sub-system, not around of any other features of nfs
> sub-system.
> 

Actually all 3 issues look pretty similar to me. The root cause is that you have knfsd in a situation where it needs to reclaim memory in order to proceed. It then tries to flush dirty NFS pages to itself, which can't work until it can allocate memory.

> 
> At last:
> 
> Sorry, that I do not think it is suitable to mark this bug as CLOSED.

I'm afraid I have to disagree here. This is not something we intend to address unless you can point to a deadlock that can occur when the client and server are separate hosts.

I also want to make clear that Red Hat as a company does not support clones of our product. If you want to get Red Hat to work with your company, you need to have a business relationship with us.

Of course, we are more than happy to work on problems in the upstream community. As Jeff stated, the configuration you describe is not supported (and not useful outside of testing as far as I can see).

Please take any further discussion to the upstream list and I would suggest only after you reproduce the deadlock with distinct client and servers as Jeff mentions above.

Thanks!

(In reply to comment #7)
> I also want to make clear that Red Hat as a company does not support clones
> of our product. If you want to get Red Hat to work with your company, you
> need to have a business relationship with us.
> 

Sure, I think, you truly need not support another company which no relationship with Red Hat (such as Asianux).

I also agree, for Red Hat, support this issue maybe waste time resource.


> Of course, we are more than happy to work on problems in the upstream
> community. As Jeff stated, the configuration you describe is not supported
> (and not useful outside of testing as far as I can see).
>

> Please take any further discussion to the upstream list and I would suggest
> only after you reproduce the deadlock with distinct client and servers as
> Jeff mentions above.
> 
> Thanks!


Firstly:

   Thank you for still can take further discussion if I have new informations.



Secondly:

   If truly under distince client and server testing environments.

   For IssueX1 (Which is talked in my report document);
       I think it will still be a bug.
       It is still dead lock for Client side itself.
       I get a fix patch for it.
   Because I am not under distinct client and servers during original testing, so the conclusion is not suitable in report document.

   I hope Jeff Layton can help checking the patch which is in chapter "IssueX1" in report document, when he has time.



Thirdly:

   I will still read the source code of the details of nfs module on kernel-2.6.18-308.8.2.el5 to prove what Jeff Layton mentions 

   Also I need search relative information by google.

   It will truly be better that Jeff Layton can give some proof materials directly.



At last:

   I think one thing is sure at least (during our communication):

   For client & server on same machine, the nfs commit features improves the chances of nfs deadlock.

   Is it correct ?


thanks.

(In reply to comment #8)
> (In reply to comment #7)
> > I also want to make clear that Red Hat as a company does not support clones
> > of our product. If you want to get Red Hat to work with your company, you
> > need to have a business relationship with us.
> > 
> 
> Sure, I think, you truly need not support another company which no
> relationship with Red Hat (such as Asianux).
> 
> I also agree, for Red Hat, support this issue maybe waste time resource.
> 
> 
> > Of course, we are more than happy to work on problems in the upstream
> > community. As Jeff stated, the configuration you describe is not supported
> > (and not useful outside of testing as far as I can see).
> >
> 
> > Please take any further discussion to the upstream list and I would suggest
> > only after you reproduce the deadlock with distinct client and servers as
> > Jeff mentions above.
> > 
> > Thanks!
> 
> 
> Firstly:
> 
>    Thank you for still can take further discussion if I have new
> informations.
> 
> 
> 
> Secondly:
> 
>    If truly under distince client and server testing environments.
> 
>    For IssueX1 (Which is talked in my report document);
>        I think it will still be a bug.
>        It is still dead lock for Client side itself.
>        I get a fix patch for it.
>    Because I am not under distinct client and servers during original
> testing, so the conclusion is not suitable in report document.
> 
>    I hope Jeff Layton can help checking the patch which is in chapter
> "IssueX1" in report document, when he has time.
> 
> 
> 
> Thirdly:
> 
>    I will still read the source code of the details of nfs module on
> kernel-2.6.18-308.8.2.el5 to prove what Jeff Layton mentions 
> 
>    Also I need search relative information by google.
> 
>    It will truly be better that Jeff Layton can give some proof materials
> directly.
> 
> 
> 
> At last:
> 
>    I think one thing is sure at least (during our communication):
> 
>    For client & server on same machine, the nfs commit features improves the
> chances of nfs deadlock.
> 
>    Is it correct ?
> 
> 
> thanks.

Chen,

If you can reproduce this problem with stock RHEL, and document how to reproduce this with RHEL, then we can look at it.  If you can't reproduce it with RHEL, then please work it upstream.

Thanks and Regards,

Ron

(In reply to comment #9)

> Chen,

> If you can reproduce this problem with stock RHEL, and
> document how to reproduce this with RHEL, then we can look at it.  If you
> can't reproduce it with RHEL, then please work it upstream.

> Thanks and
> Regards,

> Ron


Firstly:

   Thank you reply, and face the issue directly.


Secondly:

   Sorry for my English not quite well: is "stock RHEL" means it is the Red Hat release (Red Hat built by itself), and I need download the binary from the Website ?

   I truly download kernel-2.6.18-308.8.2.el5-x86_64.rpm from web site

   can reference the pictures in attachemnt of Bug 836095;

     it has "uname -a" command, display the full version string with build time;

     it also has "ps aux | grep D" command, which display the wait_uninterruptable processes.

     it also has current time display, so you can see the time spend.


Thirdly:

   In my report document, I mentioned the test environments (but maybe too simple, and need telling in details).

   But just like what Jeff Layton said, it is not difficult to repeat the issue.

   I will still work on it to prove what Jeff Layton said (maybe what he said is correct, but I need proof).

   I truly want Jef Layton to help checking my fix patch for IssueX1 (is it valid ?).


At last I hope:
  What I have done does not waste your time resources, and our five members(which have written comments under this bug number) truly can get some valuable results.


thanks.

as you mentioned, I will also try under "upstream community" (sorry for I am not familiar with it, so I will spend a short time to familiar with it).

It will be better if you tell something to introduce it for me in short words, if you have time.

Conclusion:
    Have get confirmation both from upstream and from LTP organization. the conclusion is truly what Jeff Layton said (what he said is correct).


1) For upstream community:

   I mail to linux-nfs.org and linux-kernel.org;
   only Jeff Layton reply, and no other guys reply during a week.
   "No reply during a week" means "auto confirm what Jeff Layton said".


2) For LTP organization:

   I mail to ltp-list.net;
   I got reply, but the reply is useless for the confirmation during a week.
   "No useful reply during a week" means "auto confirm what Jeff Layton said".


3) Next:

   I will change LTP features to match the test requirement of RHEL5 and RHEL6.
   If I find another "bug", I will come back to Red Hat Bugzilla to confirm it.
   If it is truly a bug, I will try to fix it, too.

thanks.

*** Bug 836095 has been marked as a duplicate of this bug. ***