Bug 1421607

+++ This bug was initially created as a clone of Bug #1383979 +++

Description of problem:
=======================
Getting the below error messages in the glusterd log when deprobe the cluster node is done.

[2016-10-12 07:31:07.381464] E [MSGID: 106029] [glusterd-utils.c:7767:glusterd_check_files_identical] 0-management: stat on file: /var/lib/glusterd//-server.vol failed (No such file or directory) [No such file or directory]
[2016-10-12 07:31:07.381736] E [MSGID: 106570] [glusterd-utils.c:7196:glusterd_friend_remove_cleanup_vols] 0-management: Failed to reconfigure all daemon services.



Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.8.4-2


How reproducible:
=================
Always


Steps to Reproduce:
===================
1. Create a two nodes (n1 and n2) cluster using 3.8.4-2 build
2. detach the n2 node from n1 //  n1#]gluster peer detach n2
3. Check for error messages in n1 glusterd.log

Actual results:
===============
Getting error messages in glusterd.log when peer detach is done


Expected results:
=================
Error messages should not come.


Additional info:

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-10-12 05:49:09 EDT ---

This bug is automatically being proposed for the current release of Red Hat Gluster Storage 3 under active development, by setting the release flag 'rhgs‑3.2.0' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Atin Mukherjee on 2016-10-12 05:55:38 EDT ---

Root cause is @ http://post-office.corp.redhat.com/archives/a-team/2016-October/msg00043.html , in short disabling gNFS service in rhgs-3.2.0 has caused these additional log entries.

Given there is no functional impact, I'd like to move it beyond 3.2.0. However would leave it up to NFS-Ganesha team to decide.

--- Additional comment from Niels de Vos on 2016-10-12 06:27:06 EDT ---

I assume there is a Glusto or other regression test case for this. Please point to its location or attach it to this BZ. Thanks!

--- Additional comment from Niels de Vos on 2016-10-12 06:31:16 EDT ---

Also, could you mention how urgent this is? Severity is set to "high", but priority is set to "undefined". If this problem causes the log to grow rapidly, we should fix it soon, otherwise we'll move it out to a later update.

--- Additional comment from Byreddy on 2016-10-12 06:49:38 EDT ---

(In reply to Niels de Vos from comment #3)
> I assume there is a Glusto or other regression test case for this. Please
> point to its location or attach it to this BZ. Thanks!





(In reply to Niels de Vos from comment #3)
> I assume there is a Glusto or other regression test case for this. Please
> point to its location or attach it to this BZ. Thanks!

test case bug ID which caught this issue is https://bugzilla.redhat.com/show_bug.cgi?id=1246946

--- Additional comment from Byreddy on 2016-10-12 06:50:38 EDT ---

Regression test case in the polarion.

https://polarion.engineering.redhat.com/polarion/#/project/RHG3-9817

--- Additional comment from Byreddy on 2016-10-12 06:53:04 EDT ---

sorry i cleared the need info of others ..i will set it back

--- Additional comment from Byreddy on 2016-10-12 07:05:30 EDT ---

(In reply to Niels de Vos from comment #4)
> Also, could you mention how urgent this is? Severity is set to "high", but
> priority is set to "undefined". If this problem causes the log to grow
> rapidly, we should fix it soon, otherwise we'll move it out to a later
> update.

This is not urgent and it's not blocker and no functionality loss But we have regression test case ( mentioned above) which will be marked as failed during regression cycle and Regression keyword will be added to this bug.

and these error messages are not continuous, for every peer detach operation, it will throw those two error messages.

--- Additional comment from Kaleb KEITHLEY on 2016-10-12 07:33:23 EDT ---

I'm not really sure how changing the default for starting gNFS or not would have anything to do with peer probing or related log messages.

Can you elaborate?

--- Additional comment from Atin Mukherjee on 2016-10-12 07:37:08 EDT ---

(In reply to Byreddy from comment #8)
> (In reply to Niels de Vos from comment #4)
> > Also, could you mention how urgent this is? Severity is set to "high", but
> > priority is set to "undefined". If this problem causes the log to grow
> > rapidly, we should fix it soon, otherwise we'll move it out to a later
> > update.
> 
> This is not urgent and it's not blocker and no functionality loss But we
> have regression test case ( mentioned above) which will be marked as failed
> during regression cycle and Regression keyword will be added to this bug.

I disagree! Why would you want to mark a test failed given the test has actually passed? On the basis of having couple of error entries in the log a test case can not be failed and regression keyword can not be used IMO.

Rahul - please chime in with your thoughts.
> 
> and these error messages are not continuous, for every peer detach
> operation, it will throw those two error messages.

--- Additional comment from Atin Mukherjee on 2016-10-12 07:41:02 EDT ---

(In reply to Kaleb KEITHLEY from comment #9)
> I'm not really sure how changing the default for starting gNFS or not would
> have anything to do with peer probing or related log messages.
> 
> Can you elaborate?

glusterd_friend_remove () ==> glusterd_friend_remove_cleanup_vols () ==> glusterd_svcs_reconfigure () ==> glusterd_nfssvc_reconfigure () where this function is unconditionally called (should be called only if gNFS is active)

And this is for peer detach code path.

--- Additional comment from Byreddy on 2016-10-12 07:50:10 EDT ---

(In reply to Atin Mukherjee from comment #10)
> (In reply to Byreddy from comment #8)
> > (In reply to Niels de Vos from comment #4)
> > > Also, could you mention how urgent this is? Severity is set to "high", but
> > > priority is set to "undefined". If this problem causes the log to grow
> > > rapidly, we should fix it soon, otherwise we'll move it out to a later
> > > update.
> > 
> > This is not urgent and it's not blocker and no functionality loss But we
> > have regression test case ( mentioned above) which will be marked as failed
> > during regression cycle and Regression keyword will be added to this bug.
> 
> I disagree! Why would you want to mark a test failed given the test has
> actually passed? On the basis of having couple of error entries in the log a
> test case can not be failed and regression keyword can not be used IMO.
> 

As per the test case, peer detach should not populate any error messages but currently it's throwing errors and  this issue was not there in last GA release so it's regression from my side.


> Rahul - please chime in with your thoughts.
> > 
> > and these error messages are not continuous, for every peer detach
> > operation, it will throw those two error messages.

--- Additional comment from Rahul Hinduja on 2016-10-12 07:51:10 EDT ---

> 
> Rahul - please chime in with your thoughts.


Agree the functionality of detach works but with errors in log. If the test case was to check the functionality, it should be marked as Passed. But this particular case RHG3-9817 is actually to check for errors and not just the functionality. Hence this particular case will be marked failed in test run. 

Regarding regression keyword, if as engineering we take a call to move this BZ for 3.2.z? Then I am ok to ignore the Regression keyword for this special case as it deals with log and no functionality impact. 

> > 
> > and these error messages are not continuous, for every peer detach
> > operation, it will throw those two error messages.

--- Additional comment from Niels de Vos on 2016-10-12 08:18:03 EDT ---

(In reply to Byreddy from comment #6)
> Regression test case in the polarion.
> 
> https://polarion.engineering.redhat.com/polarion/#/project/RHG3-9817

This asks for a login, and I do not seem to have one (or at least non of the Kerberos based ones work).

Can you maybe point to the script itself, in some git repository?

--- Additional comment from Byreddy on 2016-10-13 00:33 EDT ---



--- Additional comment from Byreddy on 2016-10-13 00:36:31 EDT ---

(In reply to Niels de Vos from comment #14)
> (In reply to Byreddy from comment #6)
> > Regression test case in the polarion.
> > 
> > https://polarion.engineering.redhat.com/polarion/#/project/RHG3-9817
> 
> This asks for a login, and I do not seem to have one (or at least non of the
> Kerberos based ones work).
> 
> Can you maybe point to the script itself, in some git repository?

Test case is not scripted, only test steps are there. it's attached here
https://bugzilla.redhat.com/attachment.cgi?id=1209905

--- Additional comment from Niels de Vos on 2016-10-13 04:37:20 EDT ---

(In reply to Byreddy from comment #16)
> Test case is not scripted, only test steps are there. it's attached here
> https://bugzilla.redhat.com/attachment.cgi?id=1209905

If this is not automated, I do not think there is an urgent need to get this addressed.

When developers write a fix, they need to detect the problem with an automated test-case. It would be most welcome if QE can provide such a script that makes it easier/faster to develop+test+verify any needed patches.

--- Additional comment from Byreddy on 2016-10-13 06:44:29 EDT ---

(In reply to Niels de Vos from comment #17)
> (In reply to Byreddy from comment #16)
> > Test case is not scripted, only test steps are there. it's attached here
> > https://bugzilla.redhat.com/attachment.cgi?id=1209905
> 
> If this is not automated, I do not think there is an urgent need to get this
> addressed.
> 
> When developers write a fix, they need to detect the problem with an
> automated test-case. It would be most welcome if QE can provide such a
> script that makes it easier/faster to develop+test+verify any needed patches.

This is not urgent to fix

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-01-01 12:25:28 EST ---

This BZ having been considered, and subsequently not approved to be fixed at the RHGS 3.2.0 release, is being proposed for the next release of RHGS

--- Additional comment from Atin Mukherjee on 2017-02-09 00:22:10 EST ---

Gaurav - Can you start looking into this issue? I'd like to get this fixed in next release.