Bug 1409583
Summary: | seeing RPC status error messages and timeouts due to RPC (rpc-clnt.c:200:call_bail) | ||
---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Nag Pavan Chilakam <nchilaka> |
Component: | rpc | Assignee: | Raghavendra G <rgowdapp> |
Status: | CLOSED WORKSFORME | QA Contact: | Nag Pavan Chilakam <nchilaka> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | rhgs-3.2 | CC: | amukherj, atumball, mchangir, nbalacha, nchilaka, rgowdapp, rhs-bugs |
Target Milestone: | --- | Keywords: | ZStream |
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2019-07-19 07:04:10 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Nag Pavan Chilakam
2017-01-02 14:28:25 UTC
client sosreports are available at scp -r /var/tmp/$HOSTNAME qe@rhsqe-repo:/var/www/html/sosreports/nchilaka/3.2_logs/systemic_testing_logs/regression_cycle/same_dir_create_clients/ [2017-01-02 13:27:04.497549] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-sysvol-client-2: changing port to 49154 (from 0) [2017-01-02 13:27:04.499842] E [socket.c:2309:socket_connect_finish] 0-sysvol-client-5: connection to 10.70.37.86:49155 failed (Connection refused) msgs like these indicate that brick is not running on the port given by portmapper. Either, * Brick has crashed * Portmapper has given a wrong entry. (In reply to Raghavendra G from comment #5) > [2017-01-02 13:27:04.497549] I [rpc-clnt.c:1965:rpc_clnt_reconfig] > 0-sysvol-client-2: changing port to 49154 (from 0) > [2017-01-02 13:27:04.499842] E [socket.c:2309:socket_connect_finish] > 0-sysvol-client-5: connection to 10.70.37.86:49155 failed (Connection > refused) > > msgs like these indicate that brick is not running on the port given by > portmapper. Either, > > * Brick has crashed > * Portmapper has given a wrong entry. Had discussed an issue where portmapper gave stale ports with Atin earlier. @Atin/@samikshan, Can this be a stale port issue? regards, Raghavendra (In reply to Raghavendra G from comment #6) > (In reply to Raghavendra G from comment #5) > > [2017-01-02 13:27:04.497549] I [rpc-clnt.c:1965:rpc_clnt_reconfig] > > 0-sysvol-client-2: changing port to 49154 (from 0) > > [2017-01-02 13:27:04.499842] E [socket.c:2309:socket_connect_finish] > > 0-sysvol-client-5: connection to 10.70.37.86:49155 failed (Connection > > refused) > > > > msgs like these indicate that brick is not running on the port given by > > portmapper. Either, > > > > * Brick has crashed > > * Portmapper has given a wrong entry. > > Had discussed an issue where portmapper gave stale ports with Atin earlier. > > @Atin/@samikshan, > > Can this be a stale port issue? > > regards, > Raghavendra We have a case where if a brick process is killed through SIGKILL and then brought back where the port gets assigned to is lesser than the old port then in portmap traversal glusterd will pick up the old port. To confirm this case, we need to have gluster volume status output. Do we have that data in place? Milind, Can you check sosreports and get the volume status? I tried extracting sos-reports. But for some reason couldn't find the volume-status. looked at sos report at: http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/3.2_logs/systemic_testing_logs/regression_cycle/same_dir_create_clients/rhs-client23.lab.eng.blr.redhat.com/ but didn't find output of gluster volume status. sos_logs/sos.log reports that it can't find the gluster cli to run the gluster volume status command (In reply to Milind Changire from comment #9) > looked at sos report at: > > http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/3.2_logs/ > systemic_testing_logs/regression_cycle/same_dir_create_clients/rhs-client23. > lab.eng.blr.redhat.com/ > > but didn't find output of gluster volume status. > > sos_logs/sos.log reports that it can't find the gluster cli to run the > gluster volume status command Please check the corresponding brick log file and see what port was passed to it at the last init (In reply to Milind Changire from comment #9) > looked at sos report at: > > http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/3.2_logs/ > systemic_testing_logs/regression_cycle/same_dir_create_clients/rhs-client23. > lab.eng.blr.redhat.com/ > > but didn't find output of gluster volume status. > > sos_logs/sos.log reports that it can't find the gluster cli to run the > gluster volume status command My mistake; these are client side sos reports. I see no updates on this bug for more than a year. Did we have any conclusion on this issue? Was this seen in latest releases? What'd it take to close this bug? requesting re-validaiton of BZ to Nag see comment #14 Nag, feel free to open it if seen. Considering the issue is not seen from last 18+ months, would like to close for now. |