Fedora Account System
Red Hat Associate
Red Hat Customer
Created attachment 930825 [details] Scripts required to execute the case Description of problem: ========================== In glusterd log files, we are constantly observing the following error and warning message . [2014-08-26 10:35:55.446055] W [rpcsvc.c:254:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2014-08-26 10:35:55.446072] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2014-08-26 10:35:55.446096] W [socket.c:529:__socket_rwv] 0-socket.management: writev on 10.70.34.119:1014 failed (Broken pipe) Version-Release number of selected component (if applicable): ============================================================== glusterfs 3.6.0.27 built on Aug 4 2014 11:49:25 How reproducible: ================= 2/2 Steps to Reproduce: =================== 1. Create 4 2 x 2 dis-rep volume. Start the volumes. 2. Create 4 mounts for each of the volume from 2 different clients. 3. copy the attached scripts "create_dirs_files_multi_thread.py , create_dirs_files.pl " to all the mount points from one of the client. 4. From all the 4 mounts on client1 execute the following: ./create_dirs_files_multi_thread.py --number-of-threads 100 --num-files-per-dir 25 --min-file-size 1024 --max-file-size 10240 --starting-dir-num 1 --dir-depth 5 --dir-width 4 5. From all the 4 mounts on client2 execute the following: ./create_dirs_files_multi_thread.py --number-of-threads 100 --num-files-per-dir 25 --min-file-size 1024 --max-file-size 5120 --starting-dir-num 101 --dir-depth 5 --dir-width 4 Actual results: ===================== After sometime, observed the error and warning messages in the glusterd logs. Expected results: ===================== TBD Additional info: ===================== root@mia [Aug-26-2014-14:00:34] >gluster snapshot list No snapshots present root@mia [Aug-26-2014-14:41:49] > root@mia [Aug-26-2014-16:43:02] > root@mia [Aug-26-2014-16:43:02] >gluster v info Volume Name: vol1 Type: Distributed-Replicate Volume ID: 86fb01ba-be09-4734-87ab-bd77b926c1e5 Status: Started Snap Volume: no Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: rhs-client11:/rhs/device0/b1 Brick2: rhs-client12:/rhs/device0/b2 Brick3: rhs-client13:/rhs/device0/b3 Brick4: rhs-client14:/rhs/device0/b4 Options Reconfigured: features.barrier: disable performance.readdir-ahead: on snap-max-hard-limit: 256 snap-max-soft-limit: 90 auto-delete: disable Volume Name: vol2 Type: Distributed-Replicate Volume ID: f9740540-88d7-4ee6-b720-024bd827bcac Status: Started Snap Volume: no Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: rhs-client11:/rhs/device1/b1 Brick2: rhs-client12:/rhs/device1/b2 Brick3: rhs-client13:/rhs/device1/b3 Brick4: rhs-client14:/rhs/device1/b4 Options Reconfigured: features.barrier: disable performance.readdir-ahead: on snap-max-hard-limit: 256 snap-max-soft-limit: 90 auto-delete: disable Volume Name: vol3 Type: Distributed-Replicate Volume ID: 2dc6c55c-e945-4f44-bd4a-e0da47041b78 Status: Started Snap Volume: no Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: rhs-client11:/rhs/device2/b1 Brick2: rhs-client12:/rhs/device2/b2 Brick3: rhs-client13:/rhs/device2/b3 Brick4: rhs-client14:/rhs/device2/b4 Options Reconfigured: features.barrier: disable performance.readdir-ahead: on snap-max-hard-limit: 256 snap-max-soft-limit: 90 auto-delete: disable Volume Name: vol4 Type: Distributed-Replicate Volume ID: 224dc91f-530c-4dea-a289-c7b5f1239133 Status: Started Snap Volume: no Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: rhs-client11:/rhs/device3/b1 Brick2: rhs-client12:/rhs/device3/b2 Brick3: rhs-client13:/rhs/device3/b3 Brick4: rhs-client14:/rhs/device3/b4 Options Reconfigured: features.barrier: disable performance.readdir-ahead: on snap-max-hard-limit: 256 snap-max-soft-limit: 90 auto-delete: disable root@mia [Aug-26-2014-16:43:05] >gluster v status Status of volume: vol1 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick rhs-client11:/rhs/device0/b1 49152 Y 7991 Brick rhs-client12:/rhs/device0/b2 49152 Y 18408 Brick rhs-client13:/rhs/device0/b3 49152 Y 25248 Brick rhs-client14:/rhs/device0/b4 49152 Y 2898 NFS Server on localhost 2049 Y 1570 Self-heal Daemon on localhost N/A Y 1578 NFS Server on rhs-client14 2049 Y 2906 Self-heal Daemon on rhs-client14 N/A Y 2916 NFS Server on rhs-client13 2049 Y 21453 Self-heal Daemon on rhs-client13 N/A Y 21463 NFS Server on rhs-client12 2049 Y 18664 Self-heal Daemon on rhs-client12 N/A Y 18673 NFS Server on rhs-client11 2049 Y 7998 Self-heal Daemon on rhs-client11 N/A Y 8005 Task Status of Volume vol1 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vol2 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick rhs-client11:/rhs/device1/b1 49153 Y 4079 Brick rhs-client12:/rhs/device1/b2 49153 Y 18508 Brick rhs-client13:/rhs/device1/b3 49153 Y 25348 Brick rhs-client14:/rhs/device1/b4 49153 Y 31190 NFS Server on localhost 2049 Y 1570 Self-heal Daemon on localhost N/A Y 1578 NFS Server on rhs-client14 2049 Y 2906 Self-heal Daemon on rhs-client14 N/A Y 2916 NFS Server on rhs-client12 2049 Y 18664 Self-heal Daemon on rhs-client12 N/A Y 18673 NFS Server on rhs-client13 2049 Y 21453 Self-heal Daemon on rhs-client13 N/A Y 21463 NFS Server on rhs-client11 2049 Y 7998 Self-heal Daemon on rhs-client11 N/A Y 8005 Task Status of Volume vol2 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vol3 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick rhs-client11:/rhs/device2/b1 49154 Y 4150 Brick rhs-client12:/rhs/device2/b2 49154 Y 18583 Brick rhs-client13:/rhs/device2/b3 49154 Y 25417 Brick rhs-client14:/rhs/device2/b4 49154 Y 31257 NFS Server on localhost 2049 Y 1570 Self-heal Daemon on localhost N/A Y 1578 NFS Server on rhs-client13 2049 Y 21453 Self-heal Daemon on rhs-client13 N/A Y 21463 NFS Server on rhs-client14 2049 Y 2906 Self-heal Daemon on rhs-client14 N/A Y 2916 NFS Server on rhs-client12 2049 Y 18664 Self-heal Daemon on rhs-client12 N/A Y 18673 NFS Server on rhs-client11 2049 Y 7998 Self-heal Daemon on rhs-client11 N/A Y 8005 Task Status of Volume vol3 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vol4 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick rhs-client11:/rhs/device3/b1 49155 Y 4220 Brick rhs-client12:/rhs/device3/b2 49155 Y 18651 Brick rhs-client13:/rhs/device3/b3 49155 Y 25493 Brick rhs-client14:/rhs/device3/b4 49155 Y 31327 NFS Server on localhost 2049 Y 1570 Self-heal Daemon on localhost N/A Y 1578 NFS Server on rhs-client13 2049 Y 21453 Self-heal Daemon on rhs-client13 N/A Y 21463 NFS Server on rhs-client14 2049 Y 2906 Self-heal Daemon on rhs-client14 N/A Y 2916 NFS Server on rhs-client12 2049 Y 18664 Self-heal Daemon on rhs-client12 N/A Y 18673 NFS Server on rhs-client11 2049 Y 7998 Self-heal Daemon on rhs-client11 N/A Y 8005 Task Status of Volume vol4 ------------------------------------------------------------------------------ There are no active volume tasks root@mia [Aug-26-2014-16:43:08] > Refer to bug https://bugzilla.redhat.com/show_bug.cgi?id=923809#c8 for more context.
SOS Reports: http://rhsqe-repo.lab.eng.blr.redhat.com/bugs_necessary_info/1133874/ Client1: dj Client2: king BRICK1 : rhs-client11 BRICK2 : rhs-client12 BRICK3 : rhs-client13 BRICK4 : rhs-client14 MGMT_NODE : mia
Upstream patch link : http://review.gluster.org/5956
After having a discussion with Pranith we came to an agreement that this is something which needs to be fixed at client side. Client xlator first connects to glusterd to get the port. If client xlator sends fops even before server handshake completes, it would lead to these problems and error logs would be seen. Since the fix is not in the scope of GlusterD removing ateam from devel whiteboard.
(In reply to Atin Mukherjee from comment #4) > After having a discussion with Pranith we came to an agreement that this is > something which needs to be fixed at client side. Client xlator first > connects to glusterd to get the port. If client xlator sends fops even > before server handshake completes, it would lead to these problems and error > logs would be seen. Since the fix is not in the scope of GlusterD removing > ateam from devel whiteboard. Duplicate of bz 1583937