Hide Forgot
Description of problem: I/O error while parallel create 1000 qcow2 images by gluster. Version-Release number of selected component (if applicable): rhs: glusterfs-fuse-3.4.0.36rhs-1.el6rhs.x86_64 glusterfs-libs-3.4.0.36rhs-1.el6rhs.x86_64 glusterfs-server-3.4.0.36rhs-1.el6rhs.x86_64 glusterfs-3.4.0.36rhs-1.el6rhs.x86_64 glusterfs-rdma-3.4.0.36rhs-1.el6rhs.x86_64 glusterfs-api-3.4.0.36rhs-1.el6rhs.x86_64 glusterfs-geo-replication-3.4.0.36rhs-1.el6rhs.x86_64 host: qemu-kvm-0.12.1.2-2.415.el6_5.2.x86_64 glusterfs-fuse-3.4.0.36rhs-1.el6.x86_64 glusterfs-api-3.4.0.36rhs-1.el6.x86_64 glusterfs-libs-3.4.0.36rhs-1.el6.x86_64 glusterfs-rdma-3.4.0.36rhs-1.el6.x86_64 glusterfs-3.4.0.36rhs-1.el6.x86_64 How reproducible: always Steps to Reproduce: 1.create qcow2 image #!/bin/bash COUNT=0 while [ $COUNT -lt 1000 ] do qemu-img create -f qcow2 gluster://rhs/gv0/test-$COUNT 2M & COUNT=$((1+$COUNT)) done 2. 3. Actual results: Got I/O error while creating qcow2: Input/output error gluster://rhs/gv0/test-141: error while creating qcow2: Input/output error gluster://rhs/gv0/test-375: error while creating qcow2: Input/output error gluster://rhs/gv0/test-45: error while creating qcow2: Input/output error gluster://rhs/gv0/test-111: error while creating qcow2: Input/output error gluster://rhs/gv0/test-160: error while creating qcow2: Input/output error gluster://rhs/gv0/test-93: error while creating qcow2: Input/output errorgluster://rhs/gv0/test-270: error while creating qcow2: Input/output error gluster://rhs/gv0/test-12: error while creating qcow2: Input/output error gluster://rhs/gv0/test-36: error while creating qcow2: Input/output errorgluster://rhs/gv0/test-44: error while creating qcow2: Input/output error gluster://rhs/gv0/test-168: error while creating qcow2: Input/output error gluster://rhs/gv0/test-280: error while creating qcow2: Input/output error gluster://rhs/gv0/test-133: error while creating qcow2: Input/output error gluster://rhs/gv0/test-14: error while creating qcow2: Input/output errorFormatting 'gluster://rhs/gv0/test-684', fmt=qcow2 size=2048 encryption=off cluster_size=65536 Expected results: no error Additional info: mount glusterfs on local directory, then create images not hit this problem
about parallel create 100 images would triggered this problem.
dmesg got following. ... qemu-img[5720]: segfault at 20 ip 00007f9895762116 sp 00007fff02a24420 error 4 in libgfapi.so.0.0.0[7f9895755000+13000]
I am able to easily reproduce the original problem in comment #0. This is mainly a configuration issue. By default, a gluster volume will not allow incoming connections from unprivileged ports (src ports > 1024). When you create a large number of simultaneous connections, even though operating as root the source port numbers will wrap around (counting downwards) to unprivileged ports. To prevent this from being rejected, you can configure the gluster volume to allow connections from unprivileged ports. From the gluster server, run: # gluster volume set gv0 server.allow-insecure on On comment #3: if this issue is still seen, that is probably a cause for a different / new BZ. Maosheng, could you test after setting the gluster server to allow insecure port connections (as described above)? If that does fix it (as it did for me), I would suggest either closing this as NOTABUG. Alternatively, if the ability to have that many simultaneous connections to the gluster server via libglusterfs is desired as default operation, then I would reassign this to the gluster/rhs team to change the default values. If you are still seeing the issue from comment #3, please open a new BZ for it.
Hit comment#3 after allow connections from unprivileged ports, file a new bug bz1088817 .
Red Hat Enterprise Linux 6 is in the Production 3 Phase. During the Production 3 Phase, Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available. The official life cycle policy can be reviewed here: http://redhat.com/rhel/lifecycle This issue does not meet the inclusion criteria for the Production 3 Phase and will be marked as CLOSED/WONTFIX. If this remains a critical requirement, please contact Red Hat Customer Support to request a re-evaluation of the issue, citing a clear business justification. Note that a strong business justification will be required for re-evaluation. Red Hat Customer Support can be contacted via the Red Hat Customer Portal at the following URL: https://access.redhat.com/