Description of problem: When running BenE's GFAPI benchmark over RDMA we are seeing segfaults when run with greater than two threads. Version-Release number of selected component (if applicable): glusterfs-3.6.0.38-1.el6rhs.x86_64 How reproducible: Every time we tested with greater than 2 threads. Steps to Reproduce: 1. git clone https://github.com/bengland2/parallel-libgfapi 2. gcc -pthread -g -O0 -Wall --pedantic -o gfapi_perf_test -I /usr/include/glusterfs/api gfapi_perf_test.c -lgfapi -lrt 3. export GFAPI_HOSTNAME=gqas015 4. export GFAPI_VOLNAME=testvol 5. PGFAPI_PROCESSES=8 PGFAPI_FILES=1 PGFAPI_RECSZ=16384 PGFAPI_TOPDIR=/ben PGFAPI_FILESIZE=4194304 PGFAPI_MOUNTPOINT=/gluster-mount PGFAPI_LOAD=seq-wr PGFAPI_DIRECT=1 ./parallel_gfapi_test.sh Actual results: The GFAPI benchmark app segfaults. Expected results: The GFAPI benchmark behaves similar to TCP. Additional info:
Here is what happens when I run the benchmark: [root@gqac022 parallel-libgfapi-master]# PGFAPI_PROCESSES=8 PGFAPI_FILES=1 PGFAPI_RECSZ=16384 PGFAPI_TOPDIR=/ben PGFAPI_FILESIZE=4194304 PGFAPI_MOUNTPOINT=/gluster-mount PGFAPI_LOAD=seq-wr PGFAPI_DIRECT=1 ./parallel_gfapi_test.sh /usr/local/bin/gfapi_perf_test volume name: testvol Gluster server in the volume: gqas015 workload: seq-wr list of clients in file: clients.list record size (KB): 64 file size (KB): 4194304 files per thread: 1 processes per client: 8 threads per process: 1 test driver glusterfs mountpoint: /gluster-mount top directory within Gluster volume: /ben each thread (process) runs program at: gfapi_perf_test using direct I/O log files for each libgfapi process at /tmp/parallel_gfapi_logs.6286 starting gun timeout = 12 removing any previous files ./parallel_gfapi_test.sh: line 163: 6319 Segmentation fault (core dumped) GFAPI_LOAD=unlink GFAPI_FUSE=0 GFAPI_FILES=1 GFAPI_BASEDIR=/ben/smf-gfapi-localhost.06 GFAPI_VOLNAME=testvol GFAPI_HOSTNAME=gqas015 GFAPI_THREADS_PER_PROC=1 gfapi_perf_test > /tmp/unlink.localhost.06.log 2>&1 log directory is /tmp/par-for-all.6425 Mon Dec 15 16:47:20 EST 2014: starting 1 clients ... localhost ls: cannot access /gluster-mount//ben/*.ready: No such file or directory Mon Dec 15 16:47:25 EST 2014: clients are all ready Mon Dec 15 16:47:25 EST 2014 : clients should all start running within a few seconds process 6509 exited with status 255 process 6526 exited with status 255 process 6560 exited with status 255 Mon Dec 15 16:48:38 EST 2014: clients completed ERROR: at least one process exited with error status 255 per-thread results in /tmp/parallel_gfapi_logs.6286/result.csv 5 threads finished out of 8 transfer-rate: 357.00 MBytes/s file-rate: 5712.12 files/sec IOPS: 0.00 requests/sec