Bug 731808 - Using Solarflare 10Gb card, MRG-M perf test fails when testing packet sizes grater than 512K
Summary: Using Solarflare 10Gb card, MRG-M perf test fails when testing packet sizes g...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: realtime-kernel
Version: Development
Hardware: All
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Red Hat Real Time Maintenance
QA Contact: Jiri Kastner
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-08-18 18:33 UTC by Tom Tracy
Modified: 2014-09-04 13:26 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-09-04 13:26:57 UTC


Attachments (Terms of Use)

Description Tom Tracy 2011-08-18 18:33:12 UTC
Description of problem:
Using Solarflare 10Gb card, MRG-M perf test fails when testing packet sizes grater than 512MB

Version-Release number of selected component (if applicable):

2.6.33.9-rt31.74.el6rt.x86_64


How reproducible:

consistently

Steps to Reproduce:
1. boot RT kernel
2. Start qpidd broker one 1 node 
3./usr/sbin/qpidd --data-dir /var/lib/qpidd --auth no -m no --log-enable=info+
  
Actual results:

Testing runs in a loop for 100 times, sometime into the run get the following error on the broker node

2011-08-18 13:56:29 error Error reading socket: Bad address(14)
2011-08-18 14:00:20 notice Shut down

There are no messages relating to this issue in dmesg or /var/log/messages. 

Expected results: Should be able to run a complete suite of testing with various packet sizes ranging from 8 Bytes to 32K. This can be completed using non-RT kernel. Also, do not see this issue with Mellanox 10Gb cards.


Additional info:

Comment 1 Tom Tracy 2011-08-18 18:35:35 UTC
The packet size should state to be 512K or greater

Comment 2 Arnaldo Carvalho de Melo 2011-08-18 18:56:41 UTC
Which non-RT kernel was used?

Comment 3 Arnaldo Carvalho de Melo 2011-08-18 18:58:23 UTC
And what is the client used? Some standard test found in the qpidd packages? What command line was used with it?

Comment 4 Tom Tracy 2011-08-18 19:39:15 UTC
On the broker side, 

/usr/sbin/qpidd --data-dir /var/lib/qpidd --auth no -m no --log-enable=info+


Client side

[root@perf46 ~]# cat run_amqp_perf_perf48.sh
#!/bin/bash
for i in 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768;
#for i in 1024 2048 4096 8192 16384 32768;
#for i in 32768;
do echo $i;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
.
.
.


testNnics.sh
#!/bin/sh
# Usage: 
usage() {
    echo "Usage: $0 pubsubs_per_nic count size nic1 [ nic2 ... ]"
    echo "Uses $PERFTEST if set in env."
    exit 1
}
test $# -lt 4 && usage 

pubsubs_per_nic=$1; shift
count=$1; shift
size=$1; shift
nclient=`expr $pubsubs_per_nic \* $#`

#echo "Run $nclient pubs & subs with $pubsubs_per_nic on each NIC: $* "
test "`expr $count / $nclient`" = 0 || {
    count=`expr $count + $nclient - \( $count % $nclient \)`
#   echo "WARNING: Adjusted count to  $count, nearest multiple of client count $nclient"
}

perftest=${PERFTEST:-/usr/bin/qpid-perftest --tcp-nodelay}
#perftest=${PERFTEST:-/usr/bin/qpid-perftest -P rdma}


args="--nsubs $nclient --npubs $nclient --count $count --size $size --qt 16 --summary"

# Can use any nic for --setup and --control clients
nic1=$1
$perftest --setup -b $nic1 $args --bounds-multiplier 102400 # NB: not in background.

# Split pubsubs_per_nic pubs & subs for each nics.
while [ -n "$*" ]; do
    nic=$1; shift
    for (( i = 0 ; i < pubsubs_per_nic ; ++i )) ; do
	$perftest --publish -b $nic $args &
	$perftest --subscribe -b $nic $args &
    done
done


$perftest --control -b $nic1 $args


Tried reducing the number of queues (qt) to default of 4 and still see the same issues

Tom

Comment 5 Tom Tracy 2011-08-18 20:16:42 UTC
With the 2.6.33.9-rt31.66.el6rt.x86_64 still see the same issue

[root@perf48 ~]# /usr/sbin/qpidd --data-dir /var/lib/qpidd --auth no -m no
2011-08-18 16:06:48 notice SASL disabled: No Authentication Performed
2011-08-18 16:06:48 notice Listening on TCP port 5672
2011-08-18 16:06:48 notice Broker running
2011-08-18 16:12:10 error Error reading socket: Bad address(14)

Comment 6 John Kacur 2014-09-04 13:26:16 UTC
This is too old, out of scope. Perhaps the test is worth snagging


Note You need to log in before you can comment on or make changes to this bug.