Bug 731808

Summary: Using Solarflare 10Gb card, MRG-M perf test fails when testing packet sizes grater than 512K
Product: Red Hat Enterprise MRG Reporter: Tom Tracy <ttracy>
Component: realtime-kernelAssignee: Red Hat Real Time Maintenance <rt-maint>
Status: CLOSED WONTFIX QA Contact: Jiri Kastner <jkastner>
Severity: high Docs Contact:
Priority: unspecified    
Version: DevelopmentCC: acme, bhu, dledford, jkacur, jkastner, lgoncalv, williams
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-04 13:26:57 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Tom Tracy 2011-08-18 18:33:12 UTC
Description of problem:
Using Solarflare 10Gb card, MRG-M perf test fails when testing packet sizes grater than 512MB

Version-Release number of selected component (if applicable):

2.6.33.9-rt31.74.el6rt.x86_64


How reproducible:

consistently

Steps to Reproduce:
1. boot RT kernel
2. Start qpidd broker one 1 node 
3./usr/sbin/qpidd --data-dir /var/lib/qpidd --auth no -m no --log-enable=info+
  
Actual results:

Testing runs in a loop for 100 times, sometime into the run get the following error on the broker node

2011-08-18 13:56:29 error Error reading socket: Bad address(14)
2011-08-18 14:00:20 notice Shut down

There are no messages relating to this issue in dmesg or /var/log/messages. 

Expected results: Should be able to run a complete suite of testing with various packet sizes ranging from 8 Bytes to 32K. This can be completed using non-RT kernel. Also, do not see this issue with Mellanox 10Gb cards.


Additional info:

Comment 1 Tom Tracy 2011-08-18 18:35:35 UTC
The packet size should state to be 512K or greater

Comment 2 Arnaldo Carvalho de Melo 2011-08-18 18:56:41 UTC
Which non-RT kernel was used?

Comment 3 Arnaldo Carvalho de Melo 2011-08-18 18:58:23 UTC
And what is the client used? Some standard test found in the qpidd packages? What command line was used with it?

Comment 4 Tom Tracy 2011-08-18 19:39:15 UTC
On the broker side, 

/usr/sbin/qpidd --data-dir /var/lib/qpidd --auth no -m no --log-enable=info+


Client side

[root@perf46 ~]# cat run_amqp_perf_perf48.sh
#!/bin/bash
for i in 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768;
#for i in 1024 2048 4096 8192 16384 32768;
#for i in 32768;
do echo $i;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
/root/testNnics.sh 1 100000 $i  192.168.10.48;
.
.
.


testNnics.sh
#!/bin/sh
# Usage: 
usage() {
    echo "Usage: $0 pubsubs_per_nic count size nic1 [ nic2 ... ]"
    echo "Uses $PERFTEST if set in env."
    exit 1
}
test $# -lt 4 && usage 

pubsubs_per_nic=$1; shift
count=$1; shift
size=$1; shift
nclient=`expr $pubsubs_per_nic \* $#`

#echo "Run $nclient pubs & subs with $pubsubs_per_nic on each NIC: $* "
test "`expr $count / $nclient`" = 0 || {
    count=`expr $count + $nclient - \( $count % $nclient \)`
#   echo "WARNING: Adjusted count to  $count, nearest multiple of client count $nclient"
}

perftest=${PERFTEST:-/usr/bin/qpid-perftest --tcp-nodelay}
#perftest=${PERFTEST:-/usr/bin/qpid-perftest -P rdma}


args="--nsubs $nclient --npubs $nclient --count $count --size $size --qt 16 --summary"

# Can use any nic for --setup and --control clients
nic1=$1
$perftest --setup -b $nic1 $args --bounds-multiplier 102400 # NB: not in background.

# Split pubsubs_per_nic pubs & subs for each nics.
while [ -n "$*" ]; do
    nic=$1; shift
    for (( i = 0 ; i < pubsubs_per_nic ; ++i )) ; do
	$perftest --publish -b $nic $args &
	$perftest --subscribe -b $nic $args &
    done
done


$perftest --control -b $nic1 $args


Tried reducing the number of queues (qt) to default of 4 and still see the same issues

Tom

Comment 5 Tom Tracy 2011-08-18 20:16:42 UTC
With the 2.6.33.9-rt31.66.el6rt.x86_64 still see the same issue

[root@perf48 ~]# /usr/sbin/qpidd --data-dir /var/lib/qpidd --auth no -m no
2011-08-18 16:06:48 notice SASL disabled: No Authentication Performed
2011-08-18 16:06:48 notice Listening on TCP port 5672
2011-08-18 16:06:48 notice Broker running
2011-08-18 16:12:10 error Error reading socket: Bad address(14)

Comment 6 John Kacur 2014-09-04 13:26:16 UTC
This is too old, out of scope. Perhaps the test is worth snagging