Bug 1494846

Summary: Sending durable messages via AMQP 1.0 on 1.36.0-9 broker is tragically slow
Product: Red Hat Enterprise MRG Reporter: Pavel Moravec <pmoravec>
Component: qpid-cppAssignee: messaging-bugs <messaging-bugs>
Status: CLOSED NOTABUG QA Contact: Messaging QE <messaging-qe-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 3.2CC: jross, pematous
Target Milestone: ---Keywords: Performance, Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-25 09:39:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2017-09-23 13:26:09 UTC
Description of problem:
Sending durable messages via AMQP 1.0 on 1.36.0-9 broker exhibit tragic performance / message throughput of approx 10msgs/s. It is a regression from 1.36.0-6 broker that shows normal message throughput.

From tests performed, the broker extremely delays sending message acknowledgments back to the probducer, what causes the tragic performance results.


Version-Release number of selected component (if applicable):
qpid-cpp-*-1.36.0-9


How reproducible:
100%


Steps to Reproduce:
rm -rf /var/lib/qpidd/*
service qpidd restart
qpid-config add queue PerfQueue --durable
time qpid-send -a PerfQueue --durable=yes -m 100 --connection-options "{protocol:'amqp1.0'}"


Actual results:
the qpid-send takes from 6 seconds to 15, with average around 9-10


Expected results:
qpid-send to finish within a fraction of second


Additional info:
- sending transient messages (or to non-durable queue) is fast
- sending the messages via AMQP 0-10 is afst as well
- using 1.36.0-6 broker (in all the scenarios) is fast as well

Comment 2 Pavel Moravec 2017-09-23 13:53:19 UTC
I might used invalid config, since the performance depends highly on setting of journal-flush-timeout parameter.

journal-flush-timeout=1
journal-flush-timeout=15
journal-flush-timeout=500

shows as bad performance as the value is. While using "1ms" there (or "15ms") shows good performance.

(that config is quite confusing since it is backward incompatible as 1.36.0-6 doesnt accept "ms" at all - upgrading qpidd with the same parameter value would bring this problem)

Comment 4 Pavel Moravec 2017-09-25 09:39:45 UTC
Closing as it works as expected, config error behind the report.