Idea is to catch real requests that appears in ovn cluster in scaling scenarios and replicate them using ovsdb-client or pure network packets without bringing up the whole [fake] multinode cluster. This should allow to test ovsdb-sb performance on a fairly small systems without bothering the scale lab.
I'm working on third generation of the tests concept.
First idea was to catch requests to ovsdb-sb, parse and re-play using external tools
like ovsdb-client. However, it turned out to be not that convenient to dump
the traffic, parse it with some external scripts and run the system. Also,
it's not clear how to measure the actual work done by the ovsdb since there might
be a lot of spikes and delays caused by connections and actual running of big
number of clients on the same machine.
Second idea was to generate synthetic database and transactions to approximate
the ovsdb-sb load. But this is not that good too since we're changing the ways
how ovn-controller interacts with db and it's actually hard to estimate what
will happen in a real setup.
The last approach I working on right now is to create re-play scenarios right
from the inside of ovsdb. What I've implemented is a new cmdline argument
(--stream-replay-record) that makes ovsdb to write all the received connections and
requests to special "replay" files via altered stream implementation. All the
writes are done in the exact same order as connections and data appears on sockets.
With passing another argument (--stream-replay) ovsdb will start "receiving" same
connections and same data in exactly same order, but without any delays and
actual communications. Assuming that the database is in the same original state,
ovsdb should go through exactly same load as in a run where "replay" was recorded.
Performance in this case could be measured by catching the end of the last recorded
I have a draft implementation here:
One positive thing about "replay" solution is that it actually could be used not
only for performance testing but also for debugging in case you have recorded bad
conditions and replaying them locally, maybe with debugger attached or more logs
Since implementation is on a "stream" level, this will record all the db connections,
openflow and unixctl. ovsdb-server is a good application to record because it has
no other sources of events beside stream. ovn-controller might be also a good candidate.
The first version of stream record/replay functionality posted upstream:
v2 posted upstream:
v3 of the OVSDB record/replay functionality got accepted and will be part
of upstream 2.16 release. Integration of this feature into OVN daemons
and tests is tracked in BZ 1980793.