I logged into the setup and looked into  a bit.
Something seems wrong with ovs-vswitchd. ovn-controller is breaking connection with the ovs-vswitchd (openflow connection) and it is reconnecting all the time.
That is why we are seeing high cpu usage in ovn-controller.
Looks like we need to investigate ovs-vswitchd and see what is going on there.

I don't think this is OVN issue.

I have tried with dell-per740-12.rhts.eng.pek2.redhat.com and dell-per730-40.rhts.eng.pek2.redhat.com to reproduce the issue. the adding of logical ports are still in progress. but the cpu usage for ovs-vswitched is already very high. you can log in with root/redhat

Hi,

Looks like runtest.sh is trying to add one port at a time using ovs-vsctl command and already added 1762 ports.
The problem is that every time a port is added, OvS has to reconfigure itself, flush data like FDB and so on.
The load issue is an old complain and we already have a few bugs about that, but the questions remain open.

See: https://bugzilla.redhat.com/show_bug.cgi?id=1731165

I am not sure about the goal of the test. If you just want to add as many ports as possible to run other tests
maybe the batch solution proposed in the BZ above solves the issue.

Thanks
fbl

(In reply to Flavio Leitner from comment #5)
> Hi,
> 
> Looks like runtest.sh is trying to add one port at a time using ovs-vsctl
> command and already added 1762 ports.
> The problem is that every time a port is added, OvS has to reconfigure
> itself, flush data like FDB and so on.
> The load issue is an old complain and we already have a few bugs about that,
> but the questions remain open.
> 
> See: https://bugzilla.redhat.com/show_bug.cgi?id=1731165
> 
> I am not sure about the goal of the test. If you just want to add as many
> ports as possible to run other tests
> maybe the batch solution proposed in the BZ above solves the issue.
> 
> Thanks
> fbl

confirmed that cpu usage would be low after finish adding ports.
but it's really slow to add mass of ports, any bug to track that?

(In reply to Jianlin Shi from comment #6)
> confirmed that cpu usage would be low after finish adding ports.
> but it's really slow to add mass of ports, any bug to track that?

The recommendation is to add ports in batches then.
It's one single operation inside OvS to add the batch.

fbl

(In reply to Flavio Leitner from comment #7)
> (In reply to Jianlin Shi from comment #6)
> > confirmed that cpu usage would be low after finish adding ports.
> > but it's really slow to add mass of ports, any bug to track that?
> 
> The recommendation is to add ports in batches then.
> It's one single operation inside OvS to add the batch.
> 
> fbl

ok

(In reply to Jianlin Shi from comment #8)
Can we close the bug?
fbl

(In reply to Flavio Leitner from comment #9)
> (In reply to Jianlin Shi from comment #8)
> Can we close the bug?
> fbl

yes, please close it per comment 7

Thank you!
fbl

*** Bug 1749610 has been marked as a duplicate of this bug. ***