Bug 2223578

Summary: [RFE] [OVN-IC] [Debugging] Enhance ovn-*bctl to connect across multiple sets of DBs
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Surya Seetharaman <surya>
Component: OVNAssignee: OVN Team <ovnteam>
Status: NEW --- QA Contact: Jianlin Shi <jishi>
Severity: unspecified Docs Contact:
Priority: medium    
Version: FDP 23.ACC: ctrautma, dcbw, jiji, mmichels
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Surya Seetharaman 2023-07-18 10:17:06 UTC 
Description of problem:
As OVNK moves to using the Interconnect architecture, we are going to have to pay attention to the debugging aspects of OVN a bit closer.

Since scale is primary aim for the IC architecture; imagine a 500 or 1000 node cluster where we will collect 500 or 1000 NBDB's and SBDB's, we need to be able to run the bctl commands against all of them at the same time

See https://redhat-internal.slack.com/archives/C050MC61LVA/p1689675049225319?thread_ts=1689626101.828479&cid=C050MC61LVA for details

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Dan Williams 2023-07-21 13:49:51 UTC 
Wouldn't it be trivial to write a Go wrapper like:

multi-ctl -p 9641 ovn-nbctl -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt list Logical_Switch_Port

which then

(1) reads kAPI for list of nodes, grabs their internal address (or somehow gets the IP to use)
(2) constructs an actual args list with argv but putting "--db=<node IP>:9641" into the list at the right place
(3) optionally constructs a logfile path unique to each invocation, perhaps based on node name
(4) starts N goroutines where N = # nodes; executes the *ctl util and writes output to named logfile
(5) waits till they all finish
(6) maybe tars up all the logfiles and gzips it

I'd think it would be a lot easier in Go than in Bash or C.