[This report is mostly a copy of https://github.com/ovn-org/ovn/issues/83] ovn-ctl CLI has parameters --db-nb-create-insecure-remote and --db-sb-create-insecure-remote which can take values yes or no (default). The name of these parameters strongly suggests that when set to yes, communication to the databases will be secured via SSL/TLS over TCP. However, when the same parameters are set to yes, no TCP socket is created. Let's start with an insecure remote with key and certificates set: /usr/share/ovn/scripts/ovn-ctl \ --db-nb-create-insecure-remote=yes \ --db-nb-addr=192.168.14.32 \ --db-nb-port=6641 \ --ovn-nb-db-ssl-key=/root/testca/testcert.key \ --ovn-nb-db-ssl-cert=/root/testca/testcert.pem \ --ovn-nb-db-ssl-ca-cert=/root/testca/ca.cert.pem \ run_nb_ovsdb ovn-ctl spawned an ovsdb-server process with the following parameters: + exec ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/ovn/ovsdb-server-nb.log --remote=punix:/var/run/ovn/ovnnb_db.sock --pidfile=/var/run/ovn/ovnnb_db.pid --unixctl=/var/run/ovn/ovnnb_db.ctl --remote=db:OVN_Northbound,NB_Global,connections --private-key=/root/testca/testcert.key --certificate=/root/testca/testcert.pem --ca-cert=/root/testca/ca.cert.pem --ssl-protocols=db:OVN_Northbound,SSL,ssl_protocols --ssl-ciphers=db:OVN_Northbound,SSL,ssl_ciphers --remote=ptcp:6641:192.168.14.32 /etc/ovn/ovnnb_db.db The TCP socket is created and listening on 192.168.14.32:6641 as expected: [root@ovn1 ~]# ss -natulpe '( sport = 6641 )' Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port tcp LISTEN 0 10 192.168.14.32:6641 0.0.0.0:* users:(("ovsdb-server",pid=84022,fd=15)) ino:76722480 sk:99 <-> Even though certificates and key were passed in, ovsdb-server does not load them up (as expected): [root@ovn1 ~]# openssl s_client -connect 192.168.14.32:6641 CONNECTED(00000003) write:errno=0 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 289 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- Now, let's start ovsdb-server in secure remote mode with the same certificates and key set: /usr/share/ovn/scripts/ovn-ctl \ --db-nb-create-insecure-remote=no \ --db-nb-addr=192.168.14.32 \ --db-nb-port=6641 \ --ovn-nb-db-ssl-key=/root/testca/testcert.key \ --ovn-nb-db-ssl-cert=/root/testca/testcert.pem \ --ovn-nb-db-ssl-ca-cert=/root/testca/ca.cert.pem \ run_nb_ovsdb This time there is no TCP socket: [root@ovn1 ~]# ss -natulpe '( sport = 6641 )' Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port [root@ovn1 ~]# The problem appears to be that ovn-ctl spawned an ovsdb-server process without a proper --remote= value set: + exec ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/ovn/ovsdb-server-nb.log --remote=punix:/var/run/ovn/ovnnb_db.sock --pidfile=/var/run/ovn/ovnnb_db.pid --unixctl=/var/run/ovn/ovnnb_db.ctl --remote=db:OVN_Northbound,NB_Global,connections --private-key=/root/testca/testcert.key --certificate=/root/testca/testcert.pem --ca-cert=/root/testca/ca.cert.pem --ssl-protocols=db:OVN_Northbound,SSL,ssl_protocols --ssl-ciphers=db:OVN_Northbound,SSL,ssl_ciphers /etc/ovn/ovnnb_db.db
Additionally, we tried to workaround this issue by executing the following command on each OpenStack controller node: $ ovn-nbctl set-connection pssl:6641:$CTRL_IP In doing so, the OVN databases now listen on port 6641 but are bound to the ctrl-1-0 IP address (172.30.1.1): [root@ctrl-3-0 ~]# ss -natulpe |grep 664 tcp LISTEN 0 10 172.30.1.1:6641 0.0.0.0:* users:(("ovsdb-server",pid=118443,fd=14)) ino:469572 sk:3e <-> tcp LISTEN 0 10 172.30.1.1:6642 0.0.0.0:* users:(("ovsdb-server",pid=121404,fd=13)) ino:482657 sk:43 <-> tcp LISTEN 0 10 172.30.3.1:6643 0.0.0.0:* users:(("ovsdb-server",pid=118443,fd=16)) ino:469577 sk:45 <-> tcp LISTEN 0 10 172.30.3.1:6644 0.0.0.0:* users:(("ovsdb-server",pid=121404,fd=16)) ino:482663 sk:47 <-> It was suggested using "ovn-nbctl set-connection pssl:6641" to force the database servers to listen on 0.0.0.0:6641. That is though not desired for security reasons.
(In reply to Carlos Goncalves from comment #1) > Additionally, we tried to workaround this issue by executing the following > command on each OpenStack controller node: > > $ ovn-nbctl set-connection pssl:6641:$CTRL_IP > > In doing so, the OVN databases now listen on port 6641 but are bound to the > ctrl-1-0 IP address (172.30.1.1): Can you clarify this? What was the value of $CTRL_IP given to set-connection?
Value of $CTRL_IP was 172.30.1.1 (ctrl-1-0, OpenStack controller / bootstrap node): $ ovn-nbctl set-connection pssl:6641:172.30.1.1 Once we execute the above command on ctrl-1-0, the OVN northbound database server on all three OpenStack controllers bound to 172.30.1.1:6641. The same is also valid for the southbound database server on port 6642. Output from ctrl-3-0 in comment #1 shows that.
There are two things here: 1. It's not possible to tweak advanced parameters of connections via ovsdb command line arguments (e.g. inactivity probes) and also it's not possible to add/del remotes in runtime. To solve this ovsdb supports getting configuration of remotes from the database via --remote=db:<database>,<table>,<row> 2. Since the table that '--remote=db:...' points to is typically part of the same Northbound/Southbound database, it gets replicated to all the servers, i.e. content of this row is identical on all cluster members and all of them starts listening on the exactly same addresses and ports, which is obviously a problem. The same problem is valid for active-backup model. Solution might be to add one more standalone database that will have only SSL and Connections tables (maybe one more to hold list of connections?). Each ovsdb-server will have this one database for itself, so each server could be configured separately with different IPs and ports. Something like this: $ cat connections.ovsschema { "name": "Connections", "version": "1.0.0", "cksum": "<checksum>", "tables": { "Connections": { "columns": { "connections": { "type": {"key": {"type": "uuid", "refTable": "Connection"}, "min": 0, "max": "unlimited"}}, "ssl": { "type": {"key": {"type": "uuid", "refTable": "SSL"}, "min": 0, "max": 1}}}, "maxRows": 1, "isRoot": true}, "Connection": { "columns": { "target": {"type": "string"}, "max_backoff": {"type": {"key": {"type": "integer", "minInteger": 1000}, "min": 0, "max": 1}}, "inactivity_probe": {"type": {"key": "integer", "min": 0, "max": 1}}, "other_config": {"type": {"key": "string", "value": "string", "min": 0, "max": "unlimited"}}, "external_ids": {"type": {"key": "string", "value": "string", "min": 0, "max": "unlimited"}}, "is_connected": {"type": "boolean", "ephemeral": true}, "status": {"type": {"key": "string", "value": "string", "min": 0, "max": "unlimited"}, "ephemeral": true}}, "indexes": [["target"]]}, "SSL": { "columns": { "private_key": {"type": "string"}, "certificate": {"type": "string"}, "ca_cert": {"type": "string"}, "bootstrap_ca_cert": {"type": "boolean"}, "ssl_protocols": {"type": "string"}, "ssl_ciphers": {"type": "string"}, "external_ids": {"type": {"key": "string", "value": "string", "min": 0, "max": "unlimited"}}}, "maxRows": 1}, } } And the ovsdb-server invokation will look like this: ovsdb-server -vconsole:off -vfile:info \ --log-file=/var/log/ovn/ovsdb-server-nb.log \ --remote=punix:/var/run/ovn/ovnnb_db.sock \ --pidfile=/var/run/ovn/ovnnb_db.pid \ --unixctl=/var/run/ovn/ovnnb_db.ctl \ --private-key=/root/testca/testcert.key \ --certificate=/root/testca/testcert.pem \ --ca-cert=/root/testca/ca.cert.pem \ --ssl-protocols=db:Connections,SSL,ssl_protocols \ --ssl-ciphers=db:Connections,SSL,ssl_ciphers \ --remote=db:Connections,Connections,connections \ /etc/ovn/ovnnb_db.db /etc/ovn/ovnnb_connections.db 2 databases: one for Northbound database and one for local configuration of remotes for this server. First is clustered and the second one is standalone. ovn-ctl could be taught to do that. 'ovn-nbctl set-connection' command will need some changes to choose the database, or we need a separate utility to talk with "Connections" database (ovn-connctl?). Any thoughts?
I never read create-insecure-remote as a "use insecure or secure" flag, just "add this one insecure remote". With that said, now that we have Local_Config which looks similar to c4, there are some issues. 1) If we have ovn-ctl with a --use-local-config option, presumably --db-${DB}-use-remote-in-db would point to the new Local_Config,Config,connections remote. a) This means that things like ovn-nbctl set-connection would write to NB.Connections, which would do nothing--which is a bit confusing from a support/documentation standpoint. 2) Currently, we don't really set up connections, except via CLI which is limited (and why Local_Config exists), so it seems like it would be better to just specify DB_${DB}_PROTOCOL=(p)tcp|ssl and then either add it to NB.Connections or Local_Config.Connections depending on --use-local-config. a) The set-connection operation on *ctl is destructive and sets up only one connection. It is technically possible for someone to add multiple connections. It gets complex if you want to handle the situation where someone modifies the connection info passed in /etc/sysconfig/ovn, and you want to also not destroy connections that could have been added manually. We could punt on having ovn-ctl do anything w/ Local_Config other than create the DB if needed/point to it for --remote, though it would mean tripleo-heat-templates would need to run something annoying like: podman exec ovn_cluster_north_db_server ovs-confctl --db $correct_unix_sock_addr set-connection ... Another option is to just set up the local_config.connection at db creation with ovsdb-tool in ovn-ctl and make it clear that after that, the connection is managed with ovs-confctl. Or just have do a ovs-confctl get-connection/set_conenction if it is empty after starting ovsdb-server. There are tradeoffs with anything I've come up with so far. Easier would have been to just allow things like: --remote $prot:$port:$ip;col=val,... and have a generic --db-${DB}-create-remote that takes a protocol. This would have meant almost no changes at all on our part. :) Things are a little tricky with db-defined Connections and configuring via /etc/sysconfig files. I'm currently leaning towards "just create Local_Config DB/set it up as a remote, and make tripleo set connections" (which is similar to what the workaround for this bz does for SSL connections already). I'd also be happy with the "set connection with ovsdb-tool on Local_Config DB creation" method as well. Thoughts?
I'm not completely sure what to do with this BZ/feature. There are some things I don't like about the Local_Config DB: 1) Ideally, you'd have the DB created and populated before running ovsdb-server, but this is a bit of a pain because you have to use ovsdb-tool transact and it's a pretty hairy bit of json to set Connection/{N,S}B_Global.connections. This can be hidden from the user in ovn-ctl, but it's not ideal. 2) It requires a completely new ctl command for users to change values easily, but since multiple ovsdb-server instances will be running on the machine, you have to pass in --db to connect to the right one. You could have symlinked ovsdb-ctl-{ovs,ovnsb,ovnnb} or something like that that then pick the right connection via environment variables or something similar, but it's ugly and different from the "normal" ctl binaries. 3) We fix the problem of having 2 different ways to configure remotes by making a 3rd way to configure remotes 4) Commands like ovn-nbctl set-connection now either a) write a value to the db that is never used because the table it writes to is no longer passed as a remote or b) we pass both Local_Config.Connection and the existing DB.Connection as remotes, and using ovn-*ctl will set connections that are listened by all servers and ovsdb-ctl set-connection would be used to set "local" values. Either way, it's messy. 4 is the thing that bugs me the most. Maybe if we could have some kind of mechanism for replacing where the remotes are written so that any ctl command that tries to write a remote to the db, ovsdb-server could just know to use the Local_Config db for that? Not sure I like it, but I'm having trouble thinking of other solutions. With the workaround we've implemented in tripleo of just listening on all interfaces and restricting traffic via iptables, the pressure for me to drive this to a solution is lessened considerably. But I don't want to be a jerk and just drop the work mostly-done, either. Any ideas, imaximets?