Bug 1875337
| Summary: | IPv6 load balancer VIP error in ovn-controller | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Alexander Constantinescu <aconstan> | ||||||||
| Component: | ovn2.13 | Assignee: | lorenzo bianconi <lorenzo.bianconi> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Jianlin Shi <jishi> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | high | ||||||||||
| Version: | RHEL 8.0 | CC: | ctrautma, jishi, ralongi | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | ovn2.13-20.06.2-7.el7fdn | Doc Type: | If docs needed, set a value | ||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2020-10-27 09:49:14 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
Assigning to Dumitru directly, since I already talked to him and he's aware of the context. Created attachment 1713582 [details]
OVN SB DB master-1
Created attachment 1713584 [details]
ovn-controller logs from master-1
reproduced on ovn2.13.0-39 with following script:
systemctl start openvswitch
systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:1.1.23.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=1.1.23.25
systemctl restart ovn-controller
ip netns add server0
ip link add veth0_s0 netns server0 type veth peer name veth0_s0_p
ip netns exec server0 ip link set lo up
ip netns exec server0 ip link set veth0_s0 up
ip netns exec server0 ip link set veth0_s0 address 00:00:00:01:01:02
ip netns exec server0 ip addr add 192.168.1.1/24 dev veth0_s0
ip netns exec server0 ip -6 addr add 2001::1/64 dev veth0_s0
ip netns exec server0 ip route add default via 192.168.1.254 dev veth0_s0
ip netns exec server0 ip -6 route add default via 2001::a dev veth0_s0
ovs-vsctl add-port br-int veth0_s0_p
ip link set veth0_s0_p up
ovs-vsctl set interface veth0_s0_p external_ids:iface-id=ls1p1
ovn-nbctl ls-add ls1
ovn-nbctl lsp-add ls1 ls1p1
#ovn-nbctl lsp-set-addresses ls1p1 "00:00:00:01:01:02 2001::1 192.168.1.1"
ovn-nbctl lsp-set-addresses ls1p1 "00:00:00:01:01:02 192.168.1.1 2001::1"
ovn-nbctl lsp-add ls1 ls1p2
ovn-nbctl lsp-set-addresses ls1p2 "00:00:00:01:02:02 192.168.1.2 2001::2"
ovn-nbctl lr-add lr1
ovn-nbctl lrp-add lr1 lr1-ls1 00:00:00:00:00:01 192.168.1.254/24 2001::a/64
ovn-nbctl lsp-add ls1 ls1-lr1
ovn-nbctl lsp-set-addresses ls1-lr1 "00:00:00:00:00:01 192.168.1.254 2001::a"
ovn-nbctl lsp-set-type ls1-lr1 router
ovn-nbctl lsp-set-options ls1-lr1 router-port=lr1-ls1
ovn-nbctl lrp-add lr1 lr1-ls2 00:00:00:00:00:02 192.168.2.254/24 2002::a/64
ovn-nbctl ls-add ls2
ovn-nbctl lsp-add ls2 ls2-lr1
ovn-nbctl lsp-set-addresses ls2-lr1 "00:00:00:00:00:02 192.168.2.254 2002::a"
ovn-nbctl lsp-set-type ls2-lr1 router
ovn-nbctl lsp-set-options ls2-lr1 router-port=lr1-ls2
ovn-nbctl lsp-add ls2 ls2p1
ovn-nbctl lsp-set-addresses ls2p1 "00:00:00:02:01:02 192.168.2.1 2002::1"
ovn-nbctl lsp-add ls1 ls1p3
ovn-nbctl lsp-set-addresses ls1p3 "00:00:00:01:03:02 192.168.1.3 2001::3"
ip netns add server2
ip link add veth0_s2 netns server2 type veth peer name veth0_s2_p
ip netns exec server2 ip link set lo up
ip netns exec server2 ip link set veth0_s2 up
ip netns exec server2 ip link set veth0_s2 address 00:00:00:01:03:02
ip netns exec server2 ip addr add 192.168.1.3/24 dev veth0_s2
ip netns exec server2 ip -6 addr add 2001::3/64 dev veth0_s2
ip netns exec server2 ip route add default via 192.168.1.254 dev veth0_s2
ip netns exec server2 ip -6 route add default via 2001::a dev veth0_s2
ovs-vsctl add-port br-int veth0_s2_p
ip link set veth0_s2_p up
ovs-vsctl set interface veth0_s2_p external_ids:iface-id=ls1p3
ovn-nbctl lb-add lb0 192.168.1.100 192.168.1.1,192.168.1.2
ovn-nbctl lb-add lb0 [2001::100]:50051 [2001::1]:50051,[2001::2]:50051
ovn-nbctl ls-lb-add ls2 lb0
ovn-nbctl set nb_global . options:controller_event=true
ovn-nbctl create load_balancer name="foo" protocol="tcp" vips='"[2001::101]:50052"=""'
ovn-nbctl ls-lb-add ls2 foo
[root@wsfd-advnetlab18 bz1875337]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch
ovn2.13-2.13.0-39.el8fdp.x86_64
openvswitch2.13-2.13.0-61.el8fdp.x86_64
ovn2.13-central-2.13.0-39.el8fdp.x86_64
ovn2.13-host-2.13.0-39.el8fdp.x86_64
+ ovn-nbctl ls-lb-add ls2 lb0
+ ovn-nbctl set nb_global . options:controller_event=true
+ ovn-nbctl create load_balancer name=foo protocol=tcp 'vips="[2001::101]:50052"=""'
240e552e-1f34-478a-9b22-46347f12c6ad
+ ovn-nbctl ls-lb-add ls2 foo
[root@wsfd-advnetlab18 bz1875337]# tail -5 /var/log/ovn/ovn-controller.log
2020-10-14T07:55:55.638Z|00015|binding|INFO|ls1p1: Claiming 00:00:00:01:01:02 192.168.1.1 2001::1
2020-10-14T07:55:55.638Z|00016|binding|INFO|Claiming lport ls1p3 for this chassis.
2020-10-14T07:55:55.638Z|00017|binding|INFO|ls1p3: Claiming 00:00:00:01:03:02 192.168.1.3 2001::3
2020-10-14T07:55:55.653Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connected
2020-10-14T07:55:55.658Z|00018|lflow|WARN|error parsing actions "trigger_event(event = "empty_lb_backends", meter = "", vip = "2001::101:50052", protocol = "tcp", load_balancer = "240e552e-1f34-478a-9b22-46347f12c6ad");": Invalid load balancer VIP '2001::101:50052'
<=== error in ovn-controller.log
Verified on ovn20.09.0-2:
[root@wsfd-advnetlab18 bz1875337]# tail -5 /var/log/ovn/ovn-controller.log
2020-10-14T07:57:18.900Z|00014|binding|INFO|Claiming lport ls1p3 for this chassis.
2020-10-14T07:57:18.900Z|00015|binding|INFO|ls1p3: Claiming 00:00:00:01:03:02 192.168.1.3 2001::3
2020-10-14T07:57:18.901Z|00016|binding|INFO|Claiming lport ls1p1 for this chassis.
2020-10-14T07:57:18.901Z|00017|binding|INFO|ls1p1: Claiming 00:00:00:01:01:02 192.168.1.1 2001::1
2020-10-14T07:57:18.915Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connected
[root@wsfd-advnetlab18 bz1875337]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch
ovn2.13-central-20.09.0-2.el8fdp.x86_64
openvswitch2.13-2.13.0-61.el8fdp.x86_64
ovn2.13-20.09.0-2.el8fdp.x86_64
ovn2.13-host-20.09.0-2.el8fdp.x86_64
also verified on rhel7 version: [root@wsfd-advnetlab16 bz1875337]# tail -5 /var/log/ovn/ovn-controller.log 2020-10-14T07:59:17.982Z|00014|binding|INFO|Claiming lport ls1p1 for this chassis. 2020-10-14T07:59:17.982Z|00015|binding|INFO|ls1p1: Claiming 00:00:00:01:01:02 192.168.1.1 2001::1 2020-10-14T07:59:17.982Z|00016|binding|INFO|Claiming lport ls1p3 for this chassis. 2020-10-14T07:59:17.982Z|00017|binding|INFO|ls1p3: Claiming 00:00:00:01:03:02 192.168.1.3 2001::3 2020-10-14T07:59:17.996Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connected <=== no error [root@wsfd-advnetlab16 bz1875337]# ovn-nbctl list load_balancer _uuid : b53934e1-06cc-4331-a2fa-132dcb177fcf external_ids : {} health_check : [] ip_port_mappings : {} name : foo protocol : tcp selection_fields : [] vips : {"[2001::101]:50052"=""} _uuid : c34a02ab-4fea-4f23-9739-65680a06c889 external_ids : {} health_check : [] ip_port_mappings : {} name : lb0 protocol : tcp selection_fields : [] vips : {"192.168.1.100"="192.168.1.1,192.168.1.2", "[2001::100]:50051"="[2001::1]:50051,[2001::2]:50051"} [root@wsfd-advnetlab16 bz1875337]# rpm -qa | grep -E "openvswitch|ovn" kernel-kernel-networking-openvswitch-ovn-common-1.0-11.noarch python3-openvswitch2.13-2.13.0-51.el7fdp.x86_64 kernel-kernel-networking-openvswitch-ovn-acl-1.0-19.noarch openvswitch2.13-2.13.0-51.el7fdp.x86_64 ovn2.13-20.09.0-2.el7fdp.x86_64 ovn2.13-host-20.09.0-2.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch ovn2.13-central-20.09.0-2.el7fdp.x86_64 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4356 |
Created attachment 1713581 [details] OVN NB DB from master-1 Description of problem: Seen on an IPv6 single-stack Openshift cluster: In ovn-controller we have a lot of logs complaining about: 2020-09-03T04:21:56Z|00058|lflow|WARN|error parsing actions "trigger_event(event = "empty_lb_backends", meter = "", vip = "fd02::3a5a:50051", protocol = "tcp", load_balancer = "52e4aa4a-456b-4ee6-81a2-90b67bcd8b55");": Invalid load balancer VIP 'fd02::3a5a:50051' This load-balancer VIP is however properly defined in the OVN NB DB, see: $ ovn-nbctl -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt --db 'ssl:[2620:52:0:1386::91]:9641,ssl:[2620:52:0:1386::92]:9641,ssl:[2620:52:0:1386::93]:9641' list load_balancer | grep fd02::3a5a vips : {.... "[fd02::3a5a]:50051"="",...} Version-Release number of selected component (if applicable): rpm -qa | grep ovn ovn2.13-2.13.0-39.el7fdp.x86_64 ovn2.13-central-2.13.0-39.el7fdp.x86_64 ovn2.13-host-2.13.0-39.el7fdp.x86_64 ovn2.13-vtep-2.13.0-39.el7fdp.x86_64 How reproducible: - Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: