Bug 1904489 - systen-ovn.at: Load balancer health checks always fails
Summary: systen-ovn.at: Load balancer health checks always fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: OVN
Version: FDP 20.I
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: lorenzo bianconi
QA Contact: ying xu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-04 15:45 UTC by lorenzo bianconi
Modified: 2021-05-12 06:48 UTC (History)
1 user (show)

Fixed In Version: ovn2.13-20.09.0-21.el8fdn
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-03 21:55:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:0407 0 None None None 2021-02-03 21:55:27 UTC

Description lorenzo bianconi 2020-12-04 15:45:35 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 ying xu 2021-01-05 12:07:13 UTC
script:
server:
# r1
                ovn-nbctl lr-add r1
                ovn-nbctl lrp-add r1 r1_s2 00:de:ad:ff:01:02 172.16.102.1/24 2001:db8:102::1/64
                ovn-nbctl lrp-add r1 r1_s3 00:de:ad:ff:01:03 172.16.103.1/24 2001:db8:103::1/64

                # s2
                ovn-nbctl ls-add s2
                ovn-nbctl set Logical-switch s2 other_config:subnet=172.16.102.0/24
                ovn-nbctl set Logical-switch s2 other_config:ipv6_prefix=2001:db8:102::0

                # s2 - r1
                ovn-nbctl lsp-add s2 s2_r1
                ovn-nbctl lsp-set-type s2_r1 router
                ovn-nbctl lsp-set-addresses s2_r1 "00:de:ad:ff:01:02 172.16.102.1 2001:db8:102::1"
                ovn-nbctl lsp-set-options s2_r1 router-port=r1_s2

                # s2 - hv1_vm00_vnet1
                ovn-nbctl lsp-add s2 vm10
                ovn-nbctl lsp-set-addresses vm10 "00:de:ad:01:00:01 172.16.102.11 2001:db8:102::11"

                # s2 - hv1_vm01_vnet1
                ovn-nbctl lsp-add s2 vm11
                ovn-nbctl lsp-set-addresses vm11 "00:de:ad:01:01:01 172.16.102.12 2001:db8:102::12"
                # s3
                ovn-nbctl ls-add s3
                ovn-nbctl set Logical-switch s3 other_config:subnet=172.16.103.0/24
                ovn-nbctl set Logical-switch s3 other_config:ipv6_prefix=2001:db8:103::0
                # s3 - r1
                ovn-nbctl lsp-add s3 s3_r1
                ovn-nbctl lsp-set-type s3_r1 router
                ovn-nbctl lsp-set-addresses s3_r1 "00:de:ad:ff:01:03 172.16.103.1 2001:db8:103::1"
                ovn-nbctl lsp-set-options s3_r1 router-port=r1_s3

                # s3 - hv0_vm00_vnet1
                ovn-nbctl lsp-add s3 vm00
                ovn-nbctl lsp-set-addresses vm00 "00:de:ad:00:00:01 172.16.103.11 2001:db8:103::11"

                # s3 - hv0_vm01_vnet1
                ovn-nbctl lsp-add s3 vm01
                ovn-nbctl lsp-set-addresses vm01 "00:de:ad:00:01:01 172.16.103.12 2001:db8:103::12"

                ip netns add vm10
                ovs-vsctl add-port br-int vm10 -- set interface vm10 type=internal
                ip link set vm10 netns vm10
                ip netns exec vm10 ip link set vm10 address 00:de:ad:01:00:01
                ip netns exec vm10 ip addr add 172.16.102.11/24 dev vm10
                ip netns exec vm10 ip link set vm10 up
                ip netns exec vm10 ip link set lo up
                ip netns exec vm10 ip route add default via 172.16.102.1
                ovs-vsctl set Interface vm10 external_ids:iface-id=vm10
                ip netns add vm11
                ovs-vsctl add-port br-int vm11 -- set interface vm11 type=internal
                ip link set vm11 netns vm11
                ip netns exec vm11 ip link set vm11 address 00:de:ad:01:01:01
                ip netns exec vm11 ip addr add 172.16.102.12/24 dev vm11
                ip netns exec vm11 ip link set vm11 up
                ip netns exec vm11 ip link set lo up
                ip netns exec vm11 ip route add default via 172.16.102.1
                ovs-vsctl set Interface vm11 external_ids:iface-id=vm11


                ovn-nbctl set  Logical_Router r1 options:chassis="hv1"
                #add load balance on logical switch
                rlRun "ovn-nbctl  lb-add lb1 172.16.103.10:8000 172.16.103.12:80,172.16.102.12:80"
                rlRun "ovn-nbctl  lb-add lb1 172.16.103.100:8000 172.16.103.12:80,172.16.102.12:80"
                rlRun "ovn-nbctl  lb-add lb1 172.16.103.200:8000 172.16.103.12:80,172.16.102.12:80"
                rlRun "ovn-nbctl  lb-add lb1 172.16.103.110:8000 172.16.103.12:80,172.16.102.12:80"
                uuid=`ovn-nbctl --bare --columns _uuid find load_balancer name=lb1`
                rlRun "ovn-nbctl --wait=sb set load_balancer $uuid ip_port_mappings:172.16.103.12=vm01:172.16.103.2"
                rlRun "ovn-nbctl --wait=sb set load_balancer $uuid ip_port_mappings:172.16.102.12=vm11:172.16.102.2"

#          #create load balance check
                uuid3=`ovn-nbctl --id=@hc create Load_Balancer_Health_Check vip="172.16.103.10\:8000" -- add Load_Balancer $uuid health_check @hc`
                uuid5=`ovn-nbctl --id=@hc create Load_Balancer_Health_Check vip="172.16.103.100\:8000" -- add Load_Balancer $uuid health_check @hc`
                uuid6=`ovn-nbctl --id=@hc create Load_Balancer_Health_Check vip="172.16.103.200\:8000" -- add Load_Balancer $uuid health_check @hc`
                uuid6=`ovn-nbctl --id=@hc create Load_Balancer_Health_Check vip="172.16.103.110\:8000" -- add Load_Balancer $uuid health_check @hc`
                ovn-nbctl set Load_Balancer_Health_Check $uuid3 options:interval=5 options:timeout=20 options:success_count=3 options:failure_count=3
                ovn-nbctl set logical_switch s2 load_balancer=$uuid
                ovn-nbctl set logical_router r1 load_balancer=$uuid
                ovn-nbctl set logical_switch s3 load_balancer=$uuid
client:
                ip netns add vm00
                ovs-vsctl add-port br-int vm00 -- set interface vm00 type=internal
                ip link set vm00 netns vm00
                ip netns exec vm00 ip link set vm00 address 00:de:ad:00:00:01
                ip netns exec vm00 ip addr add 172.16.103.11/24 dev vm00
                ip netns exec vm00 ip link set vm00 up
                ip netns exec vm00 ip link set lo up
                ip netns exec vm00 ip route add default via 172.16.103.1
                ovs-vsctl set Interface vm00 external_ids:iface-id=vm00

                ip netns add vm01
                ovs-vsctl add-port br-int vm01 -- set interface vm01 type=internal
                ip link set vm01 netns vm01
                ip netns exec vm01 ip link set vm01 address 00:de:ad:00:01:01
                ip netns exec vm01 ip addr add 172.16.103.12/24 dev vm01
                ip netns exec vm01 ip link set vm01 up
                ip netns exec vm01 ip link set lo up
                ip netns exec vm01 ip route add default via 172.16.103.1
                ovs-vsctl set Interface vm01 external_ids:iface-id=vm01


I reproduced on version as below
# rpm -qa|grep ovn
ovn2.13-central-20.09.0-20.el8fdn.x86_64
ovn2.13-20.09.0-20.el8fdn.x86_64
ovn2.13-host-20.09.0-20.el8fdn.x86_64


after add 4 vip for the same lb, check the log there are crash can be found.
cat /var/log/ovn/ovn-northd.log|grep ERR'
2021-01-05T10:08:55.506Z|00003|daemon_unix(monitor)|ERR|1 crashes: pid 3054193 died, killed (Segmentation fault), restarting
2021-01-05T10:09:05.531Z|00006|daemon_unix(monitor)|ERR|2 crashes: pid 3055555 died, killed (Segmentation fault), restarting
2021-01-05T10:09:15.558Z|00009|daemon_unix(monitor)|ERR|3 crashes: pid 3055588 died, killed (Segmentation fault), restarting
2021-01-05T10:09:25.578Z|00012|daemon_unix(monitor)|ERR|4 crashes: pid 3055608 died, killed (Segmentation fault), restarting
2021-01-05T10:09:35.597Z|00015|daemon_unix(monitor)|ERR|5 crashes: pid 3055617 died, killed (Segmentation fault), restarting
2021-01-05T10:09:45.616Z|00018|daemon_unix(monitor)|ERR|6 crashes: pid 3055620 died, killed (Segmentation fault), restarting
2021-01-05T10:09:55.635Z|00021|daemon_unix(monitor)|ERR|7 crashes: pid 3055621 died, killed (Segmentation fault), restarting
2021-01-05T10:10:05.654Z|00024|daemon_unix(monitor)|ERR|8 crashes: pid 3055622 died, killed (Segmentation fault), restarting
2021-01-05T10:10:15.675Z|00027|daemon_unix(monitor)|ERR|9 crashes: pid 3055624 died, killed (Segmentation fault), restarting
2021-01-05T10:10:25.697Z|00030|daemon_unix(monitor)|ERR|10 crashes: pid 3055625 died, killed (Segmentation fault), restarting
2021-01-05T10:10:35.715Z|00033|daemon_unix(monitor)|ERR|11 crashes: pid 3055634 died, killed (Segmentation fault), restarting
2021-01-05T10:10:45.733Z|00036|daemon_unix(monitor)|ERR|12 crashes: pid 3055635 died, killed (Segmentation fault), restarting
2021-01-05T10:10:55.752Z|00039|daemon_unix(monitor)|ERR|13 crashes: pid 3055636 died, killed (Segmentation fault), restarting
2021-01-05T10:11:05.773Z|00042|daemon_unix(monitor)|ERR|14 crashes: pid 3055637 died, killed (Segmentation fault), restarting
2021-01-05T10:11:15.790Z|00045|daemon_unix(monitor)|ERR|15 crashes: pid 3055639 died, killed (Segmentation fault), restarting
2021-01-05T10:11:25.809Z|00048|daemon_unix(monitor)|ERR|16 crashes: pid 3055640 died, killed (Segmentation fault), restarting
2021-01-05T10:11:35.828Z|00051|daemon_unix(monitor)|ERR|17 crashes: pid 3055649 died, killed (Segmentation fault), restarting
2021-01-05T10:11:45.847Z|00054|daemon_unix(monitor)|ERR|18 crashes: pid 3055650 died, killed (Segmentation fault), restarting
2021-01-05T10:11:55.865Z|00057|daemon_unix(monitor)|ERR|19 crashes: pid 3055651 died, killed (Segmentation fault), restarting
2021-01-05T10:12:05.884Z|00060|daemon_unix(monitor)|ERR|20 crashes: pid 3055652 died, killed (Segmentation fault), restarting
2021-01-05T10:12:15.903Z|00063|daemon_unix(monitor)|ERR|21 crashes: pid 3055653 died, killed (Segmentation fault), restarting


verified on the version:
# rpm -qa|grep ovn
ovn2.13-central-20.12.0-1.el8fdp.x86_64
ovn2.13-20.12.0-1.el8fdp.x86_64
ovn2.13-host-20.12.0-1.el8fdp.x86_64

cat /var/log/ovn/ovn-northd.log|grep ERR          ---------------no EER found

Comment 5 errata-xmlrpc 2021-02-03 21:55:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0407


Note You need to log in before you can comment on or make changes to this bug.