Bug 1948325 - Coredump when ovn-sbctl dump-flows gets logical flow argument [NEEDINFO]
Summary: Coredump when ovn-sbctl dump-flows gets logical flow argument
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: OVN
Version: FDP 20.I
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: ---
: ---
Assignee: OVN Team
QA Contact: Jianlin Shi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-11 21:08 UTC by Alexey Roytman
Modified: 2023-07-13 07:25 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:
jishi: needinfo? (mmichels)


Attachments (Terms of Use)

Description Alexey Roytman 2021-04-11 21:08:21 UTC
Description of problem:
If ovn-sbctl dump-flows gets logicalflow argument with 0x prefix, e.g. 0x8131c8a8, 
it prints correct output, but fails with coredump:
ovn-sbctl --uuid dump-flows sw1 0x8131c8a8                                                                                                      
Datapath: "sw1" (4b1e53d8-9f0f-4768-b4a6-6cbc58a4bfda)  Pipeline: egress
  uuid=0x8131c8a8, table=10(ls_out_port_sec_l2 ), priority=100  , match=(eth.mcast), action=(output;)
free(): invalid pointer
[2]    616553 abort (core dumped)  ovn-sbctl --uuid dump-flows sw1 0x8131c8a8

However, if we provide the same argument without the 0x prefix, the utility exits without core dumps.

ovn-sbctl --uuid dump-flows sw1 8131c8a8                                                                                                       
Datapath: "sw1" (4b1e53d8-9f0f-4768-b4a6-6cbc58a4bfda)  Pipeline: egress
  uuid=0x8131c8a8, table=10(ls_out_port_sec_l2 ), priority=100  , match=(eth.mcast), action=(output;)


Version-Release number of selected component (if applicable):
ovn-sbctl --version                                                                                                                            
   ovn-sbctl 21.03.90
   Open vSwitch Library 2.15.90
   DB Schema 20.17.0


How reproducible:


Steps to Reproduce:
1. ovn-sbctl --uuid dump-flows
2. choose one or more lflows
3. run the same command with the chosen lflows (with and without the 0x prefix)

Actual results:


Expected results:


Additional info:

Comment 2 Alexey Roytman 2021-05-14 21:22:37 UTC
The fix was pushed to master, branch-21.03 and branch-20.12

Comment 3 Jianlin Shi 2021-06-04 06:26:16 UTC
tested with following script:

systemctl start openvswitch                                                
systemctl start ovn-northd                                
ovn-nbctl set-connection ptcp:6641                                                          
ovn-sbctl set-connection ptcp:6642                                    
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:1.1.40.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=1.1.40.25
systemctl restart ovn-controller                                      
                                                                

ovn-nbctl ls-add ls \
    -- lsp-add ls lsp \
    -- lsp-set-addresses lsp "00:00:00:00:00:02 10.0.0.2 2001::2"


ovs-vsctl add-port br-int lsp -- set interface lsp type=internal external_ids:iface-id=lsp options:tx_pcap=lsp.pcap options:rxq_pcap=lsp-rx.pcap
ip netns add lsp
ip link set lsp netns lsp
ip netns exec lsp ip link set lsp address 00:00:00:00:00:02
ip netns exec lsp ip link set lsp up
ip netns exec lsp ip addr add 10.0.0.2/24 dev lsp
ip netns exec lsp ip addr add 2001::2/64 dev lsp


uuid_tmp=$(ovn-sbctl --uuid dump-flows | grep "Datapath.*ingre" -A 1 | grep -o -P  "uuid.{10}" | awk -F = '{print $2}')

ovn-sbctl --uuid dump-flows ls $uuid_tmp

reproduced on ovn2.13-20.12.0-104:

[root@dell-per740-12 bz1948325]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch2.13-2.13.0-96.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-18.el7fdp.noarch
ovn2.13-20.12.0-104.el7fdp.x86_64
ovn2.13-host-20.12.0-104.el7fdp.x86_64
ovn2.13-central-20.12.0-104.el7fdp.x86_64

+ ovn-sbctl --uuid dump-flows ls 0xdd13c16                                                         
Datapath: "ls" (24ac81b5-8115-44b5-b310-5bdf1e1de75f)  Pipeline: ingress                           
  uuid=0xdd13c16d, table=0 (ls_in_port_sec_l2  ), priority=100  , match=(eth.src[40]), action=(drop;)
*** Error in `ovn-sbctl': free(): invalid pointer: 0x000055cad3d2fe52 ***                          
======= Backtrace: =========                                                                         
/lib64/libc.so.6(+0x81329)[0x7fd317c34329]                                                            
ovn-sbctl(+0x28ab0)[0x55cad2facab0]                                                                                                   
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fd317bd5555]                                              
ovn-sbctl(+0x29715)[0x55cad2fad715]                                                                   
======= Memory map: ========                                                                          
55cad2f84000-55cad31b0000 r-xp 00000000 fd:00 504513                     /usr/bin/ovn-sbctl 

the issue still exist on the latest ovn2.13-20.12.0-135:

[root@dell-per740-12 bz1948325]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch2.13-2.13.0-96.el7fdp.x86_64
ovn2.13-20.12.0-135.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-18.el7fdp.noarch
ovn2.13-central-20.12.0-135.el7fdp.x86_64
ovn2.13-host-20.12.0-135.el7fdp.x86_64
[root@dell-per740-12 bz1948325]# ovn-sbctl --version
ovn-sbctl 20.12.0
Open vSwitch Library 2.15.90
DB Schema 20.16.1

+ ovn-sbctl --uuid dump-flows ls 0x45cb5d1                                                         
Datapath: "ls" (2fcc1dc2-c5e2-4f0f-87f2-93688a98a396)  Pipeline: ingress                           
  uuid=0x45cb5d12, table=0 (ls_in_port_sec_l2  ), priority=100  , match=(eth.src[40]), action=(drop;)
*** Error in `ovn-sbctl': free(): invalid pointer: 0x00005625620f0e52 ***                          
======= Backtrace: =========                                                                         
/lib64/libc.so.6(+0x81329)[0x7f9f6a766329]                                                           
ovn-sbctl(+0x28ab0)[0x56256083fab0]                                                                  
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9f6a707555]                                             
ovn-sbctl(+0x29715)[0x562560840715]            
======= Memory map: ========

Comment 4 Alexey Roytman 2021-06-06 05:21:15 UTC
Hi @jishi, 
I rechecked it with the latest version of the branch 20.12 from the git repository. 
It works as expected:

ovn-sbctl --uuid dump-flows ls 0xfed5192                                                                                                                        
Datapath: "ls" (4d855381-d458-4a6f-a972-bb1afa50ed44)  Pipeline: ingress
  uuid=0xfed51922, table=0 (ls_in_port_sec_l2  ), priority=100  , match=(eth.src[40]), action=(drop;)
  
ovn-sbctl -V                                                                                                                                                             
ovn-sbctl 20.12.1
Open vSwitch Library 2.15.90
DB Schema 20.12.0

                               
In your tests, you used the rpm prepared binaries. Unfortunately, I cannot say when a specific git commit will be included in rpm packages.

Comment 5 Jianlin Shi 2021-06-07 00:30:48 UTC
(In reply to Alexey Roytman from comment #4)
> Hi @jishi, 
> I rechecked it with the latest version of the branch 20.12 from the git
> repository. 
> It works as expected:
> 
> ovn-sbctl --uuid dump-flows ls 0xfed5192                                    
> 
> Datapath: "ls" (4d855381-d458-4a6f-a972-bb1afa50ed44)  Pipeline: ingress
>   uuid=0xfed51922, table=0 (ls_in_port_sec_l2  ), priority=100  ,
> match=(eth.src[40]), action=(drop;)
>   
> ovn-sbctl -V                                                                
> 
> ovn-sbctl 20.12.1

version in my test is 20.12.0, that seems to be the only difference.
the rpm is the latest rpm, I'm sure which rpm contains the fixed commit


> Open vSwitch Library 2.15.90
> DB Schema 20.12.0
> 
>                                
> In your tests, you used the rpm prepared binaries. Unfortunately, I cannot
> say when a specific git commit will be included in rpm packages.


@mark could you help to check which version is the commit go in? thanks

Comment 6 Jianlin Shi 2021-07-19 07:01:04 UTC
set to ASSIGNED per comment 5


Note You need to log in before you can comment on or make changes to this bug.