Bug 1948325

Summary: Coredump when ovn-sbctl dump-flows gets logical flow argument
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Alexey Roytman <aroytman>
Component: OVNAssignee: OVN Team <ovnteam>
Status: ASSIGNED --- QA Contact: Jianlin Shi <jishi>
Severity: unspecified Docs Contact:
Priority: medium    
Version: FDP 20.ICC: ctrautma, dcbw, mmichels
Target Milestone: ---Flags: jishi: needinfo? (mmichels)
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexey Roytman 2021-04-11 21:08:21 UTC
Description of problem:
If ovn-sbctl dump-flows gets logicalflow argument with 0x prefix, e.g. 0x8131c8a8, 
it prints correct output, but fails with coredump:
ovn-sbctl --uuid dump-flows sw1 0x8131c8a8                                                                                                      
Datapath: "sw1" (4b1e53d8-9f0f-4768-b4a6-6cbc58a4bfda)  Pipeline: egress
  uuid=0x8131c8a8, table=10(ls_out_port_sec_l2 ), priority=100  , match=(eth.mcast), action=(output;)
free(): invalid pointer
[2]    616553 abort (core dumped)  ovn-sbctl --uuid dump-flows sw1 0x8131c8a8

However, if we provide the same argument without the 0x prefix, the utility exits without core dumps.

ovn-sbctl --uuid dump-flows sw1 8131c8a8                                                                                                       
Datapath: "sw1" (4b1e53d8-9f0f-4768-b4a6-6cbc58a4bfda)  Pipeline: egress
  uuid=0x8131c8a8, table=10(ls_out_port_sec_l2 ), priority=100  , match=(eth.mcast), action=(output;)


Version-Release number of selected component (if applicable):
ovn-sbctl --version                                                                                                                            
   ovn-sbctl 21.03.90
   Open vSwitch Library 2.15.90
   DB Schema 20.17.0


How reproducible:


Steps to Reproduce:
1. ovn-sbctl --uuid dump-flows
2. choose one or more lflows
3. run the same command with the chosen lflows (with and without the 0x prefix)

Actual results:


Expected results:


Additional info:

Comment 2 Alexey Roytman 2021-05-14 21:22:37 UTC
The fix was pushed to master, branch-21.03 and branch-20.12

Comment 3 Jianlin Shi 2021-06-04 06:26:16 UTC
tested with following script:

systemctl start openvswitch                                                
systemctl start ovn-northd                                
ovn-nbctl set-connection ptcp:6641                                                          
ovn-sbctl set-connection ptcp:6642                                    
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:1.1.40.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=1.1.40.25
systemctl restart ovn-controller                                      
                                                                

ovn-nbctl ls-add ls \
    -- lsp-add ls lsp \
    -- lsp-set-addresses lsp "00:00:00:00:00:02 10.0.0.2 2001::2"


ovs-vsctl add-port br-int lsp -- set interface lsp type=internal external_ids:iface-id=lsp options:tx_pcap=lsp.pcap options:rxq_pcap=lsp-rx.pcap
ip netns add lsp
ip link set lsp netns lsp
ip netns exec lsp ip link set lsp address 00:00:00:00:00:02
ip netns exec lsp ip link set lsp up
ip netns exec lsp ip addr add 10.0.0.2/24 dev lsp
ip netns exec lsp ip addr add 2001::2/64 dev lsp


uuid_tmp=$(ovn-sbctl --uuid dump-flows | grep "Datapath.*ingre" -A 1 | grep -o -P  "uuid.{10}" | awk -F = '{print $2}')

ovn-sbctl --uuid dump-flows ls $uuid_tmp

reproduced on ovn2.13-20.12.0-104:

[root@dell-per740-12 bz1948325]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch2.13-2.13.0-96.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-18.el7fdp.noarch
ovn2.13-20.12.0-104.el7fdp.x86_64
ovn2.13-host-20.12.0-104.el7fdp.x86_64
ovn2.13-central-20.12.0-104.el7fdp.x86_64

+ ovn-sbctl --uuid dump-flows ls 0xdd13c16                                                         
Datapath: "ls" (24ac81b5-8115-44b5-b310-5bdf1e1de75f)  Pipeline: ingress                           
  uuid=0xdd13c16d, table=0 (ls_in_port_sec_l2  ), priority=100  , match=(eth.src[40]), action=(drop;)
*** Error in `ovn-sbctl': free(): invalid pointer: 0x000055cad3d2fe52 ***                          
======= Backtrace: =========                                                                         
/lib64/libc.so.6(+0x81329)[0x7fd317c34329]                                                            
ovn-sbctl(+0x28ab0)[0x55cad2facab0]                                                                                                   
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fd317bd5555]                                              
ovn-sbctl(+0x29715)[0x55cad2fad715]                                                                   
======= Memory map: ========                                                                          
55cad2f84000-55cad31b0000 r-xp 00000000 fd:00 504513                     /usr/bin/ovn-sbctl 

the issue still exist on the latest ovn2.13-20.12.0-135:

[root@dell-per740-12 bz1948325]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch2.13-2.13.0-96.el7fdp.x86_64
ovn2.13-20.12.0-135.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-18.el7fdp.noarch
ovn2.13-central-20.12.0-135.el7fdp.x86_64
ovn2.13-host-20.12.0-135.el7fdp.x86_64
[root@dell-per740-12 bz1948325]# ovn-sbctl --version
ovn-sbctl 20.12.0
Open vSwitch Library 2.15.90
DB Schema 20.16.1

+ ovn-sbctl --uuid dump-flows ls 0x45cb5d1                                                         
Datapath: "ls" (2fcc1dc2-c5e2-4f0f-87f2-93688a98a396)  Pipeline: ingress                           
  uuid=0x45cb5d12, table=0 (ls_in_port_sec_l2  ), priority=100  , match=(eth.src[40]), action=(drop;)
*** Error in `ovn-sbctl': free(): invalid pointer: 0x00005625620f0e52 ***                          
======= Backtrace: =========                                                                         
/lib64/libc.so.6(+0x81329)[0x7f9f6a766329]                                                           
ovn-sbctl(+0x28ab0)[0x56256083fab0]                                                                  
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9f6a707555]                                             
ovn-sbctl(+0x29715)[0x562560840715]            
======= Memory map: ========

Comment 4 Alexey Roytman 2021-06-06 05:21:15 UTC
Hi @jishi, 
I rechecked it with the latest version of the branch 20.12 from the git repository. 
It works as expected:

ovn-sbctl --uuid dump-flows ls 0xfed5192                                                                                                                        
Datapath: "ls" (4d855381-d458-4a6f-a972-bb1afa50ed44)  Pipeline: ingress
  uuid=0xfed51922, table=0 (ls_in_port_sec_l2  ), priority=100  , match=(eth.src[40]), action=(drop;)
  
ovn-sbctl -V                                                                                                                                                             
ovn-sbctl 20.12.1
Open vSwitch Library 2.15.90
DB Schema 20.12.0

                               
In your tests, you used the rpm prepared binaries. Unfortunately, I cannot say when a specific git commit will be included in rpm packages.

Comment 5 Jianlin Shi 2021-06-07 00:30:48 UTC
(In reply to Alexey Roytman from comment #4)
> Hi @jishi, 
> I rechecked it with the latest version of the branch 20.12 from the git
> repository. 
> It works as expected:
> 
> ovn-sbctl --uuid dump-flows ls 0xfed5192                                    
> 
> Datapath: "ls" (4d855381-d458-4a6f-a972-bb1afa50ed44)  Pipeline: ingress
>   uuid=0xfed51922, table=0 (ls_in_port_sec_l2  ), priority=100  ,
> match=(eth.src[40]), action=(drop;)
>   
> ovn-sbctl -V                                                                
> 
> ovn-sbctl 20.12.1

version in my test is 20.12.0, that seems to be the only difference.
the rpm is the latest rpm, I'm sure which rpm contains the fixed commit


> Open vSwitch Library 2.15.90
> DB Schema 20.12.0
> 
>                                
> In your tests, you used the rpm prepared binaries. Unfortunately, I cannot
> say when a specific git commit will be included in rpm packages.


@mark could you help to check which version is the commit go in? thanks

Comment 6 Jianlin Shi 2021-07-19 07:01:04 UTC
set to ASSIGNED per comment 5