Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1485800

Summary: [ovs-dpdk conntrack]ftp can't open data connection in the DNAT topo
Product: Red Hat Enterprise Linux Fast Datapath Reporter: haidong li <haili>
Component: openvswitchAssignee: David Marchand <dmarchan>
openvswitch sub component: other QA Contact: Jiying Qiu <jiqiu>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: atragler, ctrautma, dmarchan, fleitner, haili, jiqiu, nsaddler, ovs-qe, pvauter, qding, rkhan, tredaelli
Version: FDP 19.B   
Target Milestone: ---   
Target Release: FDP 19.B   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1680516 (view as bug list) Environment:
Last Closed: 2019-04-29 09:26:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1680516    

Description haidong li 2017-08-28 06:35:18 UTC
Description of problem:
ftp can't open data connection in the DNAT topo.It seems that the data packet is not changed by dnat.

Version-Release number of selected component (if applicable):
    ovs_version: "2.8.0"

[root@dell-per730-18 conntrack_dpdk]# uname -a
Linux dell-per730-18.rhts.eng.pek2.redhat.com 3.10.0-693.el7.gcov.x86_64 #1 SMP Tue Jul 11 01:39:56 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-per730-18 conntrack_dpdk]#

How reproducible:
everytime

topo:

the ip of vm is 10.167.45.1, ip of remote host's nic is 172.16.45.2. Remote try to use ftp to connect 172.16.45.2, which should be change to 10.167.45.1 by ovs.

10.167.45.1           172.16.45.2 
+--------+            +------+
| vm--ovs+------------+remote|       
+--------+            +------+    


Steps to Reproduce:
1.create ovs in host,add dpdk0 and vhost to ovs. create guest with vhost0
2.configure flows on ovs, using dnat
3.ftp from remote to local.

Actual results:
can't openv data connect

Expected results:
ftp works well

log:
[root@dell-per730-18 conntrack_dpdk]# dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
0000:06:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=vfio-pci unused=ixgbe

Network devices using kernel driver
===================================
0000:01:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em1 drv=tg3 unused=vfio-pci 
0000:01:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em2 drv=tg3 unused=vfio-pci 
0000:02:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em3 drv=tg3 unused=vfio-pci 
0000:02:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em4 drv=tg3 unused=vfio-pci 
0000:04:00.0 'MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]' if=p6p2,p6p1 drv=mlx4_core unused=vfio-pci 
0000:06:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=p7p2 drv=ixgbe unused=vfio-pci 

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>
[root@dell-per730-18 conntrack_dpdk]#
[root@dell-per730-18 conntrack_dpdk]# virsh console g1
Connected to domain g1
Escape character is ^]

[root@localhost ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:00:00:01:03:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::200:ff:fe01:301/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:00:00:01:04:01 brd ff:ff:ff:ff:ff:ff
    inet 10.167.45.1/24 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 2001:db8:ffff:45::1/64 scope global 
       valid_lft forever preferred_lft forever
[root@localhost ~]# 
[root@dell-per730-18 conntrack_dpdk]# ovs-ofctl show
ovs-ofctl: 'show' command requires at least 1 arguments
[root@dell-per730-18 conntrack_dpdk]# ovs-vsctl show
0b1eb18c-370c-417f-a954-6dbddfb8c81c
    Bridge "ovsbr0"
        Port "vhost1"
            Interface "vhost1"
                type: dpdkvhostuser
                options: {n_rxq="1"}
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "dpdk0"
            Interface "dpdk0"
                type: dpdk
                options: {dpdk-devargs="0000:06:00.0", n_rxq="1"}
        Port "vhost0"
            Interface "vhost0"
                type: dpdkvhostuser
                options: {n_rxq="1"}
        Port "vhost2"
            Interface "vhost2"
                type: dpdkvhostuser
                options: {n_rxq="1"}
    ovs_version: "2.8.0"
[root@dell-per730-18 conntrack_dpdk
[root@dell-per730-18 conntrack_dpdk]# ovs-ofctl show ovsbr0
OFPT_FEATURES_REPLY (xid=0x2): dpid:000090e2ba29bf14
n_tables:254, n_buffers:0
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 10(dpdk0): addr:90:e2:ba:29:bf:14
     config:     0
     state:      0
     current:    10GB-FD
     speed: 10000 Mbps now, 0 Mbps max
 11(vhost0): addr:00:00:00:00:00:00
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 21(vhost1): addr:00:00:00:00:00:00
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 31(vhost2): addr:00:00:00:00:00:00
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(ovsbr0): addr:90:e2:ba:29:bf:14
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
[root@dell-per730-18 conntrack_dpdk]# 

[root@dell-per730-18 conntrack_dpdk]# ovs-ofctl dump-flows ovsbr0
 cookie=0x0, duration=702.611s, table=0, n_packets=33, n_bytes=3633, priority=100,ct_state=-trk,ip actions=ct(table=1)
 cookie=0x0, duration=702.564s, table=0, n_packets=48, n_bytes=2862, priority=100,arp,arp_op=1 actions=move:NXM_OF_ARP_TPA[]->NXM_NX_REG2[],resubmit(,8),resubmit(,10)
 cookie=0x0, duration=702.558s, table=0, n_packets=1, n_bytes=60, priority=10,arp actions=NORMAL
 cookie=0x0, duration=702.553s, table=0, n_packets=1087, n_bytes=141633, priority=1 actions=drop
 cookie=0x0, duration=702.600s, table=1, n_packets=1, n_bytes=74, ct_state=+new+trk,tcp,nw_dst=172.16.45.254,tp_dst=21 actions=ct(commit,nat(dst=10.167.45.1),alg=ftp),output:vhost0
 cookie=0x0, duration=702.590s, table=1, n_packets=1, n_bytes=74, ct_state=+new+rel+trk,ip,nw_dst=172.16.45.254 actions=ct(commit,nat(dst=10.167.45.1)),output:vhost0
 cookie=0x0, duration=702.582s, table=1, n_packets=14, n_bytes=973, ct_state=+est+trk,ip,nw_dst=172.16.45.254 actions=ct(nat),output:vhost0
 cookie=0x0, duration=702.574s, table=1, n_packets=10, n_bytes=830, ct_state=+est+trk,ip,nw_src=10.167.45.1 actions=ct(nat),output:dpdk0
 cookie=0x0, duration=624.459s, table=8, n_packets=1, n_bytes=60, reg2=0xac102dfe actions=load:0x10401->OXM_OF_PKT_REG0[]
 cookie=0x0, duration=607.910s, table=8, n_packets=46, n_bytes=2742, priority=0 actions=load:0->OXM_OF_PKT_REG0[]
 cookie=0x0, duration=607.903s, table=10, n_packets=46, n_bytes=2742, priority=100,arp,reg0=0,reg1=0 actions=NORMAL
 cookie=0x0, duration=607.877s, table=10, n_packets=1, n_bytes=60, priority=10,arp,arp_op=1 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:OXM_OF_PKT_REG0[0..47]->NXM_NX_ARP_SHA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],move:NXM_NX_REG2[]->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],move:OXM_OF_PKT_REG0[0..47]->NXM_OF_ETH_SRC[],move:NXM_OF_IN_PORT[]->NXM_NX_REG3[0..15],load:0->NXM_OF_IN_PORT[],output:NXM_NX_REG3[0..15]
 cookie=0x0, duration=607.871s, table=10, n_packets=0, n_bytes=0, priority=0 actions=drop

[root@dell-per730-18 conntrack_dpdk]# ovs-appctl dpctl/dump-conntrack
tcp,orig=(src=172.16.45.2,dst=172.16.45.254,sport=39082,dport=21),reply=(src=10.167.45.1,dst=172.16.45.2,sport=21,dport=39082),protoinfo=(state=ESTABLISHED),helper=ftp
tcp,orig=(src=172.16.45.2,dst=172.16.45.254,sport=45344,dport=64004),reply=(src=172.16.45.254,dst=172.16.45.2,sport=64004,dport=45344),protoinfo=(state=SYN_SENT)



remote:
[root@dell-per730-20 conntrack_dpdk]# ip add show p7p1
8: p7p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:07:43:11:27:30 brd ff:ff:ff:ff:ff:ff
    inet 172.16.45.2/24 scope global p7p1
       valid_lft forever preferred_lft forever
    inet6 2001::207:43ff:fe11:2730/64 scope global mngtmpaddr dynamic 
       valid_lft 86400sec preferred_lft 14400sec
    inet6 2001:db8:0:45::2/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::207:43ff:fe11:2730/64 scope link 
       valid_lft forever preferred_lft forever
[root@dell-per730-20 conntrack_dpdk]# 

[root@dell-per730-20 conntrack_dpdk]# ftp 172.16.45.254
Connected to 172.16.45.254 (172.16.45.254).
220 (vsFTPd 3.0.2)
Name (172.16.45.254:root): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
227 Entering Passive Mode (172,16,45,254,250,4)..
^C
receive aborted
waiting for remote to finish abort
ftp> 
[root@dell-per730-18 conntrack_dpdk]#

Additional info:

I met another issue that if the remote ftp connect timeout for several times,local's ovs flow disappeared and the interface of vm is down.This sometimes happen.

Comment 2 haidong li 2017-09-12 23:04:23 UTC
*** Bug 1485813 has been marked as a duplicate of this bug. ***

Comment 3 Flavio Leitner 2018-02-16 12:44:45 UTC
Can you please try again with 2.9?

Comment 4 Flavio Leitner 2018-02-16 12:56:58 UTC
This could be a dup of #1499645

Comment 5 haidong li 2018-02-22 05:52:51 UTC
(In reply to Flavio Leitner from comment #3)
> Can you please try again with 2.9?

Contrack feature is tested by Jiying now.
Hi Jiying,can you please help try and answer the question?

Comment 6 Jiying Qiu 2018-02-24 06:47:30 UTC
The issue is still exist in ovs2.9.

There is a job can refer to.
https://beaker.engineering.redhat.com/jobs/2325919

Comment 8 David Marchand 2018-12-18 12:51:48 UTC
*** Bug 1499645 has been marked as a duplicate of this bug. ***

Comment 11 Jiying Qiu 2019-03-28 07:04:43 UTC
verified with openvswitch2.11-2.11.0-3.el7fdp.x86_64.rpm and openvswitch-2.9.0-101.el7fdp.x86_64.rpm in RHEL-7.6 , openvswitch2.11-2.11.0-4.el8fdp.x86_64.rpm in RHEL-8.0.

Comment 13 errata-xmlrpc 2019-04-29 09:26:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0898