Bug 1335560
| Summary: | [RFE] Add tcpdump capabilities for low traffic rate | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Franck Baudin <fbaudin> | ||||
| Component: | openvswitch | Assignee: | Aaron Conole <aconole> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Jean-Tsung Hsiao <jhsiao> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.2 | CC: | aloughla, atragler, brault, fleitner, jhsiao, kzhang, mleitner, qding, rcain, rkhan, wchadwic | ||||
| Target Milestone: | rc | Keywords: | FutureFeature | ||||
| Target Release: | 7.3 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openvswitch-2.5.0-12.git20160727.el7fdb | Doc Type: | Enhancement | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-10-31 18:22:27 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1383822, 1384654 | ||||||
| Attachments: |
|
||||||
Change to right component. It's not working. [root@netqe5 ~]# rpm -qa | grep openvswitch openvswitch-2.5.0-12.git20160727.el7fdb.x86_64 [root@netqe5 ~]# rpm -ql openvswitch-2.5.0-12.git20160727.el7fdb.x86_64 | grep tcpdump [root@netqe5 ~]# ovs-tcpdump -bash: ovs-tcpdump: command not found Back to ASSIGNED. There no openvswitch-test pkg along with openvswitch-2.5.0-12.git20160727.el7fdb.x86_64. Aaron just pointed out the openvswitch-test pkg is under noarch. Back to MODIFIED. Created attachment 1211524 [details]
ovs-vswitch log file
These are the packages installed: [root@netqe10 ~]# rpm -qa | grep openvswitch openvswitch-2.5.0-14.git20160727.el7fdp.x86_64 python-openvswitch-2.5.0-14.git20160727.el7fdb.noarch openvswitch-test-2.5.0-14.git20160727.el7fdb.noarch [root@netqe10 ~]# rpm -qa | grep netiface python-netifaces-0.10.4-3.el7.x86_64 [root@netqe10 ~]# rpm -qa | grep dpdk dpdk-16.07-1.el7fdb.x86_64 dpdk-tools-16.07-1.el7fdb.x86_64 Installed openvswitch-2.5.0-12.git20160727.el7fdb with its associated python-openvswitch and openvswitch-test packages. The issue mentioned in Comment #14 did not exist. Set the status baack to ON_QA. Will try openvswitch-2.5.0-14 again. Installed again openvswitch-2.5.0-14.git20160727.el7fdp.x86_64 with its associated python-openvswitch and openvswitch-test packages. But, the same issue encountered in Comment #14 still exists. So, is this expected ? Oct 17 16:59:36 netqe10.knqe.lab.eng.bos.redhat.com ovs-ctl[16991]: EAL: probe driver: 8086:154d rte_ixgbe_pmd Oct 17 16:59:36 netqe10.knqe.lab.eng.bos.redhat.com ovs-ctl[16991]: EAL: Error - exiting with code: 1 Looks like a HW issue. Are you sure that no other dpdk app is/was using the device? Please, reboot, make sure the interface is bound to vfio-pci correctly and start OVS. (In reply to Flavio Leitner from comment #18) > Oct 17 16:59:36 netqe10.knqe.lab.eng.bos.redhat.com ovs-ctl[16991]: > EAL: probe driver: 8086:154d rte_ixgbe_pmd > Oct 17 16:59:36 netqe10.knqe.lab.eng.bos.redhat.com ovs-ctl[16991]: EAL: > Error - exiting with code: 1 > > Looks like a HW issue. Are you sure that no other dpdk app is/was using the > device? > > Please, reboot, make sure the interface is bound to vfio-pci correctly and > start OVS. On Monday I rebooted at least one time, but the issue remained. Per your suggestion I'll try one more time. (In reply to Jean-Tsung Hsiao from comment #16) > Installed openvswitch-2.5.0-12.git20160727.el7fdb with its associated > python-openvswitch and openvswitch-test packages. The issue mentioned in > Comment #14 did not exist. > > Set the status baack to ON_QA. > > Will try openvswitch-2.5.0-14 again. ovs-tcpdump worked properly in this environment. Below is a piece of its output under a very light traffic --- 20 64 bytes pkts per second: 22:55:42.068289 52:54:00:c6:10:20 (oui Unknown) > 52:54:00:c6:10:10 (oui Unknown), ethertype IPv4 (0x0800), length 60: reserved > reserved: ip-proto-255 26 22:55:42.135489 52:54:00:c6:10:20 (oui Unknown) > 52:54:00:c6:10:10 (oui Unknown), ethertype IPv4 (0x0800), length 60: reserved > reserved: ip-proto-255 26 22:55:42.202689 52:54:00:c6:10:20 (oui Unknown) > 52:54:00:c6:10:10 (oui Unknown), ethertype IPv4 (0x0800), length 60: reserved > reserved: ip-proto-255 26 22:55:42.269890 52:54:00:c6:10:20 (oui Unknown) > 52:54:00:c6:10:10 (oui Unknown), ethertype IPv4 (0x0800), length 60: reserved > reserved: ip-proto-255 26 22:55:42.337090 52:54:00:c6:10:20 (oui Unknown) > 52:54:00:c6:10:10 (oui Unknown), ethertype IPv4 (0x0800), length 60: reserved > reserved: ip-proto-255 26 22:55:42.404290 52:54:00:c6:10:20 (oui Unknown) > 52:54:00:c6:10:10 (oui Unknown), ethertype IPv4 (0x0800), length 60: reserved > reserved: ip-proto-255 26 22:55:42.471491 52:54:00:c6:10:20 (oui Unknown) > 52:54:00:c6:10:10 (oui Unknown), ethertype IPv4 (0x0800), length 60: reserved > reserved: ip-proto-255 26 22:55:42.538691 52:54:00:c6:10:20 (oui Unknown) > 52:54:00:c6:10:10 (oui Unknown), ethertype IPv4 (0x0800), length 60: reserved > reserved: ip-proto-255 26 22:55:42.605892 52:54:00:c6:10:20 (oui Unknown) > 52:54:00:c6:10:10 (oui Unknown), ethertype IPv4 (0x0800), length 60: reserved > reserved: ip-proto-255 26 22:55:42.673091 52:54:00:c6:10:20 (oui Unknown) > 52:54:00:c6:10:10 (oui Unknown), ethertype IPv4 (0x0800), length 60: reserved > reserved: ip-proto-255 26 22:55:42.740292 52:54:00:c6:10:20 (oui Unknown) > 52:54:00:c6:10:10 (oui Unknown), ethertype IPv4 (0x0800), length 60: reserved > reserved: ip-proto-255 26 (In reply to Flavio Leitner from comment #18) > Oct 17 16:59:36 netqe10.knqe.lab.eng.bos.redhat.com ovs-ctl[16991]: > EAL: probe driver: 8086:154d rte_ixgbe_pmd > Oct 17 16:59:36 netqe10.knqe.lab.eng.bos.redhat.com ovs-ctl[16991]: EAL: > Error - exiting with code: 1 > > Looks like a HW issue. Are you sure that no other dpdk app is/was using the > device? > > Please, reboot, make sure the interface is bound to vfio-pci correctly and > start OVS. Hi Flavio, This is a 100% reproducible issue on netqe9 and netqe10. But, not happening on netqe5. Here are the steps: 1. remove openvswitch and dpdk packages 2. reboot 3. install openvswitch 4. cd /etc/sysconfig; cp openvswitch.rpmsave openvswitch 5. systemctl start openvswitch 6. ps -elf | grep ovs-vs ps -elf | grep ovs-vs 1 S root 8506 1 0 70 -10 - 1380552 wait 08:43 ? 00:00:00 ovs-vswit... --dpdk -c 0xaa0000 -n 4 --socket-mem 4096 1 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor 5 S root 8507 8506 0 70 -10 - 1380630 poll_s 08:43 ? 00:00:00 ovs-vswitchd --dpdk -c 0xaa0000 -n 4 --socket-mem 4096 1 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor 7. systemctl stop openvswitch 8. install dpdk packages 9. [root@netqe9 sysconfig]# modprobe vfio-pci [root@netqe9 sysconfig]# dpdk-devbind --bind=vfio-pci 0000:81:00.0 [root@netqe9 sysconfig]# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:81:00.0 'Ethernet 10G 2P X520 Adapter' drv=vfio-pci unused=ixgbe Network devices using kernel driver =================================== 0000:01:00.0 'NetXtreme II BCM57800 1/10 Gigabit Ethernet' if=em1 drv=bnx2x unused=vfio-pci 0000:01:00.1 'NetXtreme II BCM57800 1/10 Gigabit Ethernet' if=em2 drv=bnx2x unused=vfio-pci 0000:01:00.2 'NetXtreme II BCM57800 1/10 Gigabit Ethernet' if=em3 drv=bnx2x unused=vfio-pci *Active* 0000:01:00.3 'NetXtreme II BCM57800 1/10 Gigabit Ethernet' if=em4 drv=bnx2x unused=vfio-pci 0000:03:00.0 'T580-CR Unified Wire Ethernet Controller' if= drv=cxgb4 unused=vfio-pci 0000:03:00.1 'T580-CR Unified Wire Ethernet Controller' if= drv=cxgb4 unused=vfio-pci 0000:03:00.2 'T580-CR Unified Wire Ethernet Controller' if= drv=cxgb4 unused=vfio-pci 0000:03:00.3 'T580-CR Unified Wire Ethernet Controller' if= drv=cxgb4 unused=vfio-pci 0000:03:00.4 'T580-CR Unified Wire Ethernet Controller' if=p6p2,p6p1 drv=cxgb4 unused=vfio-pci 0000:04:00.0 'MT27520 Family [ConnectX-3 Pro]' if=p7p1,p7p2 drv=mlx4_core unused=vfio-pci 0000:05:00.0 'OneConnect NIC (Skyhawk)' if=p5p1 drv=be2net unused=vfio-pci 0000:81:00.1 'Ethernet 10G 2P X520 Adapter' if=p1p2 drv=ixgbe unused=vfio-pci 0000:82:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=p4p1 drv=i40e unused=vfio-pci 0000:82:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=p4p2 drv=i40e unused=vfio-pci 0000:84:00.0 'NetXtreme II BCM57810 10 Gigabit Ethernet' if=p2p1 drv=bnx2x unused=vfio-pci 0000:84:00.1 'NetXtreme II BCM57810 10 Gigabit Ethernet' if=p2p2 drv=bnx2x unused=vfio-pci Other network devices ===================== <none> 10. [root@netqe9 sysconfig]# systemctl start openvswitch [root@netqe9 sysconfig]# !ps ps -elf | grep ovs-vs 0 S root 8910 8277 0 80 0 - 28162 pipe_w 08:48 pts/0 00:00:00 grep --color=auto ovs-vs [root@netqe9 sysconfig]# systemctl status -l openvswitch-nonetwork ● openvswitch-nonetwork.service - Open vSwitch Internal Unit Loaded: loaded (/usr/lib/systemd/system/openvswitch-nonetwork.service; static; vendor preset: disabled) Active: active (exited) since Wed 2016-10-19 08:48:20 EDT; 2min 14s ago Process: 8783 ExecStart=/usr/share/openvswitch/scripts/ovs-ctl start --system-id=random $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 8783 (code=exited, status=0/SUCCESS) CGroup: /system.slice/openvswitch-nonetwork.service ├─8891 ovsdb-server: monitoring pid 8892 (healthy) └─8892 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor Oct 19 08:48:20 netqe9.knqe.lab.eng.bos.redhat.com ovs-ctl[8783]: EAL: PCI device 0000:03:00.4 on NUMA socket 0 Oct 19 08:48:20 netqe9.knqe.lab.eng.bos.redhat.com ovs-ctl[8783]: EAL: probe driver: 1425:540d rte_cxgbe_pmd Oct 19 08:48:20 netqe9.knqe.lab.eng.bos.redhat.com ovs-ctl[8783]: EAL: Not managed by a supported kernel driver, skipped Oct 19 08:48:20 netqe9.knqe.lab.eng.bos.redhat.com ovs-ctl[8783]: EAL: PCI device 0000:81:00.0 on NUMA socket 1 Oct 19 08:48:20 netqe9.knqe.lab.eng.bos.redhat.com ovs-ctl[8783]: EAL: probe driver: 8086:154d rte_ixgbe_pmd Oct 19 08:48:20 netqe9.knqe.lab.eng.bos.redhat.com ovs-ctl[8783]: EAL: Error - exiting with code: 1 Oct 19 08:48:20 netqe9.knqe.lab.eng.bos.redhat.com ovs-ctl[8783]: Cause: Requested device 0000:81:00.0 cannot be used Oct 19 08:48:20 netqe9.knqe.lab.eng.bos.redhat.com ovs-ctl[8783]: [FAILED] Oct 19 08:48:20 netqe9.knqe.lab.eng.bos.redhat.com ovs-ctl[8783]: Enabling remote OVSDB managers [ OK ] Oct 19 08:48:20 netqe9.knqe.lab.eng.bos.redhat.com systemd[1]: Started Open vSwitch Internal Unit. So, dpdk0 is still not usable. You may have noticed that there are no scripts in this sequence of steps. Any idea ? Want to debug on netqe9 before I re-provision the system ? Thanks! Jean FWIW, selinux was enabled. Disabling it fixes the issue. (In reply to Flavio Leitner from comment #22) > FWIW, selinux was enabled. Disabling it fixes the issue. I was using netqe9/netqe10 test-bed for ovs-tcpdump testing after it had run a non-dpdk OVS test suite which has Enforcing as default of Selinux. So, that has been causing ovs-vswitchd fail to come up with dpdk0 present. Run more ovs-tcpdump tests. With the following command: ovs-tcpdump -i dpdk0 -xx -w /tmp/dpdk0.xx the drop rate is very minimal: [root@netqe5 dpdk-multique-scripts]# ovs-tcpdump -i dpdk0 -xx -w /tmp/dpdk0.xx.100000 tcpdump: WARNING: midpdk0: no IPv4 address assigned tcpdump: listening on midpdk0, link-type EN10MB (Ethernet), capture size 65535 bytes ^C7065426 packets captured 7065806 packets received by filter 380 packets dropped by kernel Set status to VERIFIED. Packages need be installed for ovs-tcpdump to work: [root@netqe5 ~]# rpm -qa | grep openvswitch openvswitch-test-2.5.0-12.git20160727.el7fdb.noarch python-openvswitch-2.5.0-12.git20160727.el7fdb.noarch openvswitch-2.5.0-12.git20160727.el7fdb.x86_64 NOTE: If you have hard time installing openvswitch-test after installing python-openvswitch, use "rpm -ivh --nodeps". [root@netqe5 ~]# rpm -qa | grep netiface python-netifaces-0.10.4-3.el7ost.x86_64 [root@netqe5 ~]# rpm -qa | grep dpdk kernel-kernel-networking-ovs-dpdk-vhostuser-1.0-5.noarch dpdk-tools-16.07-1.el7fdb.x86_64 dpdk-16.07-1.el7fdb.x86_64 dpdk-16.04 works too. I tested both. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2129.html |
Description of problem: OVS+DPDK operators lack of tcpdump tool, as interfaces are handled by DPDK (no kernel/libpcap) Version-Release number of selected component (if applicable): OVS 2.4, DPDK 2.0, and above. Expected results: operator being able to run a wrapper around tcpdump, typically named ovs-tcpdump, that will accept all tcpdump options. This tool should refuse to tcpdump twice on an interface, or works fine in that case. It should not let any OVS configuration lingering after exit, and should provide an option to cleanup any OVS configuration that been let behind in case of a tool crash. Additional info: Email on the subject from Flavio: ================================= The tool would pass all options directly to tcpdump with the exception for -i that would be replaced by the mirror device. So, in your example: "tcpdump -i dpdk0 -n -e" would be actually be: "ovs-tcpdump -i dpdk0 -n -e" that does under the hood: ovs-vsctl add-port ovsbr0 debug-port -- set interface type=internal -- set Bridge ovsbr0 mirrors=@m -- --id=@dpdk0 get Port dpdk0 \ -- --id=@debug-port get Port debug-port -- --id=@m create Mirror name=mirror1 \ select-dst-port=@dpdk0 select-src-port=@dpdk0 output-port=@debug-port ip link set debug-port up tcpdump -i debug-port -n -e <tcpdump quit> ovs-vsctl clear bridge br0 mirrors ovs-vsctl del-port ivsbr0 debug-port