Bug 1298666

Summary:	With dpdk-2.2 pktgen test on a vhostuser crashed with segmentation fault on starting traffic
Product:	Red Hat Enterprise Linux 7	Reporter:	Jean-Tsung Hsiao <jhsiao>
Component:	openvswitch-dpdk	Assignee:	Flavio Leitner <fleitner>
Status:	CLOSED NOTABUG	QA Contact:	Jean-Tsung Hsiao <jhsiao>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	7.2	CC:	aloughla, jhsiao, kzhang, rcain
Target Milestone:	rc
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	pktgen-dpdk-2.9.8-1.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-02-18 21:21:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jean-Tsung Hsiao 2016-01-14 17:17:19 UTC

Description of problem: With dpdk-2.2 pktgen test on a vhostuser crashed with segmentation fault on starting traffic.

!ERROR!: Unknown EtherType 0x0000
!ERROR!: Unknown EtherType 0x0000
!ERROR!: Unknown EtherType 0x0000
!ERROR!: Unknown EtherType 0x0000
!ERROR!: Unknown EtherType 0x0000
!ERROR!: Unknown EtherType 0x0000
!ERROR!: Unknown EtherType 0x0000
!ERROR!: Unknown EtherType 0x0000
!ERROR!: Unknown EtherType 0x0000
Segmentation fault (core dumped)

*** Command line ***
pktgen -w 0000:00:03.0 --socket-mem 1024 --file-prefix pg  -- -T -P -m "[2:2].0"

NOTE: Similar crash in a host to host environment:

PANIC in pktgen_setup_packets():
Unable to allocate 27 pkt_seq_t headers8: [/lib64/libc.so.6(clone+0x6d) [0x7fc0b66401cd]]
7: [/lib64/libpthread.so.0(+0x7dc5) [0x7fc0b6912dc5]]
6: [/lib64/librte_eal.so.2(+0xa157) [0x7fc0b8ebb157]]
5: [pktgen(pktgen_launch_one_lcore+0x9f) [0x4243bf]]
4: [pktgen() [0x4241c5]]
3: [pktgen() [0x40dd10]]
2: [/lib64/librte_eal.so.2(__rte_panic+0xd0) [0x7fc0b8eb6460]]
1: [/lib64/librte_eal.so.2(rte_dump_stack+0x2d) [0x7fc0b8ebe60d]]
Aborted (core dumped)


Version-Release number of selected component (if applicable):
[root@localhost ~]# rpm -qa | grep dpdk
dpdk-tools-2.2.0-1.el7.x86_64
dpdk-2.2.0-1.el7.x86_64
pktgen-dpdk-2.9.7-1.el7.centos.x86_64


How reproducible: reproducible


Steps to Reproduce:
1. install dpdk2-2 to a vhostuser guest
2. Run pktgen
3. load a packet sequence, then start

Actual results:

Crashed
Expected results:
Not crashed


Additional info:

Comment 2 Flavio Leitner 2016-01-28 13:51:32 UTC

Hi Jean,

What exactly is crashing? Just pktgen?
If so, can you rebuild pktgen RPM with DPDK 2.2.0 devel package installed and see if it fixes the issue?

Thanks,
fbl

Comment 3 Jean-Tsung Hsiao 2016-01-29 18:52:42 UTC

(In reply to Flavio Leitner from comment #2)
> Hi Jean,
> 
> What exactly is crashing? Just pktgen?
> If so, can you rebuild pktgen RPM with DPDK 2.2.0 devel package installed
> and see if it fixes the issue?
> 
> Thanks,
> fbl

hi fbl,

Yes, only pktgen crashed.

Please show me steps to rebuild dpktgen.

Thanks!

Jean

Comment 4 Flavio Leitner 2016-02-15 16:05:55 UTC

<fbl> jhsiao: we don't ship pktgen, so it can't be on Extras
<fbl> jhsiao: ok, could you rpm -qpi <package> and tell me the Packager and Vendor?
<jhsiao> [root@netqe5 jhsiao]# rpm -qpi pktgen-dpdk-2.9.7-1.el7.centos.x86_64.rpm
<jhsiao> warning: pktgen-dpdk-2.9.7-1.el7.centos.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 99b57f82: NOKEY
<jhsiao> [root@netqe5 jhsiao]# rpm -qpi pktgen-dpdk-2.9.7-1.el7.centos.x86_64.rpm
<jhsiao> warning: pktgen-dpdk-2.9.7-1.el7.centos.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 99b57f82: NOKEY
<jhsiao> Name        : pktgen-dpdk
<jhsiao> Version     : 2.9.7
<jhsiao> Release     : 1.el7.centos
<jhsiao> Architecture: x86_64
<jhsiao> Install Date: (not installed)
<jhsiao> Group       : Applications/Internet
<jhsiao> Size        : 13288128
<jhsiao> License     : BSD (??)
<jhsiao> Signature   : RSA/SHA1, Wed 16 Dec 2015 08:29:09 AM EST, Key ID 32e9738e99b57f82
<jhsiao> Source RPM  : pktgen-dpdk-2.9.7-1.el7.centos.src.rpm
<jhsiao> Build Date  : Wed 16 Dec 2015 08:28:50 AM EST
<jhsiao> Build Host  : copr-builder-109657657.novalocal
<jhsiao> Relocations : (not relocatable)
<jhsiao> Vendor      : Fedora Project COPR (pmatilai/dpdk)
<jhsiao> URL         : https://github.com/Pktgen/Pktgen-DPDK/
<jhsiao> Summary     : Traffic generator utilizing DPDK
<jhsiao> Description :
<jhsiao> Traffic generator utilizing DPDK

Comment 5 Flavio Leitner 2016-02-15 16:13:44 UTC

That's an old package built on Dec/2015.
You might want to get a new build against dpdk 2.2.0.
Would you mind to try with pktgen-dpdk-2.9.8-1.el7?

# rpm -qi pktgen-dpdk 
Name        : pktgen-dpdk
Version     : 2.9.8
Release     : 1.el7.centos
Architecture: x86_64
Install Date: Mon 15 Feb 2016 02:12:00 PM BRST
Group       : Applications/Internet
Size        : 13292231
License     : BSD (??)
Signature   : RSA/SHA1, Wed 20 Jan 2016 07:59:15 AM BRST, Key ID f454e0cf1faf09da
Source RPM  : pktgen-dpdk-2.9.8-1.el7.centos.src.rpm
Build Date  : Wed 20 Jan 2016 07:59:02 AM BRST
Build Host  : copr-builder-648026050.novalocal
Relocations : (not relocatable)
Vendor      : Fedora Project COPR (pmatilai/dpdk-snapshot)
URL         : https://github.com/Pktgen/Pktgen-DPDK/
Summary     : Traffic generator utilizing DPDK
Description :
Traffic generator utilizing DPDK


Thanks,
fbl

Comment 6 Jean-Tsung Hsiao 2016-02-18 21:15:03 UTC

(In reply to Flavio Leitner from comment #5)
> That's an old package built on Dec/2015.
> You might want to get a new build against dpdk 2.2.0.
> Would you mind to try with pktgen-dpdk-2.9.8-1.el7?
> 
> # rpm -qi pktgen-dpdk 
> Name        : pktgen-dpdk
> Version     : 2.9.8
> Release     : 1.el7.centos
> Architecture: x86_64
> Install Date: Mon 15 Feb 2016 02:12:00 PM BRST
> Group       : Applications/Internet
> Size        : 13292231
> License     : BSD (??)
> Signature   : RSA/SHA1, Wed 20 Jan 2016 07:59:15 AM BRST, Key ID
> f454e0cf1faf09da
> Source RPM  : pktgen-dpdk-2.9.8-1.el7.centos.src.rpm
> Build Date  : Wed 20 Jan 2016 07:59:02 AM BRST
> Build Host  : copr-builder-648026050.novalocal
> Relocations : (not relocatable)
> Vendor      : Fedora Project COPR (pmatilai/dpdk-snapshot)
> URL         : https://github.com/Pktgen/Pktgen-DPDK/
> Summary     : Traffic generator utilizing DPDK
> Description :
> Traffic generator utilizing DPDK
> 
> 
> Thanks,
> fbl

Yes, the updated openvswitch 2.5 package fixes the issue.

Comment 7 Jean-Tsung Hsiao 2016-02-18 21:21:24 UTC

(In reply to Jean-Tsung Hsiao from comment #6)
> (In reply to Flavio Leitner from comment #5)
> > That's an old package built on Dec/2015.
> > You might want to get a new build against dpdk 2.2.0.
> > Would you mind to try with pktgen-dpdk-2.9.8-1.el7?
> > 
> > # rpm -qi pktgen-dpdk 
> > Name        : pktgen-dpdk
> > Version     : 2.9.8
> > Release     : 1.el7.centos
> > Architecture: x86_64
> > Install Date: Mon 15 Feb 2016 02:12:00 PM BRST
> > Group       : Applications/Internet
> > Size        : 13292231
> > License     : BSD (??)
> > Signature   : RSA/SHA1, Wed 20 Jan 2016 07:59:15 AM BRST, Key ID
> > f454e0cf1faf09da
> > Source RPM  : pktgen-dpdk-2.9.8-1.el7.centos.src.rpm
> > Build Date  : Wed 20 Jan 2016 07:59:02 AM BRST
> > Build Host  : copr-builder-648026050.novalocal
> > Relocations : (not relocatable)
> > Vendor      : Fedora Project COPR (pmatilai/dpdk-snapshot)
> > URL         : https://github.com/Pktgen/Pktgen-DPDK/
> > Summary     : Traffic generator utilizing DPDK
> > Description :
> > Traffic generator utilizing DPDK
> > 
> > 
> > Thanks,
> > fbl
> 
> Yes, the updated openvswitch 2.5 package fixes the issue.

Sorry, it should be pktgen-dpdk-2.9.8-1.el7.