Bug 1609607

Summary: [whql][netkvm]Job named NDISTest6.5-InvalidPackets failed with mq in win8-32
Product: Red Hat Enterprise Linux 7 Reporter: xiagao
Component: virtio-winAssignee: Amnon Ilan <ailan>
virtio-win sub component: virtio-win-prewhql QA Contact: Virtualization Bugs <virt-bugs>
Status: CLOSED NOTABUG Docs Contact:
Severity: high    
Priority: high CC: ailan, lijin, lmiksik, phou, sjubran, virt-bugs, vrozenfe, wyu, xiagao, yvugenfi
Version: 7.6   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1248873 Environment:
Last Closed: 2018-08-13 03:16:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1248873    
Bug Blocks: 1288337    
Attachments:
Description Flags
hck log
none
I am confused. Attached log shows that Invalid Packets test passes...
none
all logs.
none
update another hck log none

Comment 3 xiagao 2018-07-30 03:11:51 UTC
steps are same with comment#0;

packages info:
qemu-kvm-rhev-2.12.0-8.el7.x86_64
kernel-3.10.0-926.el7.x86_64
virtio-win-prewhql-157
seabios-bin-1.11.0-2.el7.noarch

Can reproduce in virtio-win-prewhql-151 and virtio-win-prewhql-144.
So it's not a regression.

Comment 4 xiagao 2018-07-30 03:30:59 UTC
Created attachment 1471409 [details]
hck log

Comment 5 Yvugenfi@redhat.com 2018-07-30 06:17:45 UTC
Why this bug is a clone?

According to https://bugzilla.redhat.com/show_bug.cgi?id=1248873#c9 , #1248873 was fixed.

Are you should it is not a regression, did you compare to latest released version?

>>>>
Hello Yan,

Rerun the test case NDIS6.5 - InvalidPackets passed with multiqueue on build 111.  -win8.1_32 

Rerun the test case NDIS6.5 - InvalidPackets passed with multiqueue on build 111.  -win10_64 


Best Regards!
Peixiu Hou
>>>>

Comment 7 Yvugenfi@redhat.com 2018-07-30 06:31:59 UTC
Created attachment 1471419 [details]
I am confused. Attached log shows that Invalid Packets test passes...

I am confused. Attached log shows that Invalid Packets test passes...

Comment 8 xiagao 2018-07-30 06:59:00 UTC
(In reply to Yan Vugenfirer from comment #7)
> Created attachment 1471419 [details]
> I am confused. Attached log shows that Invalid Packets test passes...
> 
> I am confused. Attached log shows that Invalid Packets test passes...

That's because I run three times,two failed with mq and one pass without mq.

Comment 9 xiagao 2018-07-30 07:00:47 UTC
Created attachment 1471423 [details]
all logs.

Could you see all logs?

Comment 10 Yvugenfi@redhat.com 2018-07-30 07:18:49 UTC
(In reply to xiagao from comment #9)
> Created attachment 1471423 [details]
> all logs.
> 
> Could you see all logs?

I don't have failed results in the attached package.

Comment 11 xiagao 2018-07-30 09:03:10 UTC
Created attachment 1471462 [details]
update another hck log

I update one again, pls have a check.

Test result:
build 111 with mq,hit this issue.
build 145(released version of rhel7.5) with mq,hit it.
build 157 in win8.1-32 with mq,pass
build 157 without mq,pass

Comment 12 xiagao 2018-07-30 09:12:53 UTC
(In reply to xiagao from comment #11)
> Created attachment 1471462 [details]
> update another hck log
> 
> I update one again, pls have a check.
> 
> Test result:
> build 111 with mq,hit this issue.
> build 145(released version of rhel7.5) with mq,hit it.
> build 157 in win8.1-32 with mq,pass
> build 157 without mq,pass

I disabled netfilter on the bridge already in the test host.

# sysctl -a | grep bridge
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-pass-vlan-input-dev = 0

Comment 13 Yvugenfi@redhat.com 2018-07-30 13:19:32 UTC
Current failure:

Error 7/30/2018 11:46:41.670 AM 8 total breakpoints were hit in the protocol driver while this test was executing 
File:    Line: 0 
Error Type:   WIN32 
Error Code:   0x88888 
Error Text:   Error 0x00088888 
Message 7/30/2018 11:46:41.680 AM NDISTest Protocol driver breakpoints are usually hit when a serious error is detected. Please investigate and fix the issue causing the break to solve this failure. 
Message 7/30/2018 11:46:41.683 AM There were 2 unique breakpoints hit: 
Message 7/30/2018 11:46:41.687 AM Breakpoint Summary
NDISTest Source File	Line	Hit Count	Message	
ndistestheadermodule.cpp	696	20	CNDTNDISTestHeaderModule::OnPacketListsIn: Dropped indications, Expected 1289 but received 1483 
	
ndistestheadermodule.cpp	671	8	CNDTNDISTestHeaderModule::OnPacketListsIn: Out of order indication, Expected 29 but received 20

Comment 14 Sameeh Jubran 2018-08-05 10:04:11 UTC
(In reply to xiagao from comment #11)
> Created attachment 1471462 [details]
> update another hck log
> 
> I update one again, pls have a check.
> 
> Test result:
> build 111 with mq,hit this issue.
> build 145(released version of rhel7.5) with mq,hit it.
> build 157 in win8.1-32 with mq,pass
> build 157 without mq,pass

I am very confused. According to the comment above in build 157 the test passes with no issues. When does it fail? 

What about build 159?

Comment 15 xiagao 2018-08-06 06:06:37 UTC
(In reply to Sameeh Jubran from comment #14)
> (In reply to xiagao from comment #11)
> > Created attachment 1471462 [details]
> > update another hck log
> > 
> > I update one again, pls have a check.
> > 
> > Test result:
> > build 111 with mq,hit this issue.
> > build 145(released version of rhel7.5) with mq,hit it.
> > build 157 in win8.1-32 with mq,pass
> > build 157 without mq,pass
> 
> I am very confused. According to the comment above in build 157 the test
> passes with no issues. When does it fail? 

The point is for win8-32 without mq,the result is pass. With mq,the result is fail.

> 
> What about build 159?
Hit the same issue.

But, if I set all network card's txqueuelen value  with 8196 on host, I test five times, failed once.

Comment 16 Yvugenfi@redhat.com 2018-08-06 07:16:59 UTC
(In reply to xiagao from comment #15)
> (In reply to Sameeh Jubran from comment #14)
> > (In reply to xiagao from comment #11)
> > > Created attachment 1471462 [details]
> > > update another hck log
> > > 
> > > I update one again, pls have a check.
> > > 
> > > Test result:
> > > build 111 with mq,hit this issue.
> > > build 145(released version of rhel7.5) with mq,hit it.
> > > build 157 in win8.1-32 with mq,pass
> > > build 157 without mq,pass
> > 
> > I am very confused. According to the comment above in build 157 the test
> > passes with no issues. When does it fail? 
> 
> The point is for win8-32 without mq,the result is pass. With mq,the result
> is fail.
> 
> > 
> > What about build 159?
> Hit the same issue.
> 
> But, if I set all network card's txqueuelen value  with 8196 on host, I test
> five times, failed once.

Can you check statistics on the host: packets rx\tx\dropped and compare it to the test statistics (the test should show how much packets it sent)?

Comment 17 xiagao 2018-08-07 05:26:05 UTC
(In reply to Yan Vugenfirer from comment #16)
> (In reply to xiagao from comment #15)
> > (In reply to Sameeh Jubran from comment #14)
> > > (In reply to xiagao from comment #11)
> > > > Created attachment 1471462 [details]
> > > > update another hck log
> > > > 
> > > > I update one again, pls have a check.
> > > > 
> > > > Test result:
> > > > build 111 with mq,hit this issue.
> > > > build 145(released version of rhel7.5) with mq,hit it.
> > > > build 157 in win8.1-32 with mq,pass
> > > > build 157 without mq,pass
> > > 
> > > I am very confused. According to the comment above in build 157 the test
> > > passes with no issues. When does it fail? 
> > 
> > The point is for win8-32 without mq,the result is pass. With mq,the result
> > is fail.
> > 
> > > 
> > > What about build 159?
> > Hit the same issue.
> > 
> > But, if I set all network card's txqueuelen value  with 8196 on host, I test
> > five times, failed once.
> 
> Can you check statistics on the host: packets rx\tx\dropped and compare it
> to the test statistics (the test should show how much packets it sent)?

As beaker host is reinstalled, I installed two new guests in it to reproduce this issue with build 157/159,but didn't reproduce it. :(
Tried 5 times.

Comment 18 Sameeh Jubran 2018-08-07 08:28:04 UTC
(In reply to xiagao from comment #17)
> (In reply to Yan Vugenfirer from comment #16)
> > (In reply to xiagao from comment #15)
> > > (In reply to Sameeh Jubran from comment #14)
> > > > (In reply to xiagao from comment #11)
> > > > > Created attachment 1471462 [details]
> > > > > update another hck log
> > > > > 
> > > > > I update one again, pls have a check.
> > > > > 
> > > > > Test result:
> > > > > build 111 with mq,hit this issue.
> > > > > build 145(released version of rhel7.5) with mq,hit it.
> > > > > build 157 in win8.1-32 with mq,pass
> > > > > build 157 without mq,pass
> > > > 
> > > > I am very confused. According to the comment above in build 157 the test
> > > > passes with no issues. When does it fail? 
> > > 
> > > The point is for win8-32 without mq,the result is pass. With mq,the result
> > > is fail.
> > > 
> > > > 
> > > > What about build 159?
> > > Hit the same issue.
> > > 
> > > But, if I set all network card's txqueuelen value  with 8196 on host, I test
> > > five times, failed once.
> > 
> > Can you check statistics on the host: packets rx\tx\dropped and compare it
> > to the test statistics (the test should show how much packets it sent)?
> 
> As beaker host is reinstalled, I installed two new guests in it to reproduce
> this issue with build 157/159,but didn't reproduce it. :(
> Tried 5 times.

So it doesn't reproduce anymore with the new host?
Which version is the new host?
Did you use a newer qemu version aswel?

Comment 19 xiagao 2018-08-07 10:07:42 UTC
(In reply to Sameeh Jubran from comment #18)
> (In reply to xiagao from comment #17)
> > (In reply to Yan Vugenfirer from comment #16)
> > > (In reply to xiagao from comment #15)
> > > > (In reply to Sameeh Jubran from comment #14)
> > > > > (In reply to xiagao from comment #11)
> > > > > > Created attachment 1471462 [details]
> > > > > > update another hck log
> > > > > > 
> > > > > > I update one again, pls have a check.
> > > > > > 
> > > > > > Test result:
> > > > > > build 111 with mq,hit this issue.
> > > > > > build 145(released version of rhel7.5) with mq,hit it.
> > > > > > build 157 in win8.1-32 with mq,pass
> > > > > > build 157 without mq,pass
> > > > > 
> > > > > I am very confused. According to the comment above in build 157 the test
> > > > > passes with no issues. When does it fail? 
> > > > 
> > > > The point is for win8-32 without mq,the result is pass. With mq,the result
> > > > is fail.
> > > > 
> > > > > 
> > > > > What about build 159?
> > > > Hit the same issue.
> > > > 
> > > > But, if I set all network card's txqueuelen value  with 8196 on host, I test
> > > > five times, failed once.
> > > 
> > > Can you check statistics on the host: packets rx\tx\dropped and compare it
> > > to the test statistics (the test should show how much packets it sent)?
> > 
> > As beaker host is reinstalled, I installed two new guests in it to reproduce
> > this issue with build 157/159,but didn't reproduce it. :(
> > Tried 5 times.
> 
> So it doesn't reproduce anymore with the new host?
run seven times, didn't reproduce. 

> Which version is the new host?
KERNEL: 3.10.0-931.el7.x86_64
QEMU: qemu-kvm-rhev-2.12.0-9.el7.x86_64

> Did you use a newer qemu version aswel?
yes,the latest one.

Comment 20 xiagao 2018-08-13 03:16:34 UTC
Test it in the new host with build 160.The result is pass.
So close this bug.