Bug 838674

Summary:	intermittent network with RHS 2.0 20120621.2
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Kaleb KEITHLEY <kkeithle>
Component:	distribution	Assignee:	Bug Updates Notification Mailing List <rhs-bugs>
Status:	CLOSED EOL	QA Contact:	storage-qa-internal <storage-qa-internal>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.0	CC:	poelstra, rwheeler, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-02-17 17:19:49 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Kaleb KEITHLEY 2012-07-09 18:36:24 UTC

Description of problem:

network ping statistics on the gqaib-0x machines suggest a possible problem with the network stack in RHS-2.0 20120621.2


Version-Release number of selected component (if applicable):


How reproducible:

Install RHEL, e.g. 6.2, 5.8, etc. on gqaib-0x machines. After installation, ping the box(es). Notice 0% packet loss. Install RHS 2.0 20120621.2 on the same hardware and notice substantial packet loss.


Steps to Reproduce:
1.
2.
3.
  
Actual results:

With RHS 2.0 20120621.2 installed:

--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
1926 packets transmitted, 1781 received, 7% packet loss, time 1925745ms
rtt min/avg/max/mdev = 0.281/0.511/0.611/0.046 ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
1927 packets transmitted, 1861 received, 3% packet loss, time 1926843ms
rtt min/avg/max/mdev = 0.277/0.516/0.695/0.041 ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
1924 packets transmitted, 80 received, 95% packet loss, time 1923356ms
rtt min/avg/max/mdev = 644.441/10211.788/19839.758/5763.664 ms, pipe 20

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
5468 packets transmitted, 689 received, 87% packet loss, time 5467795ms
rtt min/avg/max/mdev = 0.288/2358.279/19916.261/5097.924 ms, pipe 20


Expected results:

E.g. with RHEL 6.2 installed:


--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
2218 packets transmitted, 2218 received, 0% packet loss, time 2217802ms
rtt min/avg/max/mdev = 0.276/0.508/0.594/0.049 ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
2216 packets transmitted, 2216 received, 0% packet loss, time 2215325ms
rtt min/avg/max/mdev = 0.279/0.483/1.375/0.086 ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
2213 packets transmitted, 2213 received, 0% packet loss, time 2212656ms
rtt min/avg/max/mdev = 0.278/0.463/1.639/0.103 ms

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
2211 packets transmitted, 2211 received, 0% packet loss, time 2210370ms
rtt min/avg/max/mdev = 0.289/0.521/0.817/0.043 ms


Additional info:

See eng-ops ticket # 158826 and 159182 and following email from eng-ops (Bill Therrien):

The network issues that are occurring seem to be caused by problems in the
RHS2.0
(out of 41 installs of this distro, only 9 of my jobs finished in the green)
https://beaker.engineering.redhat.com/jobs/258676
https://beaker.engineering.redhat.com/jobs/258645
https://beaker.engineering.redhat.com/jobs/258598
https://beaker.engineering.redhat.com/jobs/258099
https://beaker.engineering.redhat.com/jobs/258086


ping statistics whith RHS2.0 installed:

--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
1926 packets transmitted, 1781 received, 7% packet loss, time 1925745ms
rtt min/avg/max/mdev = 0.281/0.511/0.611/0.046 ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
1927 packets transmitted, 1861 received, 3% packet loss, time 1926843ms
rtt min/avg/max/mdev = 0.277/0.516/0.695/0.041 ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
1924 packets transmitted, 80 received, 95% packet loss, time 1923356ms
rtt min/avg/max/mdev = 644.441/10211.788/19839.758/5763.664 ms, pipe 20

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
5468 packets transmitted, 689 received, 87% packet loss, time 5467795ms
rtt min/avg/max/mdev = 0.288/2358.279/19916.261/5097.924 ms, pipe 20


ping statistics with other distros were rock solid, indicating that there is no
hardware problem present:

*RHEL 5.8*

--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
5473 packets transmitted, 5473 received, 0% packet loss, time 5472925ms
rtt min/avg/max/mdev = 0.266/0.455/1.663/0.099 ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
5475 packets transmitted, 5475 received, 0% packet loss, time 5474182ms
rtt min/avg/max/mdev = 0.272/0.508/1.625/0.047 ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
5471 packets transmitted, 5471 received, 0% packet loss, time 5470224ms
rtt min/avg/max/mdev = 0.271/0.509/0.601/0.038 ms

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
1922 packets transmitted, 1922 received, 0% packet loss, time 1921525ms
rtt min/avg/max/mdev = 0.291/0.524/1.644/0.043 ms


*RHEL 6.2*

--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
2218 packets transmitted, 2218 received, 0% packet loss, time 2217802ms
rtt min/avg/max/mdev = 0.276/0.508/0.594/0.049 ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
2216 packets transmitted, 2216 received, 0% packet loss, time 2215325ms
rtt min/avg/max/mdev = 0.279/0.483/1.375/0.086 ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
2213 packets transmitted, 2213 received, 0% packet loss, time 2212656ms
rtt min/avg/max/mdev = 0.278/0.463/1.639/0.103 ms

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
2211 packets transmitted, 2211 received, 0% packet loss, time 2210370ms
rtt min/avg/max/mdev = 0.289/0.521/0.817/0.043 ms


I also tried bringing the interface down and up again to see if this would
help, it seems to have stabilized the connection for two systems but the other
two it did not help:

--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
189 packets transmitted, 9 received, 95% packet loss, time 188262ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
186 packets transmitted, 6 received, 96% packet loss, time 185989ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
184 packets transmitted, 184 received, 0% packet loss, time 183758ms

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
182 packets transmitted, 182 received, 0% packet loss, time 181464ms

Comment 1 Kaleb KEITHLEY 2012-07-09 18:46:05 UTC

https://engineering.redhat.com/rt/Ticket/Display.html?id=159182

https://engineering.redhat.com/rt/Ticket/Display.html?id=158826

Comment 3 Ric Wheeler 2012-08-09 16:44:18 UTC

Is there anyone on the RHEL side working on this? Is this a kernel issue or a RHS issue (samba IP handling? other?).

Kaleb's test seems extremely straight forward.

Comment 4 Scott Haines 2012-09-18 21:49:07 UTC

Ping.

Comment 6 Amar Tumballi 2013-11-26 09:01:02 UTC

Who should take this forward?

For now, removing Corbett flag.

Comment 7 Kaleb KEITHLEY 2015-02-17 17:19:49 UTC

Never resolved, ancient release