Description of problem:

network ping statistics on the gqaib-0x machines suggest a possible problem with the network stack in RHS-2.0 20120621.2


Version-Release number of selected component (if applicable):


How reproducible:

Install RHEL, e.g. 6.2, 5.8, etc. on gqaib-0x machines. After installation, ping the box(es). Notice 0% packet loss. Install RHS 2.0 20120621.2 on the same hardware and notice substantial packet loss.


Steps to Reproduce:
1.
2.
3.
  
Actual results:

With RHS 2.0 20120621.2 installed:

--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
1926 packets transmitted, 1781 received, 7% packet loss, time 1925745ms
rtt min/avg/max/mdev = 0.281/0.511/0.611/0.046 ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
1927 packets transmitted, 1861 received, 3% packet loss, time 1926843ms
rtt min/avg/max/mdev = 0.277/0.516/0.695/0.041 ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
1924 packets transmitted, 80 received, 95% packet loss, time 1923356ms
rtt min/avg/max/mdev = 644.441/10211.788/19839.758/5763.664 ms, pipe 20

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
5468 packets transmitted, 689 received, 87% packet loss, time 5467795ms
rtt min/avg/max/mdev = 0.288/2358.279/19916.261/5097.924 ms, pipe 20


Expected results:

E.g. with RHEL 6.2 installed:


--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
2218 packets transmitted, 2218 received, 0% packet loss, time 2217802ms
rtt min/avg/max/mdev = 0.276/0.508/0.594/0.049 ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
2216 packets transmitted, 2216 received, 0% packet loss, time 2215325ms
rtt min/avg/max/mdev = 0.279/0.483/1.375/0.086 ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
2213 packets transmitted, 2213 received, 0% packet loss, time 2212656ms
rtt min/avg/max/mdev = 0.278/0.463/1.639/0.103 ms

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
2211 packets transmitted, 2211 received, 0% packet loss, time 2210370ms
rtt min/avg/max/mdev = 0.289/0.521/0.817/0.043 ms


Additional info:

See eng-ops ticket # 158826 and 159182 and following email from eng-ops (Bill Therrien):

The network issues that are occurring seem to be caused by problems in the
RHS2.0
(out of 41 installs of this distro, only 9 of my jobs finished in the green)
https://beaker.engineering.redhat.com/jobs/258676
https://beaker.engineering.redhat.com/jobs/258645
https://beaker.engineering.redhat.com/jobs/258598
https://beaker.engineering.redhat.com/jobs/258099
https://beaker.engineering.redhat.com/jobs/258086


ping statistics whith RHS2.0 installed:

--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
1926 packets transmitted, 1781 received, 7% packet loss, time 1925745ms
rtt min/avg/max/mdev = 0.281/0.511/0.611/0.046 ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
1927 packets transmitted, 1861 received, 3% packet loss, time 1926843ms
rtt min/avg/max/mdev = 0.277/0.516/0.695/0.041 ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
1924 packets transmitted, 80 received, 95% packet loss, time 1923356ms
rtt min/avg/max/mdev = 644.441/10211.788/19839.758/5763.664 ms, pipe 20

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
5468 packets transmitted, 689 received, 87% packet loss, time 5467795ms
rtt min/avg/max/mdev = 0.288/2358.279/19916.261/5097.924 ms, pipe 20


ping statistics with other distros were rock solid, indicating that there is no
hardware problem present:

*RHEL 5.8*

--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
5473 packets transmitted, 5473 received, 0% packet loss, time 5472925ms
rtt min/avg/max/mdev = 0.266/0.455/1.663/0.099 ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
5475 packets transmitted, 5475 received, 0% packet loss, time 5474182ms
rtt min/avg/max/mdev = 0.272/0.508/1.625/0.047 ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
5471 packets transmitted, 5471 received, 0% packet loss, time 5470224ms
rtt min/avg/max/mdev = 0.271/0.509/0.601/0.038 ms

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
1922 packets transmitted, 1922 received, 0% packet loss, time 1921525ms
rtt min/avg/max/mdev = 0.291/0.524/1.644/0.043 ms


*RHEL 6.2*

--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
2218 packets transmitted, 2218 received, 0% packet loss, time 2217802ms
rtt min/avg/max/mdev = 0.276/0.508/0.594/0.049 ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
2216 packets transmitted, 2216 received, 0% packet loss, time 2215325ms
rtt min/avg/max/mdev = 0.279/0.483/1.375/0.086 ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
2213 packets transmitted, 2213 received, 0% packet loss, time 2212656ms
rtt min/avg/max/mdev = 0.278/0.463/1.639/0.103 ms

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
2211 packets transmitted, 2211 received, 0% packet loss, time 2210370ms
rtt min/avg/max/mdev = 0.289/0.521/0.817/0.043 ms


I also tried bringing the interface down and up again to see if this would
help, it seems to have stabilized the connection for two systems but the other
two it did not help:

--- gqaib-01.sbu.lab.eng.bos.redhat.com ping statistics ---
189 packets transmitted, 9 received, 95% packet loss, time 188262ms

--- gqaib-02.sbu.lab.eng.bos.redhat.com ping statistics ---
186 packets transmitted, 6 received, 96% packet loss, time 185989ms

--- gqaib-03.sbu.lab.eng.bos.redhat.com ping statistics ---
184 packets transmitted, 184 received, 0% packet loss, time 183758ms

--- gqaib-04.sbu.lab.eng.bos.redhat.com ping statistics ---
182 packets transmitted, 182 received, 0% packet loss, time 181464ms