Description of problem: Our mailserver has been sending oversized packets since mid morining. No notice is take of the MTU of 1500 on eth0, we have been getting packet up to 20k !! Version-Release number of selected component (if applicable): 2.6.35.12-88.fc14.i686 How reproducible: As I don't know the cause, I can't say but it is happening a lot. Steps to Reproduce: 1. 2. 3. Actual results: 18:22:46.036994 IP agree-31.godzone.net.nz.pop3 > mail-fx0-f2.google.com.37569: Flags [.], seq 95273:110009, ack 49, win 20, options [nop,nop,TS val 7499779 ecr 612154545], length 14736 ---- 14736 !! 18:22:46.039555 IP mail-fx0-f2.google.com.37569 > agree-31.godzone.net.nz.pop3: Flags [.], ack 70713, win 1002, options [nop,nop,TS val 612154547 ecr 7499394], length 0 18:22:46.039570 IP mail-fx0-f2.google.com.37569 > agree-31.godzone.net.nz.pop3: Flags [.], ack 73169, win 1002, options [nop,nop,TS val 612154547 ecr 7499396], length 0 18:22:46.040291 IP mail-fx0-f2.google.com.37569 > agree-31.godzone.net.nz.pop3: Flags [.], ack 75625, win 1024, options [nop,nop,TS val 612154547 ecr 7499396], length 0 18:22:46.041092 IP mail-fx0-f2.google.com.37569 > agree-31.godzone.net.nz.pop3: Flags [.], ack 78081, win 1024, options [nop,nop,TS val 612154548 ecr 7499396], length 0 18:22:46.041104 IP mail-fx0-f2.google.com.37569 > agree-31.godzone.net.nz.pop3: Flags [.], ack 80537, win 1024, options [nop,nop,TS val 612154549 ecr 7499396], length 0 18:22:46.041120 IP agree-31.godzone.net.nz.pop3 > mail-fx0-f2.google.com.37569: Flags [.], seq 110009:118605, ack 49, win 20, options [nop,nop,TS val 7499783 ecr 612154549], length 8596 ---- 8596 !! 18:22:46.041142 IP agree-31.godzone.net.nz.pop3 > mail-fx0-f2.google.com.37569: Flags [P.], seq 118605:119758, ack 49, win 20, options [nop,nop,TS val 7499783 ecr 612154549], length 1153 18:22:46.041591 IP mail-fx0-f2.google.com.37569 > agree-31.godzone.net.nz.pop3: Flags [.], ack 82993, win 1024, o Expected results: Certainly not packets of length greater than the MTU, which we dropped to see if it would help. eth0 Link encap:Ethernet HWaddr 00:0C:29:8A:53:01 inet addr:192.138.251.31 Bcast:192.138.251.127 Mask:255.255.255.128 inet6 addr: 2001:df0:dc::251:31/64 Scope:Global inet6 addr: fe80::20c:29ff:fe8a:5301/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1280 Metric:1 RX packets:1704615 errors:0 dropped:0 overruns:0 frame:0 TX packets:1602035 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2174572162 (2.0 GiB) TX bytes:842868785 (803.8 MiB) Additional info: The server has been rebooted multiple time. It is a VM running on ESXi 4.0 and is using the e1000 driver. Started with pcnet32 but that has the issue as well. We currently have net.ipv4.ip_pmtu_disc set to 1.
*** Bug 702557 has been marked as a duplicate of this bug. ***
Did this just start happening with a recent kernel update?
I would say no, but that may not be correct. I have tried going back to 2.6.35.11-83.fc14.i686 but still see the same issue. I tend to upgrade using yum early Sunday mornings and reboot ( if a kernel was updated later the same week ) looking at most of the other servers, they were rebooted within the last 10 days and are running 2.6.35.12-88.fc14.i686 as well. As to why this server is behaving this way and why it apparently started doing so yesterday morning is beyond me. The only other thing that makes this server different is it is the only one using iSCSI as the maildirs are on a couple of NAS boxes. Until we put the MTU down to 1280, we were seeing a lot of packets at 2880 and 2696. Also until we turned of PMTU discovery, we same the packet the retransmitted packet only smaller by exactly 40 bytes but still over size e.g. 2960 -> 2920.
I noted that there was a new kernel this morning so have upgraded a few of our servers to 2.6.35.12-90.fc14.i686, the problem is still there. Here is a dump from a different server. I get a little confused reading tcpdumps but this doesn't look right to me. 08:13:32.530129 IP agree-14.godzone.net.nz.53385 > mta-v1.mail.vip.sp2.yahoo.com.smtp: Flags [.], seq 50680:52128, ack 1, win 23, options [nop,nop,TS val 103994 ecr 2591672474], length 1448 08:13:32.765059 IP mta-v1.mail.vip.sp2.yahoo.com.smtp > agree-14.godzone.net.nz.53385: Flags [.], ack 52128, win 8326, options [nop,nop,TS val 2591672710 ecr 103994], length 0 08:13:32.765196 IP agree-14.godzone.net.nz.53385 > mta-v1.mail.vip.sp2.yahoo.com.smtp: Flags [.], seq 52128:55024, ack 1, win 23, options [nop,nop,TS val 104227 ecr 2591672710], length 2896 ----- 2896 08:13:32.765222 IP agree-7-eth0.godzone.net.nz > agree-14.godzone.net.nz: ICMP mta-v1.mail.vip.sp2.yahoo.com unreachable - need to frag (mtu 1500), length 556 08:13:33.135985 IP agree-14.godzone.net.nz.53385 > mta-v1.mail.vip.sp2.yahoo.com.smtp: Flags [.], seq 52128:53576, ack 1, win 23, options [nop,nop,TS val 104594 ecr 2591672710], length 1448 08:13:33.371540 IP mta-v1.mail.vip.sp2.yahoo.com.smtp > agree-14.godzone.net.nz.53385: Flags [.], ack 53576, win 8326, options [nop,nop,TS val 2591673316 ecr 104594], length 0 08:13:33.371668 IP agree-14.godzone.net.nz.53385 > mta-v1.mail.vip.sp2.yahoo.com.smtp: Flags [.], seq 55024:57920, ack 1, win 23, options [nop,nop,TS val 104827 ecr 2591673316], length 2896 ----- 2896 08:13:33.371691 IP agree-7-eth0.godzone.net.nz > agree-14.godzone.net.nz: ICMP mta-v1.mail.vip.sp2.yahoo.com unreachable - need to frag (mtu 1500), length 556 08:13:33.750814 IP agree-14.godzone.net.nz.53385 > mta-v1.mail.vip.sp2.yahoo.com.smtp: Flags [.], seq 53576:55024, ack 1, win 23, options [nop,nop,TS val 105202 ecr 2591673316], length 1448 ----- retransmitted at 1448 ? 08:13:33.986548 IP mta-v1.mail.vip.sp2.yahoo.com.smtp > agree-14.godzone.net.nz.53385: Flags [.], ack 55024, win 8326, options [nop,nop,TS val 2591673931 ecr 105202], length 0 08:13:33.986673 IP agree-14.godzone.net.nz.53385 > mta-v1.mail.vip.sp2.yahoo.com.smtp: Flags [.], seq 55024:56472, ack 1, win 23, options [nop,nop,TS val 105435 ecr 2591673931], length 1448 08:13:33.986704 IP agree-14.godzone.net.nz.53385 > mta-v1.mail.vip.sp2.yahoo.com.smtp: Flags [.], seq 56472:57920, ack 1, win 23, options [nop,nop,TS val 105435 ecr 2591673931], length 1448 08:13:34.123382 IP mta-v1.mail.vip.sp2.yahoo.com.smtp > agree-14.godzone.net.nz.53385: Flags [.], ack 57920, win 8145, options [nop,nop,TS val 2591674067 ecr 105435], length 0 08:13:34.123525 IP agree-14.godzone.net.nz.53385 > mta-v1.mail.vip.sp2.yahoo.com.smtp: Flags [.], seq 57920:62264, ack 1, win 23, options [nop,nop,TS val 105570 ecr 2591674067], length 4344 ----- 4344 08:13:34.123550 IP agree-7-eth0.godzone.net.nz > agree-14.godzone.net.nz: ICMP mta-v1.mail.vip.sp2.yahoo.com unreachable - need to frag (mtu 1500), length 556 08:13:34.510432 IP agree-14.godzone.net.nz.53385 > mta-v1.mail.vip.sp2.yahoo.com.smtp: Flags [.], seq 57920:59368, ack 1, win 23, options [nop,nop,TS val 105953 ecr 2591674067], length 1448
Here is another example from another server. In between these two captures, I have rebooted our firewall so it is also now running the latest kernel, I have also attempted to lower the MTU on this server from 1500 to 1280 but that doesn't seem to make any difference either. 08:32:34.324960 ARP, Request who-has customers.godzone.net.nz tell agree-7-eth0.godzone.net.nz, length 28 08:32:34.325076 ARP, Reply customers.godzone.net.nz is-at 00:50:56:9e:00:00 (oui Unknown), length 46 08:32:34.325089 IP router-3-nat.godzone.net.nz.50930 > customers.godzone.net.nz.https: Flags [P.], seq 1064063234:1064063271, ack 1473778411, win 16430, length 37 08:32:34.325115 IP router-3-nat.godzone.net.nz.50930 > customers.godzone.net.nz.https: Flags [F.], seq 37, ack 1, win 16430, length 0 08:32:34.325235 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50930: Flags [R], seq 1473778411, win 0, length 0 08:32:34.325278 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50930: Flags [R], seq 1473778411, win 0, length 0 08:32:34.328081 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [S], seq 1264632018, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0 08:32:34.328160 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [S.], seq 206047862, ack 1264632019, win 4960, options [mss 1240,nop,nop,sackOK,nop,wscale 8], length 0 08:32:34.366236 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 1, win 16430, length 0 08:32:34.370369 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [P.], seq 1:210, ack 1, win 16430, length 209 08:32:34.370445 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], ack 210, win 24, length 0 08:32:34.385513 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 1:2481, ack 210, win 24, length 2480 ----- length 2480 - this is bizarre 08:32:34.385540 IP agree-7-eth0.godzone.net.nz > customers.godzone.net.nz: ICMP router-3-nat.godzone.net.nz unreachable - need to frag (mtu 1500), length 556 08:32:34.385566 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [P.], seq 2481:2866, ack 210, win 24, length 385 08:32:34.422248 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 1, win 16430, options [nop,nop,sack 1 {2481:2866}], length 0 08:32:37.391021 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 1:1241, ack 210, win 24, length 1240 08:32:37.430436 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 1241, win 16430, options [nop,nop,sack 1 {2481:2866}], length 0 08:32:37.430526 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 1241:2481, ack 210, win 24, length 1240 08:32:37.470067 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 2866, win 16430, length 0 08:32:37.478325 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [P.], seq 210:408, ack 2866, win 16430, length 198 08:32:37.482757 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [P.], seq 2866:3164, ack 408, win 28, length 298 08:32:37.528462 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [P.], seq 408:1101, ack 3164, win 16355, length 693 08:32:37.567966 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], ack 1101, win 34, length 0 08:32:37.722587 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 3164:6884, ack 1101, win 34, length 3720 ----- 3720 ??? 08:32:37.722620 IP agree-7-eth0.godzone.net.nz > customers.godzone.net.nz: ICMP router-3-nat.godzone.net.nz unreachable - need to frag (mtu 1500), length 556 08:32:37.968011 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 3164:4404, ack 1101, win 34, length 1240 08:32:38.210324 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 4404, win 16430, length 0 08:32:38.210457 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 6884:9364, ack 1101, win 34, length 2480 08:32:38.210478 IP agree-7-eth0.godzone.net.nz > customers.godzone.net.nz: ICMP router-3-nat.godzone.net.nz unreachable - need to frag (mtu 1500), length 556 08:32:38.703031 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 4404:5644, ack 1101, win 34, length 1240 08:32:38.944414 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 5644, win 16430, length 0 08:32:38.944528 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [P.], seq 9364:9940, ack 1101, win 34, length 576 08:32:38.982335 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 5644, win 16430, options [nop,nop,sack 1 {9364:9940}], length 0 08:32:38.982438 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 5644:6884, ack 1101, win 34, length 1240 08:32:38.982471 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 6884:8124, ack 1101, win 34, length 1240 08:32:38.982484 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 8124:9364, ack 1101, win 34, length 1240 08:32:39.020281 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 6884, win 16430, options [nop,nop,sack 1 {9364:9940}], length 0 08:32:39.022278 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 8124, win 16430, options [nop,nop,sack 1 {9364:9940}], length 0 08:32:39.024100 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 9940, win 16430, length 0 08:32:39.080514 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [P.], seq 1101:1842, ack 9940, win 16430, length 741 08:32:39.080627 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], ack 1842, win 39, length 0 08:32:39.099886 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 9940:13660, ack 1842, win 39, length 3720 ----- 3720 08:32:39.099910 IP agree-7-eth0.godzone.net.nz > customers.godzone.net.nz: ICMP router-3-nat.godzone.net.nz unreachable - need to frag (mtu 1500), length 556 08:32:40.085070 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 9940:11180, ack 1842, win 39, length 1240 08:32:40.334548 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 11180, win 16430, length 0 08:32:40.334676 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 13660:16140, ack 1842, win 39, length 2480 08:32:40.334697 IP agree-7-eth0.godzone.net.nz > customers.godzone.net.nz: ICMP router-3-nat.godzone.net.nz unreachable - need to frag (mtu 1500), length 556 08:32:42.303096 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 11180:12420, ack 1842, win 39, length 1240 08:32:42.538299 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 12420, win 16430, length 0 08:32:42.538408 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 16140:18620, ack 1842, win 39, length 2480 08:32:42.538429 IP agree-7-eth0.godzone.net.nz > customers.godzone.net.nz: ICMP router-3-nat.godzone.net.nz unreachable - need to frag (mtu 1500), length 556 08:32:46.487157 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 12420:13660, ack 1842, win 39, length 1240 08:32:46.742394 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 13660, win 16430, length 0 08:32:46.742522 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 18620:21100, ack 1842, win 39, length 2480 08:32:46.742539 IP agree-7-eth0.godzone.net.nz > customers.godzone.net.nz: ICMP router-3-nat.godzone.net.nz unreachable - need to frag (mtu 1500), length 556 08:32:54.615323 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 13660:14900, ack 1842, win 39, length 1240 08:32:54.866579 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 14900, win 16430, length 0 08:32:54.866693 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 21100:23580, ack 1842, win 39, length 2480 08:32:54.866713 IP agree-7-eth0.godzone.net.nz > customers.godzone.net.nz: ICMP router-3-nat.godzone.net.nz unreachable - need to frag (mtu 1500), length 556 08:33:10.086308 IP unknown.scnet.net.http > customers.godzone.net.nz.61685: Flags [R], seq 1559829096, win 0, length 0 08:33:10.615625 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 14900:16140, ack 1842, win 39, length 1240 08:33:10.867293 IP router-3-nat.godzone.net.nz.50935 > customers.godzone.net.nz.https: Flags [.], ack 16140, win 16430, length 0 08:33:10.867437 IP customers.godzone.net.nz.https > router-3-nat.godzone.net.nz.50935: Flags [.], seq 23580:26060, ack 1842, win 39, length 2480 08:33:10.867463 IP agree-7-eth0.godzone.net.nz > customers.godzone.net.nz: ICMP router-3-nat.godzone.net.nz unreachable - need to frag (mtu 1500), length 556 08:33:15.103963 ARP, Request who-has customers.godzone.net.nz tell agree-7-eth0.godzone.net.nz, length 28 08:33:15.104074 ARP, Reply customers.godzone.net.nz is-at 00:50:56:9e:00:00 (oui Unknown), length 46 ^C 66 packets captured 84 packets received by filter 0 packets dropped by kernel
Progress - 2.6.35.6-45.fc14.i686 works !! So that suggests something has been changed that has broken things. I am going to downgrade all the servers and will try to then work forward on one of them.
2.6.35-48 is probably OK, the postgres server hasn't been rebooted for quite a while and seems to be running fine.
Another possibility, downgrading the kernel has also resulted in uninstalling kmod-open-vm-tools, I don't know the interaction between the tools and the kernel but perhaps it isn't the kernel at all.
Are you looking at the packet dumps on the guest or on the host?
The first tcpdump was on the mail server and was filtering on pop3. The remainder were on the firewall server and were filtering on the host.
(In reply to comment #10) > The first tcpdump was on the mail server and was filtering on pop3. But was it taken from within the VM? In other words, was it taken from within a console session attached to the guest VM? You can try turning off some of the offload features in the network driver. 'ethtool -k eth0' will show which features are enabled and 'ethtool -K <feature> [on|off]' controls them.
Yes, it was taken from inside the VM, from an ssh session into the guest. We did find an old bug re: Xen and over sized packets so have already tried turning off tx-checksumming as seemed to be the fix in that case but it made no difference.
Having now seen the same behavior from a Win 2k3 server, it would seem that this is more likely an issue with VMware ESXi, consequently, we are going to upgrade the hosts to 4.1U1 and see if that makes any difference.
This is really becoming a case of 'hunt the snark'. I need to determine what is or isn't part of the equation. 1. Upgrading to ESXi 4.1U1 would appear to indicate the ESXi wasn't the issue. 2. Upgrading to 2.6.35.13-91.fc14.i686 on a number of the VMs hasn't eliminated the issue. 3. Changing from 'flexible' to 'vmxnet3' on some VMs would seem to not matter. The biggest indicator so far was rebooting the firewall this morning. With the firewall running 2.6.35.13-91.fc14.i686 and using the vmxnet3 nics/drivers, the issue was immediately visible from tcpdumps running on the firewall. Reverting back to 'flexible' for the NICs (pcnet32), and the problem appears to have gone away again. So, with this evidence, I am starting to think that the oversized packets seen from the Win2k3 server, via tcpdump on the firewall was not actually Win2k3 at all but the firewall. It would seem to me that the difference between pcnet32 and vmxnet3 drivers is the offload stuff. The bug I found regarding Xen and oversized packets was related to offloading on a VM that was forwarding traffic. Maybe this issue still exists ? Until tomorrow :-) when I try some more tests.
I think I have now gotten to the bottom of this. I don't really know if its a bug or not but it is the same issue experienced by another person as documented here http://communities.vmware.com/thread/281839 It would seem that combining segments before they go up the stack screws things up. What I was seeing was 2 inbound packets concatenated, the size then going from 1480 to 2960. The firewall would compare this with its own MTU and send back an ICMP frag message to the sending server saying to fragment at 1500. But the sender was already doing that. Result - broken communication path. Using the 'flexible' NIC which equates to pcnet32 in the kernel seems to work as expected as this driver does not support any offload capabilities.