Bug 477012

Earlier this year, I got a lot of network hangs when the RHEL5.3-ish kernels switched to xen_vnif. I was never able to reliably reproduce it and after working with Don Dutile, we eventually figured it was something that was resolved somewhere else in the kernel.

I've just recently switched my RHEL5 xen guest back to xen_vnif and the problem has returned. The kernel is 2.6.18-128.el5.jtltest.57debug, which is basically -128.el5 with some extra patches (mostly filesystem stuff).

This time, I have a little more info...

The symptoms are basically that ssh sessions will go dead -- they'll just stop responding. I've gotten the box into this state and here's what I see on the network. This is a single keypress on one console session:

629.482804 10.11.228.36 -> 10.11.231.179 SSH Encrypted request packet len=48
629.483148 10.11.231.179 -> 10.11.228.36 TCP ssh > 40845 [ACK] Seq=369 Ack=673 Win=104 Len=0 TSV=528999 TSER=621174332

...so we got an ACK here.

During this I can also log into the serial console on the box and was able to get some lsof and strace output from the ssh session that holds the socket. Relevant fd's from lsof:

sshd    2441 root    3u  IPv6               8296            TCP dhcp231-179.rdu.redhat.com:ssh->barsoom.rdu.redhat.com:40844 (ESTABLISHED)
sshd    2441 root    4u  unix 0xffff81000acbd448           8527 socket
sshd    2441 root    5r  FIFO                0,6           8551 pipe
sshd    2441 root    6w  FIFO                0,6           8551 pipe
sshd    2441 root    7u   CHR                5,2            716 /dev/ptmx
sshd    2441 root    8u   CHR                5,2            716 /dev/ptmx
sshd    2441 root    9u   CHR                5,2            716 /dev/ptmx

...here's the strace from that single keypress:

2441  select(9, [3 5 8], [], NULL, NULL) = 1 (in [3])
2441  rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
2441  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
2441  read(3, "U\374\320\27\350\257\337\7\r\376(\377\257\31\337\371\233\t\20\26\261N\346j\306\244MO\213\262'\311K"..., 16384) = 48

...encrypted data comes into socket.

2441  select(9, [3 5 8], [7], NULL, NULL) = 1 (out [7])
2441  rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
2441  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
2441  write(7, "\r"..., 1)              = 1

...data decrypted -- sent to pty.

2441  select(9, [3 5 8], [], NULL, NULL) = 1 (in [8])
2441  rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
2441  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
2441  read(8, "\r\n"..., 16384)         = 2

...full-duplex echo of the character to send back to client (with extra cr on end).

2441  select(9, [3 5 8], [3], NULL, NULL) = 1 (out [3])
2441  rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
2441  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
2441  write(3, "\22z\302\213J]\343.,\336\233|(D\245`\23D\371\367\345\336\20$LoW\26\t.\250\255\305"..., 48) = 48

...encrypted data sent on socket.

2441  select(9, [3 5 8], [], NULL, NULL) = 1 (in [8])
2441  rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
2441  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
2441  read(8, "\33]0;root@dhcp231-179:~\7"..., 16384) = 23
2441  select(9, [3 5 8], [3], NULL, NULL) = 2 (in [8], out [3])
2441  rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
2441  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
2441  read(8, "[root@dhcp231-179 ~]# "..., 16384) = 22
2441  write(3, "\352I8\302\335\262\274\355J\324N\375\221\17\232\277\21\211|n^&Z\244}\3\3245\342\34\"+\304"..., 64) = 64
2441  select(9, [3 5 8], [3], NULL, NULL) = 1 (out [3])
2441  rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
2441  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
2441  write(3, "C\3\356\351\22m\224\256B?\33q\344\313\327\263Pp\251\246)\0\22o\202\271\247\372\236\245>\222\264"..., 64) = 64
2441  select(9, [3 5 8], [], NULL, NULL <unfinished ...>

...next shell prompt displayed and sent on socket.

The client never receives any of the data sent by the server here. From this it looks like an occasional problem sending packets on a particular socket. The curious thing is that new sessions can be started and will generally work for a while. Once they're hung though, they stay that way.

I can try to reproduce again and collect more info if you can think of anything that would be helpful.