Red Hat Bugzilla – Bug 669453
intermittent unresponsiveness in ssh session
Last modified: 2011-09-20 04:22:25 EDT
Description of problem: RHEL 6.0 is running on a Dell PowerEdge T710 server. When logging into the server with ssh, sometimes it becomes unresponsive. When typing text into the terminal, you have to wait several seconds before the characters appear after you press on the keys of the keyboard. Also, the top command doesn't update as often as it should. Further, while the problem is going on, I observe something about the CPU usage. I press the '1' key to show usage of individual CPUs. While the probem is going on, one or more of the CPUs have a large usage in the sy column. The problem seems to last for several minutes and then goes away for a while. Version-Release number of selected component (if applicable): 2.6.32-71.14.1.el6.x86_64 How reproducible: happens regularly but I don't know how to make it happen Steps to Reproduce: 1. Boot server with RHEL 6 2. work on the terminal for a short while until you notice unresponsiveness Actual results: server is unresponsive Expected results: server should not be unresponsive
I see some unresponsiveness in the host and in one guest. These both use RHEL 6. I do not notice the problem in the other guest which uses RHEL 5.
When the screen doesn't update, it seems that pressing a key can flush out undisplayed characters.
Since RHEL 6.1 External Beta has begun, and this bug remains unresolved, it has been rejected as it is not proposed as exception or blocker. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux.
I have been running a server that exhibits this bug for a few months now. It can be annoying but the server has been stable at least. I also see some issues with web server connections. Sometimes you have to reload the page to get it to appear. My best guess is that this problem is due to issues with the newer chipset in the server. I am waiting to see if it is resolved in 6.1.
This problem persists after updating the host OS and one guest OS to version 6.1. ssh sessions to the host OS are still sometimes unresponsive. The unexpected CPU usage is still occurring. The responsiveness of ssh sessions to the RHEL 6.1 OS guest may have improved since upgrading the host OS to version 6.1. The web server issues seem to have gone away. The RHEL 5 guest still seems OK.
I found some discussion about similar issues in the mailing list thread at the following URL: http://www.redhat.com/archives/redhat-sysadmin-list/2011-May/msg00001.html
I can confirm that we are having that very same issue. Can't say about HTTP performance but definitely SSH is the one that is an issue for us. I didn't notice the high system usage of CPU, neither was network load high enough to justify such behaviour. Interestingly enough guest OS'es (mix of RHEL5 and RHEL6) perform just fine with no problems. The box we're having issues with is custom-build Intel server (according to Intel specs). If it's of any interest I can post specs here.
Dmitry please give us the specs so we can try to reproduce the problem with a similar HW on our side. Without a way to reproduce we will have hard times to finding the root cause of this problem. We use SSH a lot on RHEL6 only systems and didn't see any such behavior. Thanks!
Dmitry can you also confirm that you are running on RHEL6.1?
Re: Specs Would you like me to post just HW specs or do you have some /proc and /sys entries in mind that you would like to see? Rough specs (until I get to my office): Dual Intel Xeon E5520 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz stepping : 5 cpu MHz : 2261.210 cache size : 8192 KB physical id : 1 siblings : 8 core id : 3 cpu cores : 4 apicid : 23 initial apicid : 23 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid bogomips : 4521.79 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: # lspci 00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13) 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13) 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13) 00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 13) 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13) 00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13) 00:0a.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13) 00:10.0 PIC: Intel Corporation 5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 13) 00:10.1 PIC: Intel Corporation 5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 13) 00:11.0 PIC: Intel Corporation 5520/5500 Physical and Link Layer Registers Port 1 (rev 13) 00:11.1 PIC: Intel Corporation 5520/5500 Routing & Protocol Layer Register Port 1 (rev 13) 00:13.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 13) 00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13) 00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13) 00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13) 00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 13) 00:15.0 PIC: Intel Corporation 5520/5500/X58 Trusted Execution Technology Registers (rev 13) 00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5 00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6 00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) 00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller 00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1 00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller 00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2 01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02) 01:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02) 02:00.0 PCI bridge: Intel Corporation 80333 Segment-A PCI Express-to-PCI Express Bridge 02:00.2 PCI bridge: Intel Corporation 80333 Segment-B PCI Express-to-PCI Express Bridge 03:0e.0 RAID bus controller: Adaptec AAC-RAID 05:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03) 06:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03) 07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0a:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0a:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0e:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200e [Pilot] ServerEngines (SEP1) (rev 02) I am also attaching dmesg from that machine. Re: RHEL 6.1 Yes, we have bumped this machine: $ rpm -q redhat-release-server-6Server redhat-release-server-6Server-6.1.0.2.el6.x86_64
Created attachment 512715 [details] Dmesg from the affected machine
The Dell PowerEdge R710 has an Intel 5520 chipset. The output of lspci is similar to Comment #11. The CPU in my server is the Xeon L5540.
Sorry about delay - finally got the quote for the machine with all the serial numbers etc. Hope that helps. SR2612UR Intel SERVER SR2612UR NO CPU 0GHZ server system BX80602E5520 Intel XEON E5520 2.26Ghz CPU's Kingston 12Gb memory kit's KVR1333D3D4R9SK3/12GI ST31000640SS Segate 1TB SAS drives ST3146356SS Seagate 146Gb 15K SAS harddrives AXX4GBIOMOD2 Intel 4-port NIC module AXXSATADVDRWROM Intel Slimline optical drive AXXSATADVDRWROM 2251800-R Adaptec 3405 RAID controller card 2248000-R Adaptec ABM-800 battery pack 2246800-R Adaptec SFF-8087 to SFF-8087 cable QLA2460-CK Qlogic Fiber Channel 4Gb PCI-x HBA's
I have found the bug report at the following URL: https://bugzilla.redhat.com/show_bug.cgi?id=710265 Adding "intel_idle.max_cstate=0" to the kernel command line seems to have somewhat improved the responsiveness of the host OS. I'm not sure if RHEL 6.1 ssh sessions are sometimes not as responsive as the RHEL 5.6 ssh sessions. I'll keep monitoring it.
The "intel_idle.max_cstate=0" doesn't seem to have improved the responsiveness of ssh sessions. It may have quieted down the CPU usage described in the initial comment.
As per a recent comment in Bug #710265, I tried booting with the following kernel arguments: intel_idle.max_cstate=0 processor.max_cstate=1 The problem with ssh responsiveness seems to have gone away with these kernel arguments.
Looks like this is not an SSH issue at all. Jan, what do you think?
(In reply to comment #18) > Looks like this is not an SSH issue at all. Jan, what do you think? I agree, nor sshd nor ssh can not influence the top program.
CLOSING as not an openssh bug. Please reopen and assign to other component if needed or open a new bug. Thanks!