669453 – intermittent unresponsiveness in ssh session

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 669453 - intermittent unresponsiveness in ssh session

Summary: intermittent unresponsiveness in ssh session

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	openssh
Sub Component:
Version:	6.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	low
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Red Hat Kernel Manager
QA Contact:	Red Hat Kernel QE team
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2011-01-13 18:03 UTC by Aram Agajanian
Modified:	2011-09-20 08:22 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2011-09-20 08:22:25 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
Dmesg from the affected machine (14.40 KB, application/x-gzip) 2011-07-13 18:01 UTC, Dmitry S. Makovey	no flags	Details
View All

Description Aram Agajanian 2011-01-13 18:03:12 UTC

Description of problem:
RHEL 6.0 is running on a Dell PowerEdge T710 server.

When logging into the server with ssh, sometimes it becomes unresponsive.  When typing text into the terminal, you have to wait several seconds before the characters appear after you press on the keys of the keyboard.  

Also, the top command doesn't update as often as it should.  Further, while the problem is going on, I observe something about the CPU usage.  I press the '1' key to show usage of individual CPUs.  While the probem is going on, one or more of the CPUs have a large usage in the sy column.

The problem seems to last for several minutes and then goes away for a while.


Version-Release number of selected component (if applicable):
2.6.32-71.14.1.el6.x86_64


How reproducible:
happens regularly but I don't know how to make it happen


Steps to Reproduce:
1.  Boot server with RHEL 6
2.  work on the terminal for a short while until you notice unresponsiveness

  
Actual results:
server is unresponsive


Expected results:
server should not be unresponsive

Comment 2 Aram Agajanian 2011-01-13 22:00:54 UTC

I see some unresponsiveness in the host and in one guest.  These both use RHEL 6.  

I do not notice the problem in the other guest which uses RHEL 5.

Comment 3 Aram Agajanian 2011-01-17 22:38:34 UTC

When the screen doesn't update, it seems that pressing a key can flush out undisplayed characters.

Comment 4 RHEL Program Management 2011-04-04 01:53:15 UTC

Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 5 Aram Agajanian 2011-05-02 20:28:13 UTC

I have been running a server that exhibits this bug for a few months now.  It can be annoying but the server has been stable at least.

I also see some issues with web server connections.  Sometimes you have to reload the page to get it to appear.

My best guess is that this problem is due to issues with the newer chipset in the server.  I am waiting to see if it is resolved in 6.1.

Comment 6 Aram Agajanian 2011-06-07 14:41:23 UTC

This problem persists after updating the host OS and one guest OS to version 6.1.

ssh sessions to the host OS are still sometimes unresponsive.  The unexpected CPU usage is still occurring. 

The responsiveness of ssh sessions to the RHEL 6.1 OS guest may have improved since upgrading the host OS to version 6.1.  The web server issues seem to have gone away.

The RHEL 5 guest still seems OK.

Comment 7 Aram Agajanian 2011-06-07 16:42:40 UTC

I found some discussion about similar issues in the mailing list thread at the following URL:

http://www.redhat.com/archives/redhat-sysadmin-list/2011-May/msg00001.html

Comment 8 Dmitry S. Makovey 2011-06-07 22:04:12 UTC

I can confirm that we are having that very same issue. Can't say about HTTP performance but definitely SSH is the one that is an issue for us. I didn't notice the high system usage of CPU, neither was network load high enough to justify such behaviour. Interestingly enough guest OS'es (mix of RHEL5 and RHEL6) perform just fine with no problems.

The box we're having issues with is custom-build Intel server (according to Intel specs). If it's of any interest I can post specs here.

Comment 9 Miroslav Vadkerti 2011-07-13 07:22:18 UTC

Dmitry please give us the specs so we can try to reproduce the problem with a similar HW on our side. Without a way to reproduce we will have hard times to finding the root cause of this problem. We use SSH a lot on RHEL6 only systems and didn't see any such behavior. Thanks!

Comment 10 Miroslav Vadkerti 2011-07-13 07:23:08 UTC

Dmitry can you also confirm that you are running on RHEL6.1?

Comment 11 Dmitry S. Makovey 2011-07-13 18:00:27 UTC

Re: Specs
Would you like me to post just HW specs or do you have some /proc and /sys entries in mind that you would like to see?

Rough specs (until I get to my office):

Dual Intel Xeon E5520

vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
stepping        : 5
cpu MHz         : 2261.210
cache size      : 8192 KB
physical id     : 1
siblings        : 8
core id         : 3
cpu cores       : 4
apicid          : 23
initial apicid  : 23
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips        : 4521.79
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

# lspci 
00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
00:0a.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13)
00:10.0 PIC: Intel Corporation 5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 13)
00:10.1 PIC: Intel Corporation 5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 13)
00:11.0 PIC: Intel Corporation 5520/5500 Physical and Link Layer Registers Port 1 (rev 13)
00:11.1 PIC: Intel Corporation 5520/5500 Routing & Protocol Layer Register Port 1 (rev 13)
00:13.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 13)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 13)
00:15.0 PIC: Intel Corporation 5520/5500/X58 Trusted Execution Technology Registers (rev 13)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5
00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
01:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
02:00.0 PCI bridge: Intel Corporation 80333 Segment-A PCI Express-to-PCI Express Bridge
02:00.2 PCI bridge: Intel Corporation 80333 Segment-B PCI Express-to-PCI Express Bridge
03:0e.0 RAID bus controller: Adaptec AAC-RAID
05:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
06:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0a:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0a:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0e:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200e [Pilot] ServerEngines (SEP1) (rev 02)

I am also attaching dmesg from that machine.

Re: RHEL 6.1
Yes, we have bumped this machine:
$ rpm -q redhat-release-server-6Server
redhat-release-server-6Server-6.1.0.2.el6.x86_64

Comment 12 Dmitry S. Makovey 2011-07-13 18:01:17 UTC

Created attachment 512715 [details]
Dmesg from the affected machine

Comment 13 Aram Agajanian 2011-07-13 21:01:35 UTC

The Dell PowerEdge R710 has an Intel 5520 chipset.  The output of lspci is similar to Comment #11.

The CPU in my server is the Xeon L5540.

Comment 14 Dmitry S. Makovey 2011-07-27 20:43:06 UTC

Sorry about delay - finally got the quote for the machine with all the serial numbers etc. Hope that helps.

SR2612UR     Intel SERVER SR2612UR NO CPU 0GHZ server system
BX80602E5520 Intel XEON E5520 2.26Ghz CPU's
Kingston 12Gb memory kit's  KVR1333D3D4R9SK3/12GI  
ST31000640SS Segate 1TB SAS drives
ST3146356SS Seagate 146Gb 15K SAS harddrives
AXX4GBIOMOD2 Intel 4-port NIC module
AXXSATADVDRWROM Intel Slimline optical drive    AXXSATADVDRWROM  
2251800-R Adaptec 3405 RAID controller card
2248000-R Adaptec ABM-800 battery pack
2246800-R Adaptec SFF-8087 to SFF-8087 cable
QLA2460-CK Qlogic Fiber Channel 4Gb PCI-x HBA's

Comment 15 Aram Agajanian 2011-08-11 14:22:03 UTC

I have found the bug report at the following URL:

https://bugzilla.redhat.com/show_bug.cgi?id=710265

Adding "intel_idle.max_cstate=0" to the kernel command line seems to have somewhat improved the responsiveness of the host OS.

I'm not sure if RHEL 6.1 ssh sessions are sometimes not as responsive as the RHEL 5.6 ssh sessions.  I'll keep monitoring it.

Comment 16 Aram Agajanian 2011-08-15 17:29:34 UTC

The "intel_idle.max_cstate=0" doesn't seem to have improved the responsiveness of ssh sessions.

It may have quieted down the CPU usage described in the initial comment.

Comment 17 Aram Agajanian 2011-09-16 13:07:18 UTC

As per a recent comment in Bug #710265, I tried booting with the following kernel arguments:

intel_idle.max_cstate=0 processor.max_cstate=1

The problem with ssh responsiveness seems to have gone away with these kernel arguments.

Comment 18 Miroslav Vadkerti 2011-09-19 09:13:47 UTC

Looks like this is not an SSH issue at all. Jan, what do you think?

Comment 19 Jan F. Chadima 2011-09-20 06:02:30 UTC

(In reply to comment #18)
> Looks like this is not an SSH issue at all. Jan, what do you think?

I agree, nor sshd nor ssh can not influence the top program.

Comment 20 Miroslav Vadkerti 2011-09-20 08:22:25 UTC

CLOSING as not an openssh bug. Please reopen and assign to other component if needed or open a new bug. Thanks!

Note You need to log in before you can comment on or make changes to this bug.