Bug 439043
Summary: | Swap Token issue with RHEL4 | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 4 | Reporter: | Tomen Tse <ttse> |
Component: | kernel | Assignee: | Michal Schmidt <mschmidt> |
Status: | CLOSED ERRATA | QA Contact: | Martin Jenner <mjenner> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 4.8 | CC: | jtluka, lwoodman, mtucker, riek, rlerch |
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: |
With this update, the "swap_token_timeout" parameter has been added to /proc/sys/vm.
This file contains valid hold time of swap out protection token. The Linux Virtual Memory (VM) subsystem has a token based thrashing control mechanism and uses the token to prevent unnecessary page faults in thrashing situation. The unit of the value is in `second`. The value would be useful to tune thrashing behavior. Setting it to 0 will disable the swap token mechanism.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2009-05-18 19:21:48 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 458752, 461297 | ||
Attachments: |
Description
Tomen Tse
2008-03-26 17:26:02 UTC
Created attachment 299501 [details]
RHEL4's 2.6.9-67.0.1.ELsmp demonstrates the dramatic dropoff in query throughput when the kernel reports nearly all the physical pages as being "active"
Created attachment 299502 [details]
This graph demonstrates the effect of changing the kernel so it will not respect the Swap Token and will consider reclaiming pages from the process that holds the token
Created attachment 299503 [details]
This graph demonstrates the effect of running the same previous experiment on RHEL5
An ISV partner is experiencing a severe performance issue on RHEL4 when using server product with our customers' datasets that are larger than physical memory. This ISV partner develops and sells an Information Access Platform that includes a server component that allows our customers to answer queries that involve wsearch and navigation. The issue that we have encountered will impact all customers using RHEL4 with datasets that don't fit into the available physical memory. As part of our internal performance testing using customer datasets, we have discovered serious issues with the way that the Swap Token is handled in RHEL4 and therefore the way in which memory is reclaimed. In particular, we have found that when our server process is using a majority of the memory on a machine the kernel does not properly reclaim pages from our process to make room for future allocations or page-ins. The graphs that I have attached show the extreme negative impact that the swap token behavior is having on the performance of our server, as measured in queries per second. These graphs also demonstrate that the problem is mostly alleviated on RHEL5. From our analysis and comparison of the 2.6.9 and 2.6.18 kernel source and as confirmed by Red Hat's engineers, the issue was addressed in RHEL5 through changes to how the Swap Token is managed as well as how it can be controlled through Virtual Memory configuration options. We have done extensive research into this problem in an effort to find an acceptable workaround that could be implemented in our code or that involved RHEL4 configuration options that we could recommend to our customers. In short, we have not found a reasonable answer. It appears that the only viable option for our customers today is to consider running a different kernel. In particular, please consider the three attached graphs that chart the throughput (measured in queries answered per second) as well as various Virtual Memory statistics as gathered from /proc/meminfo and vmstat. The three graphs show the same version of the ISV partner's software being run on 3 different kernel versions (2.6.9-67.0.1.ELsmp with and without a custom patch, as well as 2.6.18-8.el5). The first graph, RHEL4's 2.6.9-67.0.1.ELsmp demonstrates the dramatic dropoff in query throughput when the kernel reports nearly all the physical pages as being "active". The second graph demonstrates the effect of changing the kernel so it will not respect the Swap Token and will consider reclaiming pages from the process that holds the token. The last graph demonstrates the effect of running the same experiment on RHEL5 - as you can see, the changes that have been made to the kernel greatly alleviate this problem. More importantly, however, RHEL5 introduced new configuration options that permit our customers to avoid this problem altogether by effectively disabling the Swap Token on their system if they know that our server component will be the primary process running on the machine. Our engineers found these results particularly surprising because most of them consider Linux to be their primary Operating System and did not expect that our scaling characteristics would be significantly worse on a RHEL platform as compared to other operating systems on the same hardware. As currently released, RHEL4 does not behave reasonably for large-scale processes that dominate physical memory and moreover is not configurable in a way that allows our customers to get good performance out of their system as they scale. We believe that this is a critical issue for making our customers successful and will also positively effect many other RHEL4 users based on the discussions and posts that we have seen on kernel discussion lists Thanks for the detailed description. Granting devel ACK. This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. Updating PM score. Created attachment 315760 [details] RHEL4 patch This patch adds /proc/sys/vm/swap_token_timeout to RHEL4. The default is 300 s. The value 0 disables the swap token. It's similar to the two upstream patches mentioned, but does not change the default behavior. A scratch Brew build is here: https://brewweb.devel.redhat.com/taskinfo?taskID=1456231 Could you test it? Here is the response from Mike Tucker of the ISV partner who is having this issue: "Thanks for your work on this. We haven't had a chance to verify this yet as we are in the middle of a very busy period for our Dev team and have been using a work-around to date. I will let you know as soon as we have results that verify your change." Created attachment 327101 [details] RHEL4: add /proc/sys/vm/swap_token_timeout This patch was posted to rhkernel-list. Brew scratch build: https://brewweb.devel.redhat.com/taskinfo?taskID=1615076 Committed in 78.23.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/ Release note added. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: RHEL4.8 adds the "swap_token_timeout" parameter in /proc/sys/vm. This file contains valid hold time of swap out protection token. The Linux VM has token based thrashing control mechanism and uses the token to prevent unnecessary page faults in thrashing situation. The unit of the value is in `second`. The value would be useful to tune thrashing behavior. Hi - I noticed there was no note on here about verification. In fact, we took the build from Michal and determined that the changes resolved our issue. Thanks for your help with this. Release note updated. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -3,4 +3,4 @@ This file contains valid hold time of swap out protection token. The Linux VM has token based thrashing control mechanism and uses the token to prevent unnecessary page faults in thrashing situation. The unit of the value is in -`second`. The value would be useful to tune thrashing behavior.+`second`. The value would be useful to tune thrashing behavior. Setting it to 0 will disable the swap token mechanism. Release note updated. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,6 +1,3 @@ -RHEL4.8 adds the "swap_token_timeout" parameter in /proc/sys/vm. +With this update, the "swap_token_timeout" parameter has been added to /proc/sys/vm. -This file contains valid hold time of swap out protection token. The Linux +This file contains valid hold time of swap out protection token. The Linux Virtual Memory (VM) subsystem has a token based thrashing control mechanism and uses the token to prevent unnecessary page faults in thrashing situation. The unit of the value is in `second`. The value would be useful to tune thrashing behavior. Setting it to 0 will disable the swap token mechanism.-VM has token based thrashing control mechanism and uses the token to prevent -unnecessary page faults in thrashing situation. The unit of the value is in -`second`. The value would be useful to tune thrashing behavior. Setting it to 0 will disable the swap token mechanism. Patch is in -88.EL. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2009-1024.html The link in the previous comment doesn't work. Can you fix the link and/or post its contents in the bug report? Michael, the link works for me. Can you retry? Maybe there was a temporary glitch. |