Bug 623232 - Kernel prone to swapping out pages unnecessarily
Summary: Kernel prone to swapping out pages unnecessarily
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 13
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-08-11 16:36 UTC by Will Newton
Modified: 2011-06-28 14:43 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-06-28 14:43:34 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Will Newton 2010-08-11 16:36:28 UTC
Description of problem:

The system seems prone to large bouts of swapping even when a resonable amount of free memory is available. For example, below is the results of running vmstat 1 at the same time as untarring a gcc release tarball.

Notice how there is a mild amount of swap activity until I run the command, then suddenly I get huge amounts of swapouts even though I have half a gig of free RAM. At this point the system has become unresponsive, the mouse cursor moves erratically and the output of tar has stalled.

A bit later I kill the tar job and swapping in resumes. The system is in such a state that all applications feel laggy and performance is very poor. Firefox reports unresponsive scripts, and switching tabs in konsole takes 10s of seconds. Swapping in continues for a minute or two after the tar job has been killed.

The system has 4G RAM, a 2-way 3GHz Core 2 Duo and a 7200 RPM disk which according to bonnie++ gets about 80Mb/s read and write.

I have a 1Gb VirtualBox VM running.

r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  2 529204 575992   6456 107924    6    5    75    47    5    5 12  3 83  2  0
 0  1 528880 575932   6464 107624  184    0   184    60 1288 2243  4  2 77 16  0
 0  1 528796 575868   6464 107636   40    0    40     0 1156 1864  1  2 93  3  0
 0  1 527708 574904   6464 107960  472    0   828     0 1376 2310  2  3 49 46  0
 0  1 527708 574504   6464 107960  572    0   572     0 1238 1922  2  1 49 47  0
 0  1 527708 573884   6464 107956  612    0   612    36 1212 1941  1  2 49 48  0
 0  2 527708 573140   6468 107952  760    0   760    40 1257 1955  2  2 49 47  0
 0  1 527284 577788   6472 107964  540    0   540     4 1612 2044 11  7 46 35  0
 1  0 527008 577900   6472 108000  136    0   136     0 1411 2183  5  6 84  5  0
 0  0 526748 577852   6472 107972    8    0     8     0 1147 2110  3  3 93  1  0
 1  0 526748 577852   6472 107972    0    0     0    64 1098 1572  2  3 95  0  0
 0  0 526732 579092   6472 107988    0    0     0     0 1090 1572  2  2 96  0  0
 1  0 526652 579092   6480 107968   72    0    72    20 1207 1737  1  3 91  4  0
 1  1 526556 578968   6748 108008   20    0   288     0 1103 2057  2  3 92  2  0
 0  1 526552 578968   6752 107964    4    0     8     0 1217 1805  2  3 94  1  0
 0  0 526348 578720   6884 107972  160    0   292     4 1202 1853  2  2 82 14  0
 1  0 526348 578720   6884 107972    0    0     0     0 1199 1821  1  2 96  0  0
 0  0 526348 578656   6884 107972    0    0     0     0 1148 1662  2  2 96  0  0
 0  0 526348 578656   6884 107972    0    0     0     0 1179 1662  2  3 95  0  0
 0  0 526348 578720   6892 107968    0    0     0    12 1150 1824  2  2 94  1  0
 0  0 526348 578720   6892 107972    0    0     0     0 1126 1793  2  2 96  0  0
 0  0 526268 578596   6896 107972   64    0    68     0  963 1640  1  1 92  5  0
 0  0 526252 578844   7028 107972    0    0   132     0 1235 1675  7  3 89  1  0
 2  0 526200 566056   7624 115716    0    0  3072     0 1599 4452 14  4 74  8  0
 3  0 525728 529428   7956 151100  192    0  5056     0 2427 14430 57 16 22  5  0
 3  0 525384 496488   9840 178048   12    0   924  6604 2372 7489 33 42 22  2  0
 1  0 525384 459232  11380 208328    0    0  1548 33048 2417 7013 37 32 25  7  0
 4  0 525384 577608   7104  99340    0    0  1284 32360 2722 7974 30 48 19  3  0
 0 14 544652 711612   6720  85580  620 19788  5912 20116 2743 2553  1 14  0 86  0
 0 15 544596 706404   6768  90308  520    0  2892     0 2420 2934  0  0  0 100  0
 0 10 544804 705164   6624  91796  392  272  1924   272 2262 3022  1  1  0 99  0
 0  9 544740 703180   6628  93424  308    0  1892     4 2256 2945  1  1  0 99  0
 0 10 544576 700916   6760  95584  476    0  2752     0 2257 2884  0  1  0 99  0
 2  9 544404 699552   6896  96584  300    0  1416     0 2275 3617  4  5  0 92  0
 0  9 544084 697848   7040  97364  564    0  1608     0 1591 2241  3  1  0 96  0
 0  6 544060 697504   7036  98652  704  172  1896   172 1589 1964  6  4  0 90  0
 1  5 543832 695864   7036  99812  956    0  2172     0 1824 2215  4  5  0 90  0
 0  4 543676 694936   7052 100332  464    0  1124  5808 1484 1940  4  4  0 91  0
 0  7 546392 700420   7216 102516  480 3076   756 21696 2623 3527  6 11  0 82  0
 0  5 547620 699400   7252 103928  660 1368  1536  1516 2056 2571  5  7  0 88  0
 0  6 547484 698052   7252 104548  736    0  1520     0 1811 2198  6  5  0 89  0
 0  3 547356 696952   7252 104628 1032    0  1044     0 1636 2699  5  5  0 90  0
 0 16 552776 740088   6780  71872  564 5556  2840 32116 3164 2705  2  8  0 90  0
 1  8 551896 736372   6784  74932  428    0  2536     0 1152 1420  3  2  0 95  0
 1 10 550312 734280   6784  76236  796    0  2112     0  710 1033  1  0  0 99  0
 1  6 550136 730624   6984  79388  976    0  1320     0 1547 2675  3  7  0 91  0
 0  4 550024 715296   7796  92408  936    0  1924     0 2097 4669 18 20  0 61  0
 0  5 549900 713804   7796  92984 1112    0  1568     0 1804 2288  4  6  0 90  0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  4 549984 713432   7604  93200  684  200   864   200 1788 2222  6  5  0 89  0
 0  6 550408 712644   7392  94232  696  488  1072  5016 1723 2452  6  6  0 87  0
 3  6 553036 716012   6752  91408 1340 2760  1724  6520 1698 1890  3 13  0 83  0
 1  5 559548 733464   6752  83716  468 6528   600  6812 2315 2680  4  9  0 87  0
 1  6 559736 723284   6636  93164 1548  412  3196  2692 1918 5119 16  8  0 76  0
 1  4 560276 702256   7000 112976 1040  628  3632  5524 2523 8048 32 13  0 54  0
 0  3 561020 684264   7212 130552  792  848  3312  1732 2310 6549 31 11  4 54  0
 2  2 567228 715884   5984 109568  232 6304  1612  7292 2716 5327 19 12  3 66  0
 2  5 571204 711248   5724 120200  480 4092  3848  4284 2810 7582 31 12  2 55  0
 1  6 572192 707056   5544 126488  744 1200  4568  1208 2546 5225 11  7  2 80  0
 2  8 572084 683960   5640 148112  592    0  4780    28 1741 7415 23  4  0 73  0
 0  8 572576 679868   5748 150824 1080  604  2100  1872 1633 2667  7  4  0 89  0
 0  9 573364 678540   5876 151580 1128  964  2260  5812 1573 2194  3  5  0 92  0
 0 10 574880 683092   5840 149564  868 1592   976  1592 1947 2161  6  7  0 88  0
 0  7 578300 687428   5840 149732  604 3704   784  3864 1972 1873  3  6  0 91  0
 1  5 579056 687596   5892 150148 1048  852  1416  2792 1659 2040  4  6  0 90  0
 0  5 580236 687424   5928 150660 1080 1360  1724  3288 2049 3243  5  8  0 87  0
 1  7 581240 687572   6004 150932 1100 1172  1436  4916 1992 2482  5  8  0 87  0
 0  6 587364 720548   5524 137248  236 6324   784  6380 2447 2843  5 11  0 84  0
 0  6 589220 728252   5296 130972  920 2392  2896  3808 1950 1905  4  8  0 88  0
 1  6 589384 727068   5292 131964  896  380  2036   384 1836 2294  5  8  0 88  0
 0  9 589972 733268   5148 126420 1572  804  2400  4808 2194 2680  4  7  2 87  0
 0  5 589816 731392   5148 127184 1324    0  1980     0 1445 1885  3  4  0 92  0
 0  9 590716 732664   5136 126692 5092 6836  6140  9440 6662 7947  5  8  0 87  0
 1  5 595512 745048   5120 129016 1608  292  2972   812 1695 2949  9  6  0 85  0
 0  6 595916 745044   5064 128276 1260  752  3204  2108 2043 3546 11 10  0 79  0
 0  6 595856 743976   5060 129112 1288  184  2004   188 1671 2129  4  5  0 90  0
 0  3 596048 742196   5140 129168 2224  548  2368  2400 1792 2179  5  5  0 90  0
 0  7 597652 744968   5160 128120 2136 1956  2208  2360 1984 2580  6  6  0 87  0
 1  7 601128 750380   5160 127984  208 3492   208  3492 1124 1104  1  3  0 95  0
 0  7 601920 749728   5096 131244 2268 1516  3544  1548 2313 3721 11  9  6 74  0
 0  6 601552 731516   5360 146688 2068    0  4016     8 2176 7161 24  8 16 52  0
 0  8 602408 730824   5376 146504 1500 1136  1524  9800 1948 2585  5  6  0 88  0
 0  7 607056 741812   5368 146100  572 4760   692  4776 1654 1775  2  8  0 90  0
 1  4 607044 736736   5476 150400 1672  520  2884 12016 2095 4450 14  9  0 77  0
 1  6 606816 739696   5504 143496 2864  340  3632 21364 2563 4097  9  9  0 81  0
 1  5 606680 733700   5620 147508 1632  128  2248   976 1848 3147 10  8  0 82  0
 0  6 606500 731712   5620 147816 1948  128  2140   128 1967 2441  9  6  0 85  0
 1  6 606360 729848   5624 148324 1668  136  2168  3808 1860 2254  7  7  0 86  0
 1  8 606016 727120   5624 148608 2800    0  3084     0 1918 2304  5  5  0 90  0
 0  7 605868 727616   5596 148192 2368  144  2788   864 1717 2043  5  5  0 90  0
 1  5 605712 726004   5608 148440 1568    0  1836    32 1779 2171  5  5  0 89  0
 0  4 605512 724392   5608 148716 1748    0  2016   320 1875 2243  6  5  0 89  0
 0  8 606416 732948   5668 139592 1756 1068  2260  4112 2102 2461  6  7 17 70  0
 1  8 607612 759360   5668 113916   48 1212    48  1212 1469 1719  4  5  0 91  0
 0  5 609516 770148   5544 110224  936 2436  1540  2564 2259 3274  6  9  0 85  0
 0 13 609532 771884   5480 107812 1436  372  2044  3924 1871 2294  5  8  0 86  0
 0 20 615660 783912   5472 107944  500 6304   708  7840 2026 2072  4 10  0 86  0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0 10 616312 784772   5464 108076 1232  872  1352   984 1745 2001  4  6  0 90  0
 1  8 619420 787508   5552 107240 1904 3316  2304  8108 1987 2171  5  7  0 88  0
 0  7 623080 792464   5556 106144 1336 3872  1520 15952 1668 1759  4  6  0 90  0
 0  8 628360 813624   5444 100000  872 5416  1304  5476 2137 1992  4 11  4 82  0
 2 10 628580 816896   5448 100336 1820  348  2124 26216 1849 2156  6  6  7 81  0
 0 14 633576 838948   5396  88236 1156 5228  1392  5228 2362 2170  5 10  0 85  0
 0  8 633488 837784   5400  88784 2076  452  2752   456 1797 2304  5  7  0 88  0
 1  5 632876 832252   5420  92112 2572    0  3092     0 1825 3524 10  6  0 84  0
 1  6 633072 830268   5420  92228 1820  276  1996   280 1850 2180  5  7  0 88  0
 0  5 633020 828532   5420  92324 1752    0  1764     0 1885 2246  5  6  0 89  0
 1  5 633004 826324   5420  92352 2144  132  2144   384 1931 2305  5  6  0 88  0
 0  5 632872 823840   5420  92736 2076    0  2448     0 1124 1495  2  2  0 96  0
 1  9 633036 823204   5588  92316 1300  276  1824  4520 1604 1978  4  5  0 91  0
 0  9 634188 827788   5588  88608  888 1192   892  1192 1350 1657  2  4  0 94  0
 0  7 638116 866228   3992  59692  544 4012  1580  4828 2119 2390  4 12  0 84  0
 1  5 638768 861372   4184  64104 1028  828  2724  3444 2251 3688  8  8  0 84  0
 2  4 639840 854820   4316  70268 1884 1360  2576 15800 2570 6056 12  9  0 79  0
 0  6 639840 851232   4336  71632 3148  488  3984  5928 2422 3785  9  7 18 65  0
 0  4 639768 848808   4344  71808 2432  292  2460   968 1981 2878  6  7 37 51  0
 1  3 639720 863500   4332  65652 1672  260  1788   264 1386 1781  3 10 16 71  0
 0 10 644216 905520   1092  34416 1980 5728  3344 10072 2620 2179  1 11  9 80  0
 1  8 644964 901056   1092  39840  588    0  5864     0 1700 2021  4  4  0 92  0
 0  7 644828 894236   1136  45968  548    0  3208     0 2247 3743  6  3  0 92  0
 0  7 645096 891384   1140  48580  656  336  3552   340 2288 3298  3  3  0 95  0
 0  6 644904 887912   1140  51028 1176    0  3424     0 1989 2684  4  7  0 89  0
 0  7 646864 886172   1112  54664  484 2072  2988  2264 2219 3297  6  6  0 88  0
 0  9 650776 898832   1180  54288  812 3968  3360  4052 3274 5079  4  4  0 91  0
 0  7 650732 896848   1184  55028 1176    0  1520     0 2711 3667  0  1  0 98  0
 0  5 650580 893856   1188  56404 1684    0  3084     4 2328 3258  3  3  0 95  0
 1  6 650924 890864   1188  58808  980  424  3440   424 2346 3496  4  5  0 92  0
 0 14 652316 898924   1088  56384 2488 1948  3488  2152 5092 6569  0  1  0 98  0
 0  5 651924 895360   1088  57316 1228    0  1900     0 2282 2960  0  1 15 85  0
 2  3 651660 893252   1088  57888 1404    0  2004     0 2289 3439  3  3  4 91  0
 0  4 651268 890556   1096  58596 1928    0  2648    12 2352 3296  2  2  2 95  0
 0  4 651084 888700   1116  59392 1124    0  1920  3632 2288 3561  3  4  1 94  0
 2  7 650888 886344   1116  60596 1156    0  2296     0 2270 3195  1  2  0 98  0
 0  4 650528 883240   1116  62276 1156    0  3004     0 2318 3591  5  5  0 91  0
 0  2 650344 880768   1116  63556 1216    0  2316     0 2330 3907  4  5  7 83  0
 2  4 650100 878292   1116  64192 2108    0  2740     0 2069 3352  4  5 21 71  0
 1  5 649880 875640   1128  64284 2324    0  2528   988 1972 2235  5  6 15 74  0
 0  3 649732 873500   1164  65132 1352    0  1772     4 2162 3022  5  6 14 75  0
 1  6 648728 870072   1240  66136 2416    0  2720     0 2362 3920  9  6  3 81  0
 0  8 648324 866136   1240  67400 2592    0  3836     0 2425 3926  5  5 11 79  0
 0  5 648056 862788   1376  68532 2016    0  3364     0 2455 3532  4  6  2 88  0
 1  6 647532 863408   1404  69624 1924    0  3052   740 2378 3741  5  5  5 86  0
 1  5 647196 860664   1408  70052 2220    0  2664  8156 2191 3368  7  6 29 58  0
 1  3 646956 858184   1428  70776 1744    0  2448     0 1802 2182  5  4 36 54  0
 0  2 646768 855580   1676  71824 1396    0  2688     0 1812 2035  5  5 41 49  0



Version-Release number of selected component (if applicable):

2.6.33.6-147.fc13.x86_64

How reproducible:

Always.

Comment 1 Will Newton 2010-08-11 16:54:18 UTC
Here's another example of it "falling of a swap cliff", this time without even running anything in particular. I'm not sure what pushed it into reclaim this time.

Setting swappiness anywhere from 60 to 0 doesn't seem to have much effect (if any).

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  1 557780 530420  14696 205092    0    0  5020     0 2235 5169 10  8 41 41  0
 0  1 557748 524840  14696 210204    4    0  5016     0 2282 3206 14 10 38 38  0
 1  1 557736 520252  14696 214756    0    0  4548     0 2390 3417 17 12 32 39  0
 0  1 557736 515664  14696 219268    0    0  4596     0 2275 3117  8  7 43 42  0
 1  1 557276 509472  14712 223696  120    0  4516  1884 2216 3119 10  7 36 47  0
 1  1 557264 504744  14712 228632    0    0  4992     0 2503 3555 14 11 40 34  0
 1  2 557232 498428  14712 234164  344    0  5852     0 2233 3092 14 10 35 41  0
 1  1 557204 493724  14716 237964  748    0  4528     0 2303 4511 15 11 38 36  0
 1  0 557192 488328  14716 243384    4    0  5412     0 2420 3420 16 11 37 36  0
 0  1 557104 484300  14724 247112   28    0  3752    68 2235 6982 12  8 39 40  0
 2  0 557080 468924  14856 253992    0    0  6964     0 2248 3019 12 10 40 38  0
 2  1 556896 433460  14856 254360  136    0   520     0 2357 4416 56 10 29  5  0
 2  0 556740 419820  14988 254828  652    0   812     0 2613 7049 19  8 55 18  0
 0  0 556208 397872  14988 255280  228    0   228     0 2823 5187 34 10 48  7  0
 0  0 556168 394136  14996 256020   36    0    36   380 3063 5859 19 19 59  4  0
 2  0 556096 393264  14996 256900   64    0    64     0 3221 5737 21 17 59  2  0
 0 28 849944 1042192    144  26416 11416 295580 28820 297852 47992 35967  3  9  1 87  0
 0 26 849188 1033140    144  28816 3292    0  4108     0 1092 1533  0  0  0 100  0
 0 23 848884 1029296    144  29356 3460    0  3956     0  862 1067  0  0  0 100  0
 0 22 848564 1025452    152  30296 2764    0  3744    28  802 1106  0  0  0 100  0
 0 19 848348 1022228    424  31108 2128    0  3216     0  764 1074  0  0  0 100  0
 0 15 848040 1019252    556  31156 2592    0  2880     0  780 1105  0  0  0 100  0
 0 16 847924 1017144    704  31680 1664    0  2108     4  677 1027  0  0  0 100  0
 0 16 847604 1014540    708  31760 2396    0  2640     0  815 1268  0  0  0 100  0
 0 14 847444 1011564    728  32576 2148    0  2956     4  774 1174  0  0  0 99  0
 0 12 847168 1008100    736  33940 2056    0  3288     0  775 1101  0  0  0 100  0
 0 12 846948 1005000    748  35468 1672    0  3196   464  747 1045  0  0  0 100  0
 0 10 846588 1001900    752  36300 2368    0  3144     4  793 1125  0  0  0 100  0

Comment 2 Will Newton 2010-08-13 11:55:25 UTC
The problem seems dependant on having VirtualBox running. Perhaps it is somehow unbalancing the VM and pushing it beyond watermark levels?

I can't really debug this further as a recent kernel upgraded broke my radeon multihead, so I'm now running a 2.6.35 Linus kernel.

Comment 3 Bug Zapper 2011-06-01 11:40:34 UTC
This message is a reminder that Fedora 13 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 13.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '13'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 13's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 13 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 4 Bug Zapper 2011-06-28 14:43:34 UTC
Fedora 13 changed to end-of-life (EOL) status on 2011-06-25. Fedora 13 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.