From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux 2.4.2-2 i686; en-US; m18) Gecko/20010131 Netscape6/6.01 Description of problem: While running enterprise kernel 2.4.6-2.3 on a pe6400 w/ A08 bios, 5GB ram, 2GB swap, perc3dc raid 1 container and ext3 filesystems. Heavy disk activity causes numerous vm low memory warnings in /var/log/messages: Jul 23 18:47:12 localhost cpcmp[2507]: glob 1, stream 1, pass 4, sev 6 Starting 4 ... Jul 23 18:48:02 localhost kernel: journal_write_metadata_buffer: ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:48:05 localhost last message repeated 13 times Jul 23 18:48:05 localhost kernel: : ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:48:05 localhost kernel: journal_write_metadata_buffer: ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:48:44 localhost last message repeated 399 times Jul 23 18:48:47 localhost last message repeated 25 times Jul 23 18:48:47 localhost kernel: : ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:48:47 localhost kernel: journal_write_metadata_buffer: ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:48:50 localhost last message repeated 398 times Jul 23 18:49:21 localhost cpcmp[2516]: glob 1, stream 2, pass 3, sev 6 ... Ending 3 Jul 23 18:49:22 localhost cpcmp[2521]: glob 1, stream 2, pass 4, sev 6 Starting 4 ... Jul 23 18:51:00 localhost cpcmp[2535]: glob 1, stream 4, pass 3, sev 6 ... Ending 3 Jul 23 18:51:00 localhost cpcmp[2540]: glob 1, stream 4, pass 4, sev 6 Starting 4 ... Jul 23 18:52:44 localhost kernel: journal_write_metadata_buffer: ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:54:19 localhost last message repeated 181 times Jul 23 18:54:22 localhost last message repeated 19 times Jul 23 18:54:22 localhost kernel: : ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:54:22 localhost kernel: journal_write_metadata_buffer: ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:54:34 localhost last message repeated 398 times Jul 23 18:55:02 localhost kernel: VM: killing process mrtg Jul 23 18:56:53 localhost syslogd: select: Cannot allocate memory Jul 23 18:57:19 localhost kernel: journal_write_metadata_buffer: ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:57:19 localhost last message repeated 12 times Jul 23 18:57:24 localhost kernel: : ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:57:27 localhost kernel: journal_write_metadata_buffer: ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:57:27 localhost last message repeated 398 times Jul 23 18:57:55 localhost syslogd: select: Cannot allocate memory Jul 23 18:57:59 localhost kernel: journal_write_metadata_buffer: ENOMEM at get_unused_buffer_head, trying again. Jul 23 18:58:05 localhost last message repeated 65 times Jul 23 18:59:58 localhost cpcmp[2572]: glob 1, stream 5, pass 4, sev 6 ... Ending 4 How reproducible: Always Steps to Reproduce: 1.pe6400, 2.4.6-2.3 enterprise kernel 5GB ram, 2GB swap, ext3 filesystems. 2.Run heavy disk activity and note vm low memory warnings 3. Actual Results: Vm low memory warnings in /var/log/messages Expected Results: No warnings Additional info: Opening this bug per request of sct in bug 49438
This defect is considered SHOULD-FIX for the Fairfax release.
VM: killing process was fixed in 2.4.6-3.1; now we are working in improving VM responsiveness while not reintroducing the bug. Warnings from ext3 have been changed to go to syslog only by default and will be throttled. They will not be removed entirely. They are an important diagnostic tool in case bugs show up. Re-assigning to Arjan since ext3 bits have been dealt with, for tracking reports on VM in kernels later than 2.4.6-3.1.
VM should be fixed in 2.4.7-0.5 and later.
More feedback from our own tests and others has indicated that we are correct that this is fixed. If you disagree, please feel free to re-open this bug with current info.
Same warnings continue to appear in /var/log/messages on original configuration running 2.4.7-0.12.0enterprise kernel. Warnings are less frequent, however, occuring every 10-15minutes.
Warnings appearing on a 1400 w/ perc3dcl raid container as boot device, running smp kernel 2.4.7-6, 2GB ram, 2GB swap with frequency of once per few hours.
Any idea under what workload this is ?
Sorry, while running cerberus.
The warnings will not be removed entirely. They are an important diagnostic tool in case bugs show up. That is, after we ship, if a previously undisclosed bug shows up, the presense of this message in the syslog may enable us to find and fix the bug faster. Note that "once every few hours under cerberus" == "almost never under real life loads" and this is even more definitely not a bug.