Bug 785384
Description
Tobi Vollebregt
2012-01-28 14:20:10 UTC
I just want to confirm the findings of Tobi Vollebregt, and add a tiny bit of information. I have turned off rhgb etc. in the kernel boot line, so I get some more information from the system. When I click on hibernate, I get a black screeen with a lot of logging running past too quickly to read. Then it stops on a line saying [1234.5678] PM: Compressing and saving image data nn% The nn is a number that increases, and when the hibernate worked, it went to 100%, and then the machine turned off. With the current kernel it will instead stop at a seemingly random number and stay there. Only hardware reset or similar will work. uname -a: Linux wimsey 3.2.2-1.fc16.i686.PAE #1 SMP Thu Jan 26 03:30:43 UTC 2012 i686 i686 i386 GNU/Linux /proc/cpuinfo (beginning): processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz stepping : 6 The end of /var/log/pm-suspend.log: /usr/lib/pm-utils/sleep.d/98video-quirk-db-handler hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/99video hibernate hibernate: /usr/lib/pm-utils/sleep.d/99video hibernate hibernate: success. Tue Feb 7 09:57:04 CET 2012: performing hibernate Problem solved, I think. I suddenly remembered that i had, and solved, the same problem under Fedora-15. I use Virtualbox, and its modules seem to do bad things with hibernate. The solution is then to make the system unload the Virtualbox module just before hibernating. This is achieved by putting a configuration file in the directory /etc/pm/config.d. In my case I name the file 'vboxdrv', and the content is the single line: SUSPEND_MODULES=vboxdrv After this hibernation works flawlessly. Strangely enough it actually did work without this configuration file in kernel 3.1.9, but the file should be there anyway. To Tobi: I hope you can find a similar solution. If it is not Virtualbox, then perhaps you can find out by unloading any special module that you may have loaded. The morally correct action now is to try to convince the Virtualbox people that they should include this little file in the distribution. We'll see. Hmm, interesting. I do use virtualbox occasionally, but I do not appear to have its modules loaded currently. Don't know for sure whether I had them loaded with 3.2.2, though I think not. Will try to check later if I can pull this trick with some other module. Hibernate works again since kernel 3.2.5-3.fc16.i686 (or possibly one of the other packages that came in the upgrade). Closing this bug. This does not work for me. I have given details in http://forums.fedoraforum.org/showthread.php?p=1556879#post1556879 In my case pm-hibernate/resume works a few times, then freezes during hibernation, after reporting use of 3 threads for compression. I have tried all available 3.2 kernels in Fedora 16 up to 3.2.6-3. However 3.1.7-1 works fine, though more slowly. You're, right, the problem is still present, with the note that with kernel 3.2.2 a hang seemed to be much more likely than with 3.2.5 and 3.2.6. Past week, with 3.2.6, I had some 4 succesful hibernates, 5th time froze. Next boot the first hibernate froze. I don't know if it will help anyone, but I took a couple of (not very clear) pictures of my laptop screen the last time it froze during pm_hibernate. The lower half, showing the hibernate frozen at 69% is here: http://www.cs.bham.ac.uk/~axs/pics/laptop-screen-lower-closeup.jpg The upper half, which seems to include some of the output of the last resume from hibernate is here: http://www.cs.bham.ac.uk/~axs/pics/laptop-screen-upper.jpg The "acpi cmd .... rejected..." lines are presumably there because I found (after following a tip on the internet somewhere) that before resume from hibernate I need to add the flag 'acpi=off' to the boot command otherwise resume tended to fail and produced a reboot. That was also necessary with 3.1 kernels in F16. It used not to be necessary in earlier versions of Fedora. (I can't include the acpi=off command when doing a full reboot as that stops some things from working, e.g. getting information about battery state.) So I have two grub2 entries, one for boot, used rarely, and one for resume, the default. I tried removing that from my resume option and it did not prevent pm_hibernate failing. I don't use virtualbox, so I suspect the apparent link mentioned above was just a coincidence. My machine is a Dell Latitude E6410 with 4GB mem 500GB drive, 1440x900 screen using intel graphics (i9150) and intel integrated wireless. I use 32bit fedora 16. The messages: "Dell_wmi: Received unknown WMI event" visible on the screen have been appearing with no known cause and no known harmful effects for many months, including, I think when I was running Fedora 13 and fedora 15. (I skipped F14 because I could not get the screen to work properly.) I'm having exactly the same behaviour since kernel 3.2 that during hibernation system hang at " PM: Compressing and saving image data nn%" ! See similar bugreport here: https://bugzilla.redhat.com/show_bug.cgi?id=789708 Using 3.2.7-1.fc16.i686 I found at first that pm-hibernate froze as previously. However, on a hunch I removed my SD card from the card-reader on the Dell E6410 (previously left in whether mounted or not, as a backup device). Since then I have hibernated and resumed 7 or 8 times without pm-hibernate freezing. Later I'll try re-inserting the card and see if the freezing returns. For now I would prefer to see how long it works without freezing. I tried reinserting the SD card. That seemed to cause pm-hibernate to freeze. Of course, that might have been a fluke. Is there anything I can do, as a fairly ignorant user, to provide more information to help the developers ? I can't imagine that only the SD card is the root of the problem. because my desktop at home is malfunctioning without any additional cards such the harddisk which is a SSD. Thanks for the comment. It sounds as if my experience with the SD card is a red-herring. As I understand the situation, the main difference between hibernate in 3.1 and 3.2 kernels is that in 3.2 multiple threads are used for compression (on a multi-core cpu), making hibernation very much faster. So maybe this bug does not affect anyone using a single core cpu? That might explain why so few people have joined this bug report? the multi thread usage sounds much more plausible... But for example on my netbook hibernation works flawless although it is also a "dual core cpu" and my desktop in the office works either. so, I played arround a little bit to find out if it is the multi threaded hibernate or not. So for this case I switched off my dual core cpu to a single core cpu via bios and it's the same behaviour... freezes during compressing the image with only one thread. aaaarrrghhh now I will try to unload all modules manually... Just to confirm that I got the freeze on hibernate without my SD card plugged in. So the earlier sequence of several non-freezes without card followed by freeze with card was just a coincidence. Another thing I've wondered about: I don't use NetworkManager because I've never been able to get it to do what I want, whereas wicd (and wicd-client) have proved a lot more usable (also for several of my colleagues, who mostly use Ubuntu). Unfortunately it seems that some linux developers write code assuming that NM is used for connectivity (e.g. for a while firefox had that flaw). Is it possible that there's some troublesome interaction between wicd and pm-hibernate, or absence of NM and pm-hibernate? If others reporting this bug are using NM, that conjecture is ruled out. Alas, something has changed and I am now unable to get 3.1.7-1 to work properly: after hibernate+resume I get system errors. I also tried the latest tuxonice kernel for F16. I previously used the tuxonice kernel until F15, after which it started freezing on resume for me. But that problem still remains. I have today noticed that there's an update for kernel-tools for 3.2.7-1. So far have managed to pm-hibernate and resume and pm-hibernate again after installing that. Is there a place to find out what's changed in the new kernel-tools and why? kernel-tools is a sub-package of the kernel (which is what changed). The 3.2.6 build had a patch included that may have improved hibernate in some situations, but there's still a lot of work to do, as it's still broken for many users, including situations where it randomly corrupts memory. Thanks very much for the feedback. It's very helpful to know this is being actively investigated by experts. I've gone back to using 3.1.7-1.fc16.i686 which for a short time after one of the upgrades caused errors. But for several days it has been working fine for me after I un-installed 3.2.7-1 The discussion about abandoning hibernate, in https://bugzilla.redhat.com/show_bug.cgi?id=781749 , is very alarming: I'm running Debian and also hit this bug, reported here: http://bugs.debian.org/659363 If I leave my computer on for a while after hibernate has frozen, I get error messages from the kernel about hung tasks. I made a photo of them; will attach it. To me it seems that it could be an I/O issue? Bojan Smojver says that he made hibernate I/O "smarter" in 3.2: https://bugzilla.redhat.com/show_bug.cgi?id=781749#c43 I have e-mailed him and asked him to take a look at this bug. Created attachment 569728 [details]
photo of kernel messages after hibernate hangs
(In reply to comment #21) > I'm running Debian and also hit this bug, reported here: > http://bugs.debian.org/659363 > > If I leave my computer on for a while after hibernate has frozen, I get error > messages from the kernel about hung tasks. I made a photo of them; will attach > it. > > To me it seems that it could be an I/O issue? > > Bojan Smojver says that he made hibernate I/O "smarter" in 3.2: > https://bugzilla.redhat.com/show_bug.cgi?id=781749#c43 > > I have e-mailed him and asked him to take a look at this bug. Looked at this and it doesn't seem like an I/O related bug from the screenshots. lzo_compress_threadfn() is doing compression in memory, not I/O. crc32_threadfn is calculating CRC32 in memory, so not I/O again. Anyhow, you can use more or less the same I/O by passing hibernate=nocompress to the kernel. Threading and compression are not used in such a case. However, when i915 corrupts memory on thaw, there is no telling what will get broken next. So, try doing you tests with nomodeset passed to the kernel. If you can reproduce the same thing, then i915 is not involved. PS. The improvements in I/O of hibernation in 3.2 are related to better batching of pages that are read/written (more of them are queued up to be used by async I/O at any given time). Underlying I/O otherwise works the same. (In reply to comment #23) > Looked at this and it doesn't seem like an I/O related bug from the > screenshots. lzo_compress_threadfn() is doing compression in memory, not I/O. > crc32_threadfn is calculating CRC32 in memory, so not I/O again. I thought that, perhaps, they were waiting on input or output. If they were not waiting for anything, how could the kernel possibly detect them as "hung"? > Anyhow, you can use more or less the same I/O by passing hibernate=nocompress > to the kernel. Threading and compression are not used in such a case. OK, thanks. Will try. > However, when i915 corrupts memory on thaw, there is no telling what will get > broken next. So, try doing you tests with nomodeset passed to the kernel. If > you can reproduce the same thing, then i915 is not involved. Will do. Note however that the bug reporter is using an Nvidia GPU. Also, I have never seen any segfaults the way others have when i915 corrupts memory. > PS. The improvements in I/O of hibernation in 3.2 are related to better > batching of pages that are read/written (more of them are queued up to be used > by async I/O at any given time). Underlying I/O otherwise works the same. I'm sure it's better. I'm just thinking that maybe it interacts badly with something else on my system (such as sleep/wake of SATA devices). (In reply to comment #24) > I thought that, perhaps, they were waiting on input or output. If they were not > waiting for anything, how could the kernel possibly detect them as "hung"? These threads are started when image saving starts and are kept alive until saving is done (they are signalled when data is ready for them, at which point they run their work function and sleep again). If anything else hangs the kernel in the meantime, they will be there, running. So, they will be reported as unfinished threads. > Will do. Note however that the bug reporter is using an Nvidia GPU. Proprietary driver there, so I doubt anyone will want to touch this with a ten foot pole. > Also, I > have never seen any segfaults the way others have when i915 corrupts memory. Yeah, not everyone is seeing segfaults. Some are seeing OOPSes. Some kernel panics. I've seen all of those on my machine. And, of course, the hang could be caused by things other than i915. And it could be caused by hibernation improvement code - I'm just not seeing evidence for that yet. If you want to verify whether other kernel versions also hang your box, you can try this: echo -n reboot > /sys/power/disk; for (( i=0; i<125; i++)); do echo $i; pm-hibernate; chvt 2; sleep 1; chvt 1; sleep 2; done This will exercise hibernation/thaw repeatedly. On my system, I usually get in trouble after about 20 cycles. Sometimes after 50. > I'm sure it's better. I'm just thinking that maybe it interacts badly with > something else on my system (such as sleep/wake of SATA devices). Possibly. Although none of the device code has been touched by threading and I/O improvements of hibernate in 3.2. We just keep deeper queues (i.e. more pages ready be written or read into) so that async I/O is fed more efficiently. (In reply to comment #25) > And it > could be caused by hibernation improvement code - I'm just not seeing evidence > for that yet. If you feel like building the kernel, you can revert threading and I/O improvements, by reverting this commit: 081a9d043c983f161b78fdc4671324d1342b86bc. http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=081a9d043c983f161b78fdc4671324d1342b86bc If it doesn't hang after that, then the problem is most definitely somewhere in there. Created attachment 570742 [details]
Half the number of pages used by hibernation I/O
If anyone would like to play by compiling the kernel, this patch will reduce the number of pages used by hibernation code in half, so that hopefully other parts of the kernel have enough and do not hang waiting for free pages. Well, that's this flimsy theory at least.
Let me know whether it helps.
I have now been able to reproduce this bug without i915 loaded, and without any virtualbox modules. I blacklisted i915 and turned off the script which loaded the vbox* modules. I think I can now safely say that this bug is not due to i915 corruption. Here is what I do to reproduce the bug: 1. Pass hibernate=nocompress to the kernel. 2. Create a big file in shared memory. I used: dd if=/dev/urandom of=/dev/shm/bigfile bs=1M count=750 3. echo disk >/sys/power/state 4. Watch the counter freeze. This works even without i915 and X. I will try to build the kernel with Bojan's commit reverted when I have time. Created attachment 571226 [details]
Better watermark the number of pages used by hibernation I/O
Just a slightly better patch to try. This one tracks the number of free pages and flushes I/O when we fall below half of available pages.
(In reply to comment #28) > I have now been able to reproduce this bug without i915 loaded, and without any > virtualbox modules. I blacklisted i915 and turned off the script which loaded > the vbox* modules. I think I can now safely say that this bug is not due to > i915 corruption. Thanks. Obviously a different bug and not the garden variety i915 corruption. > Here is what I do to reproduce the bug: > > 1. Pass hibernate=nocompress to the kernel. > 2. Create a big file in shared memory. I used: > dd if=/dev/urandom of=/dev/shm/bigfile bs=1M count=750 > 3. echo disk >/sys/power/state > 4. Watch the counter freeze. Tried this on my ThinkPad T510: --------------------------- [root@shrek ~]# df -h /dev/shm Filesystem Size Used Avail Use% Mounted on tmpfs 3.9G 76K 3.9G 1% /dev/shm [root@shrek ~]# dd if=/dev/urandom of=/dev/shm/bigfile bs=1M count=4000 dd: writing `/dev/shm/bigfile': No space left on device 3892+0 records in 3891+0 records out 4080844800 bytes (4.1 GB) copied, 301.019 s, 13.6 MB/s [root@shrek ~]# df -h /dev/shm Filesystem Size Used Avail Use% Mounted on tmpfs 3.9G 3.9G 0 100% /dev/shm [root@shrek ~]# echo -n reboot > /sys/power/disk; for (( i=0; i<10; i++)); do echo $i; pm-hibernate; done 0 1 2 3 4 5 6 7 8 9 [root@shrek ~]# df -h /dev/shm Filesystem Size Used Avail Use% Mounted on tmpfs 3.9G 3.9G 0 100% /dev/shm [root@shrek ~]# uname -r 3.2.10-3.fc16.x86_64 [root@shrek ~]# grep linux /boot/grub2/grub.cfg ### BEGIN /etc/grub.d/10_linux ### menuentry 'Fedora (3.2.10-3.fc16.x86_64)' --class gnu-linux --class gnu --class os { linux /vmlinuz-3.2.10-3.fc16.x86_64 root=/dev/mapper/vg00-lv00 ro SYSFONT=latarcyrheb-sun16 LANG=en_AU.UTF-8 KEYTABLE=us hibernate=nocompress nomodeset --------------------------- So, cannot replicate it readily. How many tries did it take you to get a hang? I'm having a feeling this may be related to your particular hardware combo... PS. I did the above test with i915 loaded as well. After 10th thaw, I got segfaults, followed by a kernel panic. Just FYI. (In reply to comment #30) > So, cannot replicate it readily. How many tries did it take you to get a hang? > I'm having a feeling this may be related to your particular hardware combo... I only need to hibernate once, it always happens the first time. I also think it has something to do with the particular hardware. My guess is that there is a bug somewhere which is triggered by the deeper I/O queues. Maybe some ACPI weirdness with the SATA disks. When I hibernate, the kernel writes several messages related to ACPI sending commands to SATA disks: [ 8391.960027] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 8392.015413] ata1.00: ACPI cmd ef/03:0c:00:00:00:a0 (SET FEATURES) filtered out [ 8392.015417] ata1.00: ACPI cmd ef/03:45:00:00:00:a0 (SET FEATURES) filtered out [ 8392.015450] ata1.00: ACPI cmd c6/00:10:00:00:00:a0 (SET MULTIPLE MODE) succeeded [ 8392.015487] ata1.00: ACPI cmd e3/00:00:00:00:00:a0 (IDLE) succeeded [ 8392.015490] ata1.00: ACPI cmd b1/c1:00:00:00:00:a0 (DEVICE CONFIGURATION OVERLAY) filtered out [ 8392.015494] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out [ 8392.015497] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out [ 8392.016631] ata1.00: ACPI cmd ef/03:0c:00:00:00:a0 (SET FEATURES) filtered out [ 8392.016636] ata1.00: ACPI cmd ef/03:45:00:00:00:a0 (SET FEATURES) filtered out [ 8392.016665] ata1.00: ACPI cmd c6/00:10:00:00:00:a0 (SET MULTIPLE MODE) succeeded [ 8392.016698] ata1.00: ACPI cmd e3/00:00:00:00:00:a0 (IDLE) succeeded [ 8392.016702] ata1.00: ACPI cmd b1/c1:00:00:00:00:a0 (DEVICE CONFIGURATION OVERLAY) filtered out [ 8392.016706] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out [ 8392.016710] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out (In reply to comment #31) > I only need to hibernate once, it always happens the first time. Right. So, very deterministic. At least some good news. Kind of... :-( > I also think it has something to do with the particular hardware. My guess is > that there is a bug somewhere which is triggered by the deeper I/O queues. > Maybe some ACPI weirdness with the SATA disks. When I hibernate, the kernel > writes several messages related to ACPI sending commands to SATA disks: Possibly. Although this could also be a red herring. Here is the suspend (hibernate) workflow from http://www.kernel.org/doc/Documentation/power/swsusp.txt: ============================ Suspend part ~~~~~~~~~~~~ running system, user asks for suspend-to-disk user processes are stopped suspend(PMSG_FREEZE): devices are frozen so that they don't interfere with state snapshot state snapshot: copy of whole used memory is taken with interrupts disabled resume(): devices are woken up so that we can write image to swap write image to swap suspend(PMSG_SUSPEND): suspend devices so that we can power off turn the power off ============================ So, there is a resume just before the image is written to disk. If there is a bug in anything that's being resumed (i.e. various modules), it may hang the kernel at any time during the image writing phase. Of course, once you revert my commit that changed buffering, we'll know for sure. In relation to that, the only thing I can think of is that some device that was resumed just ran out of pages (because we now use more for hibernation queues) and the kernel hangs waiting for more. But, then again, it's very hard to find bugs in your own code. :-( (In reply to comment #32) > > I only need to hibernate once, it always happens the first time. same for me. happens every first time. On the other hand if I add acpi=off as kernel parameter it fails on the second hibernate. since updated to kernel 3.3 no more hangs during hibernation. OK, now I have done some more testing. All tests were done from the console, without compression, without i915 loaded, and with a big file in tmpfs. Linux 3.2.12: Hibernate hangs. Linux 3.2.12 with commit 081a9d043c983f161b78fdc4671324d1342b86bc reverted: Hibernate works fine Linux 3.3.0: Hibernate works, but during image writing (when it usually hangs) the kernel prints an error message/oops related to memory pages. The hibernation continues, though. The error messages flashes through too quickly for me to read. It also isn't present in dmesg after resuming. Is it possible to pause hibernation after writing the image but before shutting down the system? (On a sidenote: when compression is disabled, one can use /dev/zero instead of /dev/urandom to write a big file on a tmpfs. It is much faster.) Created attachment 572081 [details]
Better watermark the number of pages used by hibernation I/O
Same patch, v3.
(In reply to comment #35) > OK, now I have done some more testing. All tests were done from the console, > without compression, without i915 loaded, and with a big file in tmpfs. > > Linux 3.2.12: Hibernate hangs. > Linux 3.2.12 with commit 081a9d043c983f161b78fdc4671324d1342b86bc reverted: > Hibernate works fine > Linux 3.3.0: Hibernate works, but during image writing (when it usually hangs) > the kernel prints an error message/oops related to memory pages. The > hibernation continues, though. Aha, so it does look like memory pressure causes this. I'm guessing some piece of hardware you have needs more pages, can't get them because they are used by hibernation buffers, which then causes the kernel to hang. > The error messages flashes through too quickly for me to read. It also isn't > present in dmesg after resuming. Is it possible to pause hibernation after > writing the image but before shutting down the system? Only if we insert some explicit code to sleep. I can get you a patch for that. > (On a sidenote: when compression is disabled, one can use /dev/zero instead of > /dev/urandom to write a big file on a tmpfs. It is much faster.) Yeah, pages containing zeros compress pretty good. :-) Anyhow, could you please put back commit 081a9d043c983f161b78fdc4671324d1342b86bc to 3.2.12, apply the patch that I attached here and try with that? Thanks for testing! Created attachment 572093 [details]
Sleep 10 seconds after image write, so that photo can be taken
Per,
You can try this patch for pausing after the image has been written. Let me know if it doesn't work for you and we can try something else.
Yesterday I installed vmlinuz-3.3.0-2.fc16.i686 on Dell Latitude D6410 (with core i5 cpu and intel graphics: 1440x900). First resume after hibernate caused a reboot, so I tried again using acpi=off for resume. Since then it has hibernated and resumed several times without any problem. (Single thread compression for pm-hibernate?) Used much of the day, without any problems. Tonight noticed that 3.3.0-4 was available, so installed it and tested. Hibernates very fast (three threads for compression) First hibernate+resume worked. Second one did a reboot after resume failed, so inserted acpi=off for resume. And tried again. Resume crashed with segfaults reported, but just in case I had done something wrong, re-started. Since then have managed four or five hibernate+resume cycles (with acpi=off for resume) and it seems to be working fine. (Could the earlier crash have been due to something left over from earlier kernel. Perhaps I should clear swap are for each new kernel?) During boot and resume I get this, just before screen resolution changes: kernel: [ 7.038582] ata2: link is slow to respond, please be patient (ready=0) It pauses for a second or two before printing that then continues after another second or so. Will go back to 3.3.0-2 if I have any more problems, but so far looks fine. (In reply to comment #39) > During boot and resume I get this, just before screen resolution changes: > kernel: [ 7.038582] ata2: link is slow to respond, please be patient > (ready=0) Hmm, this looks like some kind of hardware problem (or driver problem) related to SATA. > Could the earlier crash have been due to something left over from earlier kernel. Perhaps I should clear swap are for each new kernel? This should never be the case. You should never need to manually clean anything. (In reply to comment #40) > > Could the earlier crash have been due to something left over from earlier kernel. Perhaps I should clear swap are for each new kernel? > > This should never be the case. You should never need to manually clean > anything. I've done several more hibernate-resume cycles without any problem, using 3.3.0-4 Perhaps on the occasion it went wrong I inadvertently resumed using the wrong kernel. 3.3. looks like a major advance. If only it was not necessary to have two grub.cfg entries, one with acpi=off for resume, that would be excellent. I expect most people would not tolerate having to edit grub.cfg. (In reply to comment #41) > 3.3. looks like a major advance. Given that threading and buffering that I submitted in 3.2 did not change with 3.3 (apart from lack of required free space calculation when compression is used, which is before image writing starts, so irrelevant), I do not think your problem is the same as Per's problem. > If only it was not necessary to have two > grub.cfg entries, one with acpi=off for resume, that would be excellent. I > expect most people would not tolerate having to edit grub.cfg. The fact that turning ACPI off on resume is making a difference on your system would suggest that you are having hardware/driver issues. You should never need two different grub entries to hibernate/thaw. PS. The reason hibernate/thaw works faster for you now is that Fedora patch to reduce the number of threads to 1 has been reverted. So, not related to 3.3. Created attachment 572114 [details]
Better watermark the number of pages used by hibernation I/O
A more aggressive patch - required that 3/4 of pages be free at all times.
hibernating will not hang anymore with kernel 3.3.x but still a lot of troubles. I saw several failures: most of the time I go a "PM: LZO decompression failed/"Invalid LZO uncompressed length" and the machine booted instead of resuming. on the other hand I got some kernel panics which I cannot post here. ( could not find anywhere on logs ) and sometimes call traces like this: Mar 26 19:25:49 kernel: [ 8858.103137] Call Trace: Mar 26 19:25:49 kernel: [ 8858.103154] [<c058f530>] ? dquot_quota_off+0x20/0x20 Mar 26 19:25:49 kernel: [ 8858.103178] [<c056dce7>] __sync_filesystem+0x87/0x90 Mar 26 19:25:49 kernel: [ 8858.103202] [<c056dd07>] sync_one_sb+0x17/0x20 Mar 26 19:25:49 kernel: [ 8858.103226] [<c0549c38>] iterate_supers+0xb8/0xc0 Mar 26 19:25:49 kernel: [ 8858.103249] [<c056dcf0>] ? __sync_filesystem+0x90/0x90 Mar 26 19:25:49 kernel: [ 8858.103274] [<c056dd9f>] sys_sync+0x3f/0x60 Mar 26 19:25:49 kernel: [ 8858.103299] [<c0951cdf>] sysenter_do_call+0x12/0x28 Mar 26 19:25:49 kernel: [ 8858.103323] Code: 00 00 00 00 39 c3 8d b0 68 ff ff ff 75 1a eb 6c 8d 74 26 00 80 07 01 8b 86 98 00 00 00 39 45 bc 8d b0 68 ff ff ff 74 54 8d 7e 4c <8b> 5e 20 89 f8 e8 47 30 3e 00 f6 46 64 38 75 d9 8b 43 3c 85 c0 Mar 26 19:25:49 kernel: [ 8858.103704] EIP: [<c0567aff>] sync_inodes_sb+0xbf/0x150 SS:ESP 0068:ee7f9f20 Mar 26 19:25:49 kernel: [ 8858.103743] CR2: 00000000ffffff89 Mar 26 19:25:49 kernel: [ 8858.103761] ---[ end trace 94ea8aff47ec29aa ]--- However, I have also a desktop in the office with not troubles at all!! (In reply to comment #44) > hibernating will not hang anymore with kernel 3.3.x but still a lot of > troubles. I added a response to this in comment 9 in bug #789708 (In reply to comment #44) > most of the time I go a "PM: LZO decompression failed/"Invalid LZO uncompressed > length" and the machine booted instead of resuming. If you are seeing this, the image data that was saved to disk is corrupt. This should never happen. Meaning, you are probably having hardware or driver problems in your I/O path and the image data is not being saved properly. This is why we now check CRC32 of the image, but your thaw attempt didn't even make it that far. > However, I have also a desktop in the office with not troubles at all!! Yeah, more than likely hardware related. (In reply to comment #46) > (In reply to comment #44) > > > most of the time I go a "PM: LZO decompression failed/"Invalid LZO uncompressed > > length" and the machine booted instead of resuming. > > If you are seeing this, the image data that was saved to disk is corrupt. This > should never happen. Meaning, you are probably having hardware or driver > problems in your I/O path and the image data is not being saved properly. I have a question about this. Suppose, as happened to me, 'yum update' (apparently using 'grubby') inserted the wrong root partition ID in the grub.cfg menu entry for the new kernel. Then what would happen when booting? It would presumably use the new files in the /boot partition for 3.3.0-4 but in combination with an older collection of files in the old root partition (in my case a fedora 15 root partition). It seemed to boot OK when that happened to me, but at a later stage I got a variety of bizarre errors, which disappeared after I rebooted into an older kernel, edited the grub.cfg file to use the right partition for new kernel, then rebooted into the new kernel. Could the errors reported in comment #44 be a result of running the new startup files in the wrong root partition? This would of course only be possible if the /boot partition was separate from the root partition. > However, I have also a desktop in the office with not troubles at all!! This could simply be a manifestation of the semi-random nature of the errors made by grubby in creating grub.cfg -- it could depend on whether another partition had previously been used as a root partition, e.g. for an earlier version of fedora. (as on my laptop). There are several reports of the bug in grubby, e.g. bug #751875 The buggy grubby seems to produce slightly different problems for different users. (In reply to comment #47) > I have a question about this. Suppose, as happened to me, 'yum update' > (apparently using 'grubby') inserted the wrong root partition ID in the > grub.cfg menu entry for the new kernel. Then what would happen when booting? It > would presumably use the new files in the /boot partition for 3.3.0-4 but in > combination with an older collection of files in the old root partition (in my > case a fedora 15 root partition). The hibernation image data is not stored on the root partition, it is stored in swap. This swap is specially marked as a partition that has a hibernation image (S1SUSPEND is the signature). Once that swap partition is identified, the image is read from there and what is being read is verified after decompression (that's how we know we got the correct stuff). So, very unlikely that these two would be related. (In reply to comment #48) Thanks for rapid response. > The hibernation image data is not stored on the root partition, it is stored in > swap. This swap is specially marked as a partition that has a hibernation image > (S1SUSPEND is the signature). Once that swap partition is identified, the image > is read from there and what is being read is verified after decompression > (that's how we know we got the correct stuff). > > So, very unlikely that these two would be related. But long before any hibernate or resume is attempted the system will have booted using libraries from an old root partition (/usr/lib, kernel headers, and others?) could this not mean that by the time hibernation begins many things have been corrupted which lead to errors? Or is the hibernate/resume code sufficiently encapsulated as to be impermeable to any such corruption by mixing versions? Unfortunately I did not keep detailed records of all the bizarre things that I experienced before I discovered that grubby was putting the wrong root UUID in grub.cfg. I think some of them involved hibernate or resume, and others involved programs simply generating errors and aborting. I am reluctant to try deliberately booting with the wrong root partition uuid to investigate this! (In reply to comment #49) > Or is the hibernate/resume code sufficiently encapsulated as to be impermeable > to any such corruption by mixing versions? Hibernation/thaw that we are talking about here is done entirely by kernel code - no user libraries or other code is ever used. The image that is being thawed is stored entirely on the swap partition. Each compressed chunk is verified against obvious decompression errors. If any are detected, thaw will fail. Also, a CRC32 of uncompressed data is calculated and verified against what's been calculated/stored at hibernation time. If CRC32 is not the same, again thaw will fail. So, short of your kernel RPM being somehow corrupt (meaning your /boot/vmlinuz-`uname -r` being changed by something), the userspace will have zero impact on hibernate/thaw. Try: rpm -qvf -V /boot/vmlinuz-`uname -r` | grep vmlinuz It should print all dots before the file name. (In reply to comment #50) > Try: > > rpm -qvf -V /boot/vmlinuz-`uname -r` | grep vmlinuz > > It should print all dots before the file name. Yes. So if by the time hibernation starts there's anything corrupt in memory because the wrong root partition was used, it will simply be compressed and stored on swap, and then during resume it will all be uncompressed and the same corrupt system restored. But there will not be evidence during hibernate or resume of the corruption. Thanks for the clarification. (In reply to comment #46) > (In reply to comment #44) > If you are seeing this, the image data that was saved to disk is corrupt. This > should never happen. Meaning, you are probably having hardware or driver > problems in your I/O path and the image data is not being saved properly. > > This is why we now check CRC32 of the image, but your thaw attempt didn't even > make it that far. > > > However, I have also a desktop in the office with not troubles at all!! > > Yeah, more than likely hardware related. Hmm... the main diffrence is the amount of RAM ( 2GB in the office and 6GB at home ) and radeon graphic driver instead of i915 at home. However is there anything I can afford to find the issue faster? or do I just lean back and wait if a furhter update solve the problem? (In reply to comment #52) > Hmm... the main diffrence is the amount of RAM ( 2GB in the office and 6GB at > home ) and radeon graphic driver instead of i915 at home. > > However is there anything I can afford to find the issue faster? or do I just > lean back and wait if a furhter update solve the problem? Try looking for similar kernel traces reported by other folks. This may give you clues. You may have different disk controllers in these two machine, which may be causing curruption problems. A guess only... (In reply to comment #53) > > You may have different disk controllers in these two machine, which may be > causing curruption problems. A guess only... yes, this is also true. I'm havin an SSD at home and a standard sata disk at offices desktop. is it helpful to try also with a sata disk on my failing computer? (In reply to comment #54) > (In reply to comment #53) > > > > > You may have different disk controllers in these two machine, which may be > > causing curruption problems. A guess only... > > yes, this is also true. > I'm havin an SSD at home and a standard sata disk at offices desktop. > > is it helpful to try also with a sata disk on my failing computer? I do not know for sure. It may make a difference. (In reply to comment #48) > So, very unlikely that these two would be related. At the risk of pointing out the obvious, let me just qualify this a bit. If you have two OSes "sharing" file systems and you hibernate one, then boot the other, you are risking file system corruption. Also, if you hibernate one and then the other somehow modifies the swap where hibernation image is written, all bets are off. http://www.kernel.org/doc/Documentation/power/swsusp.txt so, actually I removed 4 gigs of RAM ( only 2GB left ) and did some tests. I hibernate/thaw 5 times my computer with the commandline loop described in post #25 with success. then I worked again with my computer and did this loop again. also with success. so, does this depend on the amount of RAM and/or the amount of RAM devices inserted? maybe it is also another red herring... (In reply to comment #57) > so, does this depend on the amount of RAM and/or the amount of RAM devices > inserted? The more RAM you have, the more is being used for buffering during hibernation, to improve performance. Of course, if more memory is used by the kernel, swap space needs to be bigger to hold a larger image as well. I have 8 GB in my laptop and that never caused any particular problems for me. So, yeah, it could be a red herring. Then again, does your machine pass memory stress tests with all the RAM? (In reply to comment #58) > I have 8 GB in my laptop and that never caused any particular problems for me. Is there a recommended minimum ratio between swap size and RAM for hibernate to work? There used to be a general recommendation to have at least twice as much swap size as RAM for running unix, e.g. to allow forking of large processes. I expect hibernate does not require as much as that. Presumably it will provide a warning and abort if there is not enough swap space. I have never seen that happen. The man file for pm-utils does not mention swap space, surprisingly. (In reply to comment #59) > (In reply to comment #58) > > > I have 8 GB in my laptop and that never caused any particular problems for me. > > Is there a recommended minimum ratio between swap size and RAM for hibernate to > work? There used to be a general recommendation to have at least twice as much > swap size as RAM for running unix, e.g. to allow forking of large processes. > > I expect hibernate does not require as much as that. Presumably it will provide > a warning and abort if there is not enough swap space. I have never seen that > happen. The man file for pm-utils does not mention swap space, surprisingly. In kernel hibernation can save at most half of RAM. Default I think is 2/5 (but check the source ob this). In 3.2, size of the swap is checked when compression is used. In 3.3, it is not - hibernation fails when space is short. When compression is not used, size is always checked. [Mass hibernate bug update] Dave Airlied has found an issue causing some corruption in the i915 fbdev after a resume from hibernate. I have included his patch in this scratch build: http://koji.fedoraproject.org/koji/taskinfo?taskID=3940545 This will probably not solve all of the issues being tracked at the moment, but it is worth testing when the build completes. If this seems to clear up the issues you see with hibernate, please report your results in the bug. Created attachment 573466 [details]
page allocation failure message photo (1)
Created attachment 573467 [details]
page allocation failure message photo (2)
Hi, I have now finally had time to take photos. Sorry for the bad quality, my camera is not very good (nor the lighting). I guess I would be able to capture the message through a serial console also. I do not seem to have a null modem cable, but I can get one if it helps. Bojan, do you want me to try your latest patch also? (In reply to comment #64) > I have now finally had time to take photos. Sorry for the bad quality, my > camera is not very good (nor the lighting). Never mind the quality - they give precisely the information that is required. > I guess I would be able to capture the message through a serial console also. I > do not seem to have a null modem cable, but I can get one if it helps. Nah, don't worry about that now. Images will do. > Bojan, do you want me to try your latest patch also? Yes, please. I will attach the latest version, which will land in 3.5. Created attachment 573507 [details]
Better watermark the number of pages used by hibernation I/O
Version of the patch that is queued for 3.5 by Rafael.
Created attachment 574743 [details]
Better watermark the number of pages used by hibernation I/O
One more tweak to the patch: enable super-paranoia mode and flush BIO chain when number of free pages is equal or less than required number of free pages. Just in case required free pages (somehow) turns up being zero.
I can confirm that your latest patch works. 3.2.12 no longer freezes, and 3.3 no longer prints an error. (In reply to comment #68) > I can confirm that your latest patch works. 3.2.12 no longer freezes, and 3.3 > no longer prints an error. Thank you very much for testing. Truly appreciated! I will ask Rafael to queue up for 3.4 and stable as well. (In reply to comment #69) > Thank you very much for testing. Truly appreciated! And thank you for fixing the problem for me! :-) added this patch to all branches. will be in next updates. Thanks Bojan. Sorry, I was too quick. The problem still exists in 3.2 with the patch. It's just that my testcase no longer triggers the bug. The bug is still sometimes triggered during normal use. The only difference is that the counter goes up higher before freezing (over 90%). I also let the computer on for a while and I then received messages like those in the first attached photo. It happened again now, but this time the counter froze at 64%. The counter seems to advance slower with the patch applied, before it freezes. On a side note: I am running a 32-bit kernel with PAE (CONFIG_HIGHMEM64G=y) and I have 4 GB physical memory. Do you think it matters? Should I try a 64-bit kernel? (In reply to comment #72) > Sorry, I was too quick. > > The problem still exists in 3.2 with the patch. It's just that my testcase no > longer triggers the bug. The bug is still sometimes triggered during normal > use. The only difference is that the counter goes up higher before freezing > (over 90%). I also let the computer on for a while and I then received messages > like those in the first attached photo. OK, thanks for the feedback. What we know for sure is that kernel runs out of pages during hibernation. I see two possibilities: 1. Pages are not freed as they should be when BIO chain is flushed. So, change in buffering triggered this problem. 2. Something hardware specific is eating pages on your system during hibernation. I will try to create a debugging patch, so that we can determine where the pages went. Question: are you setting a specific image size on your system? Also, it would be good to see output of free before the problematic hibernation starts. (In reply to comment #73) > It happened again now, but this time the counter froze at 64%. The counter > seems to advance slower with the patch applied, before it freezes. > > On a side note: I am running a 32-bit kernel with PAE (CONFIG_HIGHMEM64G=y) and > I have 4 GB physical memory. Do you think it matters? Should I try a 64-bit > kernel? Each kernel will present a slightly different memory situation during hibernation, so it may make a difference. But, it should work with both (that is the theory, at least). (In reply to comment #73) > It happened again now, but this time the counter froze at 64%. The counter > seems to advance slower with the patch applied, before it freezes. One more note. That the counter slows down is the sign that less and less free pages are available for buffering. So, smaller and smaller chunks are being repeatedly given to underlying I/O, therefore making hibernation slower. That is how old code used to work and that is the reason it was slow. I have been using kernel 3.3.0-8.fc16.i686 (32 bit, Fedora 16) on both a desktop PC and a Dell Latitude E6410 laptop for several days. Hibernate and resume work fine on both except that for resume to complete on the E6410 I need to use acpi=off in the kernel flags in the grub menu for resume only, not for booting. The Desktop PC does not need this special mode for resume. Both have 4GB RAM and swap partitions at least twice the size. 3.3 seems to have fixed things that were broken in 3.2. So it may be worth trying the latest 3.3 kernel. (In reply to comment #74) > 2. Something hardware specific is eating pages on your system during > hibernation. Or maybe some running process? > Question: are you setting a specific image size on your system? I don't know what that means, so I guess not? > Also, it would be good to see output of free before the problematic hibernation > starts. OK, I'll try to do that. (In reply to comment #78) > (In reply to comment #74) > > 2. Something hardware specific is eating pages on your system during > > hibernation. > > Or maybe some running process? This should not be the case. User processes should be frozen. > > Question: are you setting a specific image size on your system? > > I don't know what that means, so I guess not? It is /sys/power/image_size knob of hibernation. It is set to 2/5 of your RAM by default, but can be changed (up to 1/2). (In reply to comment #74) > 1. Pages are not freed as they should be when BIO chain is flushed. So, change > in buffering triggered this problem. I just did a quick test on my ThinkPad T510 to see whether this is the case and I could not reproduce it. Essentially, after each BIO chain flush during hibernation, the number of free pages would come back to the previous number (give or take a few). I did that with /dev/shm holding large amount of random data and with /sys/power/image_size set to half of RAM available on my box. So, at least on my system, pages used for hibernation buffering appear to be returned back to the kernel when BIO chain is flushed. Created attachment 575923 [details]
hogmem.c
This is as small program which hogs a chosen amount of memory. Run it like
./hogmem 1000 &
to hog 1000 MB memory.
(In reply to comment #71) > added this patch to all branches. will be in next updates. > Thanks Bojan. Feel free to remove it. As you can see from Per's responses, it only postponed the hang by a bit. Somehow, pages are being leaked, which then eventually causes hibernation to hang. I will work on reverting the I/O changes for image saving. Or I will come up with an asynchronous approach, so that we do not go back to the slowness we had before 3.2. I am somewhat mystified. I thought the problem of hibernate hanging had been solved in 3.3.0-8.fc16.i686 Using that kernel, pm-hibernate and resume now work very reliably for me on two different machines (one laptop, one desktop, both using i915) which previously repeatedly gave problems with hibernate freezing and resume rebooting. Both problems seem to be fixed (except that on the laptop I need acpi=off for resume to work reliably). Doesn't this make work on kernel 3.2 redundant? Apologies if I've misunderstood something? (In reply to comment #83) > I am somewhat mystified. I thought the problem of hibernate hanging had been > solved in 3.3.0-8.fc16.i686 > > Using that kernel, pm-hibernate and resume now work very reliably for me on two > different machines (one laptop, one desktop, both using i915) which previously > repeatedly gave problems with hibernate freezing and resume rebooting. Both > problems seem to be fixed (except that on the laptop I need acpi=off for resume > to work reliably). > > Doesn't this make work on kernel 3.2 redundant? Apologies if I've misunderstood > something? It is my understanding that Per is having hangs with all versions of the kernel that have latest I/O "improvements" done by yours truly. As such, I will either fix the breakage or revert that particular change until enough is understood as to why it may be happening. This will make hibernation part slower, but I would rather have that than hangs. Also, Per's troubles are not i915 related. We know that for sure too. PS. I sent some more patches to Per privately and we already know that when the buffering part of the hibernation changes in 3.2 is reverted, no more hangs occur. Of course, more testing is welcome, but thus far that's what we know. I also prepared and sent to Per another patch that may help, but without feedback (and I am sure most folks have better things to do than test patches around Easter), it is impossible to tell. Anyhow, any new developments will be posted here. (In reply to comment #84) > (In reply to comment #83) > > I am somewhat mystified. I thought the problem of hibernate hanging had been > > solved in 3.3.0-8.fc16.i686 > > > > Using that kernel, pm-hibernate and resume now work very reliably for me on two > > different machines (one laptop, one desktop, both using i915) which previously > > repeatedly gave problems with hibernate freezing and resume rebooting. Both > > problems seem to be fixed (except that on the laptop I need acpi=off for resume > > to work reliably). > > > > Doesn't this make work on kernel 3.2 redundant? Apologies if I've misunderstood > > something? > > It is my understanding that Per is having hangs with all versions of the kernel > that have latest I/O "improvements" done by yours truly. As such, I will either > fix the breakage or revert that particular change until enough is understood as > to why it may be happening. This will make hibernation part slower, but I would > rather have that than hangs. It doesn't hang on 3.3, but prints an error message about a page allocation failure (see attached photos). Hibernation continues though. If you don't see the error, that might be because your system is not configured to display any messages during hibernation. I don't know if the error message is a problem, but I would prefer things to work correctly without allocation failures. I also care about 3.2 because Debian's next release (wheezy) will use 3.2. I know this is Fedora's bug tracker, but this report is what I found when I googled for the problem. I can move the discussion to bugzilla.kernel.org if you want. (In reply to comment #85) Thanks for the explanations. I was mystified regarding tests based on an earlier kernel that had never worked properly for me, unlke 3.3. But you've explained the need to fix 3.2. > It doesn't hang on 3.3, but prints an error message about a page allocation > failure (see attached photos). Hibernation continues though. If you don't see > the error, that might be because your system is not configured to display any > messages during hibernation. I don't know if the error message is a problem, > but I would prefer things to work correctly without allocation failures. I don't get much chance to read the messages printed during hibernation because they go so fast. I am travelling now, but would be willing (if given detailed instructions, since I am not a developer) to try to get more data after I get home tomorrow. In case it is relevant here's the contents of /var/log/pmsuspend.log, using 3.3.0-8.fc16.i686 on Dell Latitude E6410: : Sun Apr 8 08:25:38 BST 2012: Running hooks for hibernate. Running hook /usr/lib/pm-utils/sleep.d/00logging hibernate hibernate: Linux lape 3.3.0-8.fc16.i686 #1 SMP Thu Mar 29 18:33:55 UTC 2012 i686 i686 i386 GNU/Linux Module Size Used by cdc_ncm 17952 0 usbnet 25791 1 cdc_ncm mii 13311 1 usbnet usb_storage 42728 0 xts 12687 8 gf128mul 13980 1 xts dm_crypt 22308 1 lp 17486 0 lockd 73543 0 fcoe 22673 0 libfcoe 41950 1 fcoe libfc 101889 2 fcoe,libfcoe scsi_transport_fc 47165 2 fcoe,libfc scsi_tgt 18993 1 scsi_transport_fc 8021q 23401 0 garp 13744 1 8021q stp 12719 1 garp llc 13770 2 garp,stp be2iscsi 62864 0 iscsi_boot_sysfs 15121 1 be2iscsi bnx2i 49424 0 cnic 57653 1 bnx2i uio 14374 1 cnic cxgb4i 32063 0 cxgb4 96268 1 cxgb4i cxgb3i 28014 0 libcxgbi 50487 2 cxgb4i,cxgb3i cxgb3 130826 1 cxgb3i mdio 13214 1 cxgb3 ib_iser 32863 0 rdma_cm 36864 1 ib_iser ib_cm 36679 1 rdma_cm iw_cm 13715 1 rdma_cm ib_sa 23625 2 rdma_cm,ib_cm ib_mad 37189 2 ib_cm,ib_sa ib_core 61900 6 ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad ib_addr 13473 1 rdma_cm iscsi_tcp 18015 0 libiscsi_tcp 19333 4 cxgb4i,cxgb3i,libcxgbi,iscsi_tcp libiscsi 44809 8 be2iscsi,bnx2i,cxgb4i,cxgb3i,libcxgbi,ib_iser,iscsi_tcp,libiscsi_tcp scsi_transport_iscsi 45361 8 be2iscsi,bnx2i,libcxgbi,ib_iser,iscsi_tcp,libiscsi ip6t_REJECT 12782 2 ip6t_ipv6header 12505 2 nf_conntrack_ipv6 13885 6 nf_defrag_ipv6 13678 1 nf_conntrack_ipv6 ip6table_filter 12711 1 ip6_tables 17844 2 ip6t_ipv6header,ip6table_filter nf_conntrack_netbios_ns 12585 0 nf_conntrack_broadcast 12487 1 nf_conntrack_netbios_ns nf_conntrack_ipv4 14182 2 nf_defrag_ipv4 12601 1 nf_conntrack_ipv4 xt_state 12514 8 nf_conntrack 70512 5 nf_conntrack_ipv6,nf_conntrack_netbios_ns,nf_conntrack_broadcast,nf_conntrack_ipv4,xt_state snd_hda_codec_hdmi 31469 1 snd_hda_codec_idt 63835 1 arc4 12473 2 ppdev 17363 0 dell_wmi 12601 0 sparse_keymap 13342 1 dell_wmi snd_hda_intel 32323 2 snd_hda_codec 102795 3 snd_hda_codec_hdmi,snd_hda_codec_idt,snd_hda_intel snd_hwdep 13236 1 snd_hda_codec uvcvideo 66834 0 videobuf2_core 27177 1 uvcvideo snd_seq 54637 0 videodev 86829 1 uvcvideo media 19719 2 uvcvideo,videodev videobuf2_vmalloc 12839 1 uvcvideo snd_seq_device 13817 1 snd_seq videobuf2_memops 13086 1 videobuf2_vmalloc dell_laptop 13671 0 dcdbas 14364 1 dell_laptop snd_pcm 81170 3 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec microcode 18642 0 iwlwifi 332511 0 mac80211 427444 1 iwlwifi joydev 17124 0 cfg80211 169437 2 iwlwifi,mac80211 rfkill 20417 2 dell_laptop,cfg80211 snd_timer 23896 2 snd_seq,snd_pcm snd 62853 13 snd_hda_codec_hdmi,snd_hda_codec_idt,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer soundcore 14116 1 snd snd_page_alloc 13709 2 snd_hda_intel,snd_pcm iTCO_wdt 17652 0 iTCO_vendor_support 13243 1 iTCO_wdt intel_ips 17920 0 e1000e 175665 0 parport_pc 27442 0 i2c_i801 17485 0 parport 39179 3 lp,ppdev,parport_pc sunrpc 196934 2 lockd mmc_block 26446 0 firewire_ohci 39585 0 firewire_core 55025 1 firewire_ohci crc_itu_t 12523 1 firewire_core sdhci_pci 18140 0 sdhci 32642 1 sdhci_pci mmc_core 97466 3 mmc_block,sdhci_pci,sdhci wmi 18273 1 dell_wmi i915 413229 7 drm_kms_helper 30903 1 i915 drm 199418 3 i915,drm_kms_helper i2c_algo_bit 12980 1 i915 i2c_core 28151 6 videodev,i2c_i801,i915,drm_kms_helper,drm,i2c_algo_bit video 18500 1 i915 total used free shared buffers cached Mem: 3548884 1312292 2236592 0 123572 410108 -/+ buffers/cache: 778612 2770272 Swap: 10489064 24324 10464740 /usr/lib/pm-utils/sleep.d/00logging hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/00powersave hibernate hibernate: /usr/lib/pm-utils/sleep.d/00powersave hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/01grub hibernate hibernate: /usr/lib/pm-utils/sleep.d/01grub hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/49bluetooth hibernate hibernate: /usr/lib/pm-utils/sleep.d/49bluetooth hibernate hibernate: not applicable. Running hook /usr/lib/pm-utils/sleep.d/55NetworkManager hibernate hibernate: Having NetworkManager put all interfaces to sleep...Done. /usr/lib/pm-utils/sleep.d/55NetworkManager hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/55wicd hibernate hibernate: /usr/lib/pm-utils/sleep.d/55wicd hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/56atd hibernate hibernate: /usr/lib/pm-utils/sleep.d/56atd hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/56dhclient hibernate hibernate: /usr/lib/pm-utils/sleep.d/56dhclient hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/75modules hibernate hibernate: Unloading kernel module vboxdrv...Done. /usr/lib/pm-utils/sleep.d/75modules hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/90clock hibernate hibernate: /usr/lib/pm-utils/sleep.d/90clock hibernate hibernate: not applicable. Running hook /usr/lib/pm-utils/sleep.d/94cpufreq hibernate hibernate: /usr/lib/pm-utils/sleep.d/94cpufreq hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/95led hibernate hibernate: /usr/lib/pm-utils/sleep.d/95led hibernate hibernate: not applicable. Running hook /usr/lib/pm-utils/sleep.d/95packagekit hibernate hibernate: /usr/lib/pm-utils/sleep.d/95packagekit hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/98video-quirk-db-handler hibernate hibernate: Kernel modesetting video driver detected, not using quirks. /usr/lib/pm-utils/sleep.d/98video-quirk-db-handler hibernate hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/99video hibernate hibernate: /usr/lib/pm-utils/sleep.d/99video hibernate hibernate: success. Sun Apr 8 08:25:40 BST 2012: performing hibernate Sun Apr 8 10:17:40 BST 2012: Awake. Sun Apr 8 10:17:40 BST 2012: Running hooks for thaw Running hook /usr/lib/pm-utils/sleep.d/99video thaw hibernate: /usr/lib/pm-utils/sleep.d/99video thaw hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/98video-quirk-db-handler thaw hibernate: /usr/lib/pm-utils/sleep.d/98video-quirk-db-handler thaw hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/95packagekit thaw hibernate: method return sender=:1.248 -> dest=:1.247 reply_serial=2 /usr/lib/pm-utils/sleep.d/95packagekit thaw hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/95led thaw hibernate: /usr/lib/pm-utils/sleep.d/95led thaw hibernate: not applicable. Running hook /usr/lib/pm-utils/sleep.d/94cpufreq thaw hibernate: /usr/lib/pm-utils/sleep.d/94cpufreq thaw hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/90clock thaw hibernate: /usr/lib/pm-utils/sleep.d/90clock thaw hibernate: not applicable. Running hook /usr/lib/pm-utils/sleep.d/75modules thaw hibernate: Reloaded unloaded modules. /usr/lib/pm-utils/sleep.d/75modules thaw hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/56dhclient thaw hibernate: /usr/lib/pm-utils/sleep.d/56dhclient thaw hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/56atd thaw hibernate: /usr/lib/pm-utils/sleep.d/56atd thaw hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/55wicd thaw hibernate: /usr/lib/pm-utils/sleep.d/55wicd thaw hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/55NetworkManager thaw hibernate: Having NetworkManager wake interfaces back up...Done. /usr/lib/pm-utils/sleep.d/55NetworkManager thaw hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/49bluetooth thaw hibernate: /usr/lib/pm-utils/sleep.d/49bluetooth thaw hibernate: not applicable. Running hook /usr/lib/pm-utils/sleep.d/01grub thaw hibernate: /usr/lib/pm-utils/sleep.d/01grub thaw hibernate: not applicable. Running hook /usr/lib/pm-utils/sleep.d/00powersave thaw hibernate: /usr/lib/pm-utils/sleep.d/00powersave thaw hibernate: success. Running hook /usr/lib/pm-utils/sleep.d/00logging thaw hibernate: /usr/lib/pm-utils/sleep.d/00logging thaw hibernate: success. Sun Apr 8 10:17:42 BST 2012: Finished. Note: the resume option in my grub.cfg uses acpi=off Otherwise it's the same as my boot command: ## menuentry 'Fedora (3.3.0-8.fc16.i686)RESUME' --class fedora --class gnu-linux --class gnu --class os { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod ext2 set root='(hd0,msdos6)' search --no-floppy --fs-uuid --set=root edcc4306-8eab-4f2e-990e-57520dfa2ab5 echo 'Loading Fedora (3.3.0-8.fc16.i686)' linux /vmlinuz-3.3.0-8.fc16.i686 root=UUID=29cae70a-ba5d-4178-8d93-30208331e510 ro rd.md=0 rd.lvm=0 rd.dm=0 SYSFONT=latarcyrheb-sun16 KEYTABLE=uk rd.luks=0 LANG=en_US.UTF-8 norhgb acpi=off echo 'Loading initial ramdisk ...' initrd /initramfs-3.3.0-8.fc16.i686.img } (In reply to comment #85) > It doesn't hang on 3.3, but prints an error message about a page allocation > failure (see attached photos). Hibernation continues though. I know it may sound silly, but that is not an error. The code expects to see an allocation failure there from time to time and it is supposed to continue. In fact, in the last patch I sent to you privately, this warning would not even appear on allocation failure, because we can live without that page, albeit slower. The hibernation code (including buffering) did not change substantially in 3.3, as compared to 3.2. So, something else, 3.2 specific, is triggering the page shortage. The last patch I sent privately does two things to avoid shortage: 1. Does not use memory from emergency reserves for buffering. 2. Does not retry failed allocations for buffering. This should help with getting around any shortage that may happen. Hopefully... Just giving my hello here too, I'm also from the corresponding Debian bug http://bugs.debian.org/659363 . I'm using debian backports linux-image-3.2.0-0.bpo.1-686-pae, 3.2.4-1~bpo60+1. However, I'm on an nVidia nForce 570 chipset (AMD A64-x2, 4G RAM), esp. using the sata_nv driver. Original symptoms were: - properly, fast working hibernate about 50% of the time - typical fail scenario, hibernating fast until 20-40%, then very abrupt and massive slowdown of writing speed, giving another 1% every 30s or 1min. This continues for several minutes, until endless sequences of allocation failures appear. (see netconsole output in debian bug.) I've applied the 2012-04-03 patch to my kernel last Friday, and it worked for the first 10 hibernates in a row, setting a new record for kernel 3.2. Since then I had another 7 hibernates without reboot, until the bug returned once yesterday, just as before. Image size is set to default 2/5, but the real image size got bigger after 2 days of usage, so maybe thats why it worked 17 times before. I'll try hogmem now. (In reply to comment #79) > (In reply to comment #78) > > (In reply to comment #74) > > > 2. Something hardware specific is eating pages on your system during > > > hibernation. > > > > Or maybe some running process? > > This should not be the case. User processes should be frozen. Regarding hardware, there is a message from e1000e printed during image writing. > > > > Question: are you setting a specific image size on your system? > > > > I don't know what that means, so I guess not? > > It is /sys/power/image_size knob of hibernation. It is set to 2/5 of your RAM > by default, but can be changed (up to 1/2). OK. I have not touched it. (In reply to comment #87) > (In reply to comment #85) > > > It doesn't hang on 3.3, but prints an error message about a page allocation > > failure (see attached photos). Hibernation continues though. > > I know it may sound silly, but that is not an error. The code expects to see > an allocation failure there from time to time and it is supposed to continue. > In fact, in the last patch I sent to you privately, this warning would not even > appear on allocation failure, because we can live without that page, albeit > slower. > > The hibernation code (including buffering) did not change substantially in 3.3, > as compared to 3.2. So, something else, 3.2 specific, is triggering the page > shortage. But there is still page shortage in 3.3, right? It just doesn't hang waiting for a page. And your patch makes it do so in 3.2 as well. There are some MM changes in 3.3, maybe that's why there's a difference. (In reply to comment #76) > (In reply to comment #73) > > It happened again now, but this time the counter froze at 64%. The counter > > seems to advance slower with the patch applied, before it freezes. > > One more note. That the counter slows down is the sign that less and less free > pages are available for buffering. So, smaller and smaller chunks are being > repeatedly given to underlying I/O, therefore making hibernation slower. That > is how old code used to work and that is the reason it was slow. But it was not *that* slow. At this instance, the counter paused for 10 seconds or so between 63% and 64%. Created attachment 576087 [details]
Better watermark the number of pages used by hibernation I/O
Cease use of kernel emergency reserves for buffering and do not retry failed allocations.
(In reply to comment #90) > But there is still page shortage in 3.3, right? It just doesn't hang waiting > for a page. And your patch makes it do so in 3.2 as well. We do not know where the hang happens. You should be getting exactly the same warning in 3.3 and 3.2 when pages cannot be allocated for hibernation buffering and the kernel should continue. So, the hang is likely somewhere else. What is surprising to me is that we ever actually get in such a tight situation with pages. I cannot replicate this on my system at all, because once we get down to three quarters of free pages, hibernation buffers get flushed (and pages freed). Somehow, that doesn't seem to be the case on your system - something else is taking pages up to the hilt. > There are some MM changes in 3.3, maybe that's why there's a difference. Maybe. Anyhow, I attached a patch similar to what I sent to you privately here. It should help, because it doesn't reach for emergency reserves any more. It also won't bother retrying failed page allocations. (In reply to comment #91) > But it was not *that* slow. At this instance, the counter paused for 10 seconds > or so between 63% and 64%. Yes, when memory gets super tight, flushing out all that I/O will take a long time. The theory is that we should never get ourselves into such a tight situation (in fact, hibernation buffers are flushed and pages returned to the kernel periodically), but obviously something else is allocating up the the hilt and then we get in trouble. (In reply to comment #88) > I've applied the 2012-04-03 patch to my kernel last Friday, and it worked for > the first 10 hibernates in a row, setting a new record for kernel 3.2. Please try the patch I attached now. It should be a lot more relaxed about memory allocation for buffering on hibernation. Created attachment 576205 [details]
photo of 3.2 hang backtrace
Backtrace made with magic sysrq.
Sorry Bojan, it failed again, even with your latest patch :-( I attached a photo of the backtrace made with magic sysrq. As you can see, it hangs somewhere in the slab code when trying to free pages. I'm still using 3.2.12, but I'll try with 3.2.14 to see if any related bug has been fixed. I will also try with a 64-bit kernel. (In reply to comment #97) > Sorry Bojan, it failed again, even with your latest patch :-( OK, thanks for the feedback. I think I will simply ask Rafael to revert that part of the patch for now. I'm no kernel hacker but... are you sure that nr_free_pages() is the right function to use? It seems to return the global number of free pages in all zones. But you don't use the high zone, right? Grep'ing through the kernel source, it seems that nr_free_pages() isn't used much, except for printing statistics. (In reply to comment #99) > I'm no kernel hacker Neither am I, actually. Just a user that got annoyed enough by slowness of in-kernel hibernation and decided to do something about it, given that suspend2 will probably never get merged (unfortunately). > but... are you sure that nr_free_pages() is the right > function to use? It seems to return the global number of free pages in all > zones. But you don't use the high zone, right? > > Grep'ing through the kernel source, it seems that nr_free_pages() isn't used > much, except for printing statistics. It is a macro defined in power.h. But you could be right. This did not occur to me at all. See, the previous code had a hard upper limit of between 512 or 1024 pages (depending on the size of the sector_t) that would get allocated for buffering. It never used nr_free_pages() for this purpose. I will have a look at some other VM info that is available. Sorry, in swap.h. Created attachment 576257 [details]
Better watermark the number of pages used by hibernation I/O
Here is a patch that only takes into account pages that are free in ZONE_NORMAL when calculating when to start flushing block I/O.
(In reply to comment #102) > Created attachment 576257 [details] > Better watermark the number of pages used by hibernation I/O > > Here is a patch that only takes into account pages that are free in ZONE_NORMAL > when calculating when to start flushing block I/O. Note that on e.g. x86_64 there is a big ZONE_DMA32 up to 4 GB. So I don't think that ZONE_NORMAL is what you want here. I think you need to sum the free pages in all zones except those which are is_highmem() or something like that. It would also be nice to have a patch which only changes nr_free_pages() and nothing else. It is harder to trigger the bug with the other changes (1/4 pages used and such). The other changes might not be needed either. (In reply to comment #100) > (In reply to comment #99) > > I'm no kernel hacker > > Neither am I, actually. Just a user that got annoyed enough by slowness of > in-kernel hibernation and decided to do something about it, given that suspend2 > will probably never get merged (unfortunately). OK :-) > > but... are you sure that nr_free_pages() is the right > > function to use? It seems to return the global number of free pages in all > > zones. But you don't use the high zone, right? > > > > Grep'ing through the kernel source, it seems that nr_free_pages() isn't used > > much, except for printing statistics. > > It is a macro defined in power.h. But you could be right. This did not occur to > me at all. Yes, and it reads from vm_stat, which seems to always be updated for all zones: http://lxr.linux.no/linux+v3.2.14/include/linux/vmstat.h#L90 Created attachment 576266 [details]
count only low memory pages
My patch applied on 3.2.14 passes my testcase. I will run with it during normal use for a while also and see what happens. (In reply to comment #103) > Note that on e.g. x86_64 there is a big ZONE_DMA32 up to 4 GB. So I don't think > that ZONE_NORMAL is what you want here. I think you need to sum the free pages > in all zones except those which are is_highmem() or something like that. That worked on my x86_64 box, but I see your point. > It would also be nice to have a patch which only changes nr_free_pages() and > nothing else. It is harder to trigger the bug with the other changes (1/4 pages > used and such). The other changes might not be needed either. Yeah, for proving the concept, yes. However, other changes do other things that I want to give to Rafael, because they were not quite right in the original version of my patch. (In reply to comment #106) > My patch applied on 3.2.14 passes my testcase. I will run with it during normal > use for a while also and see what happens. Excellent. Let me know when you know for sure and I'll ask Rafael to queue up a patch based on that. (In reply to comment #107) > Yeah, for proving the concept, yes. However, other changes do other things that > I want to give to Rafael, because they were not quite right in the original > version of my patch. OK. But I do think it would be good to use half of the free pages rather than a quarter of them. Especially for 32-bit kernels where low memory is limited to ~850 MB. (In reply to comment #109) > OK. But I do think it would be good to use half of the free pages rather than a > quarter of them. Especially for 32-bit kernels where low memory is limited to > ~850 MB. Yes. The original intention of a quarter was to limit initial pressure, because I thought something else may have eaten the pages at some point. With that being not true, I will leave it at half, as in my original patch. I will change >> 1 to / 2 though, because Rafael pointed out to me that such micro optimisations are really not required - compiler will do it anyway - and it affects readability. Thanks for staying on this and for finding the root cause of the problem! (In reply to comment #105) > Created attachment 576266 [details] > count only low memory pages @@ -1130,7 +1135,7 @@ static int load_image_lzo(struct swap_map_handle *handle, /* * Adjust number of pages for read buffering, in case we are short. */ - read_pages = (nr_free_pages() - snapshot_get_image_size()) >> 1; + read_pages = (nonhigh_free_pages() - snapshot_get_image_size()) >> 1; read_pages = clamp_val(read_pages, LZO_CMP_PAGES, LZO_READ_PAGES); for (i = 0; i < read_pages; i++) { BTW, the above is probably not what you want. The snapshot image also contains high pages. This calculation is relevant to reading of the image, so it won't affect the hibernation hang, but still... (In reply to comment #111) > BTW, the above is probably not what you want. The snapshot image also contains > high pages. Ah, ignore me. They all eventually have to fit into non-high memory. Created attachment 576333 [details]
Better watermark the number of pages used by hibernation I/O
Includes proper calculation of low free pages, as suggested by Per.
Hopefully final version of the patch submitted to Rafael here: http://marc.info/?l=linux-kernel&m=133402615419774&w=2 Just reporting back: I've been on the 2012-04-08 patch for a week now, no issues to report. No serious slowdowns of hibernation. I've switched to the v11-patch now, and continue to test. I also have set up permanent netconsole logging, in case anything happens. (In reply to comment #116) > Just reporting back: I've been on the 2012-04-08 patch for a week now, no > issues to report. No serious slowdowns of hibernation. > I've switched to the v11-patch now, and continue to test. I also have set up > permanent netconsole logging, in case anything happens. Thank you for testing! V11 patch only changes the calculation of pages on thaw, so it should not affect hibernation hangs. So, if v10 worked for you, there is a very good chance v11 will too. Tobi, is the problem you originally reported here fixed in the 3.4 kernel update? I do not know right now, since I started always using STR as a workaround when I encountered this bug. I'll switch to STD again for a while and keep you posted. This issue is not fixed in the 3.4 kernel. 3.4.2-1.fc16.x86_64 (In reply to comment #120) > This issue is not fixed in the 3.4 kernel. 3.4.2-1.fc16.x86_64 Given this is your first post in this bug, could you please tell us more about your particular problem. This information would be useful: - a photo of your screen when hibernation hangs - what kind of machine you have If I understand things correctly, most people that encountered this were on 32-bit kernels and the problem was incorrect calculation of free pages - something that has subsequently been fixed. You may also want to try the latest F-16 kernel: kernel-3.4.4-4.fc16. PS. In order to get a good photo of the hang, you may want to create a 00verbosity file in /etc/pm/sleep.d directory and put this in it, then hibernate as usual: ------------------------- #!/bin/sh case "$1" in hibernate) echo 7 > /proc/sys/kernel/printk ;; thaw) ;; *) esac ------------------------- My machine is an Asus Maximus motherboard with a core i7. I'll be happy to provide more details and logs and to try things as needed. Please forgive me if this is the wrong bug, I'll be happy to provide details to a different ticket. I tried pm-suspend and pm-hibernate just now. Both seem to power the machine down correctly and I can wake the computer by pressing the spacebar (as set in my bios). On wake up from pm-suspend, I don't see anything on the screen, it remains black. After waiting for some time, I press the reset button. When the machine boots it does an fsck as if the file system had been forcibly unmounted. Waking this machine from pm-suspend used to work perfectly. On wake up from pm-hibernate, I can see the bios messages, a short kernel message screen and then I can see the thaw process count down. After the count down process finishes, before my desktop appears, the computer reboots normally. (In reply to comment #122) From what you described, I think you are having a different problem than the one handled in this bug. This bug is about hangs during hibernation (caused by incorrect calculation of free pages) only. I guess https://bugzilla.redhat.com/show_bug.cgi?id=654107 would be the bug report with the symptoms that match mine. That bug seems to have been unchanged since 2010. > I'll switch to STD again for a while and keep you posted.
I've tested hibernation now some 5 times; it worked flawlessly each time. So this bug can be considered fixed. To everyone involved: thanks for the effort!
|