Bug 785384

Summary: hibernate hangs since kernel version 3.2.2-1.fc16.i686
Product: [Fedora] Fedora Reporter: Tobi Vollebregt <tobivollebregt>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 16CC: a.sloman, auxsvr, bojan, gansalmon, itamar, jonathan, kernel-maint, leif.andersson, madhu.chinakonda, marmalodak, michael, pelle, udo_richter
Target Milestone: ---Keywords: Patch, Reopened
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-09-05 14:20:22 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 781749    
Attachments:
Description Flags
photo of kernel messages after hibernate hangs
none
Half the number of pages used by hibernation I/O
none
Better watermark the number of pages used by hibernation I/O
none
Better watermark the number of pages used by hibernation I/O
none
Sleep 10 seconds after image write, so that photo can be taken
none
Better watermark the number of pages used by hibernation I/O
none
page allocation failure message photo (1)
none
page allocation failure message photo (2)
none
Better watermark the number of pages used by hibernation I/O
none
Better watermark the number of pages used by hibernation I/O
none
hogmem.c
none
Better watermark the number of pages used by hibernation I/O
none
photo of 3.2 hang backtrace
none
Better watermark the number of pages used by hibernation I/O
none
count only low memory pages
none
Better watermark the number of pages used by hibernation I/O none

Description Tobi Vollebregt 2012-01-28 14:20:10 UTC
Description of problem:

With kernel version 3.1.9-1.fc16.i686, both suspend (STR) and hibernate (STD) work fine on my Asus F3SG laptop.

With the update to kernel version 3.2.2-1.fc16.i686, suspend still works, but on hibernate, the machine appears to hang indefinitely on a black text-mode screen with only a cursor. At this point there appears to be no harddisk activity. The only way to escape appears to be to reset the machine by holding the power button for 4 seconds.


How reproducible: 100% (at least on my system - no idea about other systems)


Steps to Reproduce:
1. Boot with kernel 3.2.2-1.fc16.i686
2. Hibernate
3. Observe the system hang


Expected results:

I expected the system to hibernate succesfully, as it did with kernel version 3.1.9 and before, for as long as I remember.


Additional info:

Nothing in /var/log/messages - NetworkManager disconnects the wifi and then the log from the next boot starts.

Hardware is an ASUS F3SG laptop, with nvidia Geforce 7300mg graphics. As such, I am using the proprietary nvidia driver. Software is Fedora 16 upgraded twice using pre-upgrade, i.e. original installation was Fedora 14.

I can "work around" the issue by using kernel 3.1.9 for the moment.

Comment 1 Leif Andersson 2012-02-07 09:22:21 UTC
I just want to confirm the findings of Tobi Vollebregt, and add a tiny bit of information. I have turned off rhgb etc. in the kernel boot line, so I get some more information from the system. When I click on hibernate, I get a black screeen with a lot of logging running past too quickly to read. Then it stops on a line saying
[1234.5678] PM: Compressing and saving image data nn%
The nn is a number that increases, and when the hibernate worked, it went to 100%, and then the machine turned off. With the current kernel it will instead stop at a seemingly random number and stay there. Only hardware reset or similar will work.

uname -a:
Linux wimsey 3.2.2-1.fc16.i686.PAE #1 SMP Thu Jan 26 03:30:43 UTC 2012 i686 i686 i386 GNU/Linux

/proc/cpuinfo (beginning):
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz
stepping	: 6

The end of /var/log/pm-suspend.log:
/usr/lib/pm-utils/sleep.d/98video-quirk-db-handler hibernate hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/99video hibernate hibernate:

/usr/lib/pm-utils/sleep.d/99video hibernate hibernate: success.
Tue Feb  7 09:57:04 CET 2012: performing hibernate

Comment 2 Leif Andersson 2012-02-07 19:51:04 UTC
Problem solved, I think. I suddenly remembered that i had, and solved, the same problem under Fedora-15. I use Virtualbox, and its modules seem to do bad things with hibernate. The solution is then to make the system unload the Virtualbox module just before hibernating. This is achieved by putting a configuration file in the directory /etc/pm/config.d. In my case I name the file 'vboxdrv', and the content is the single line:
SUSPEND_MODULES=vboxdrv

After this hibernation works flawlessly. Strangely enough it actually did work without this configuration file in kernel 3.1.9, but the file should be there anyway.

To Tobi: I hope you can find a similar solution. If it is not Virtualbox, then perhaps you can find out by unloading any special module that you may have loaded.

The morally correct action now is to try to convince the Virtualbox people that they should include this little file in the distribution. We'll see.

Comment 3 Tobi Vollebregt 2012-02-07 20:27:39 UTC
Hmm, interesting. I do use virtualbox occasionally, but I do not appear to have its modules loaded currently. Don't know for sure whether I had them loaded with 3.2.2, though I think not.

Will try to check later if I can pull this trick with some other module.

Comment 4 Tobi Vollebregt 2012-02-13 09:24:03 UTC
Hibernate works again since kernel 3.2.5-3.fc16.i686 (or possibly one of the other packages that came in the upgrade).

Closing this bug.

Comment 5 aaronsloman 2012-02-22 00:13:59 UTC
This does not work for me. I have given details in http://forums.fedoraforum.org/showthread.php?p=1556879#post1556879

In my case pm-hibernate/resume works a few times, then freezes during hibernation, after reporting use of 3 threads for compression. I have tried all available 3.2 kernels in Fedora 16 up to 3.2.6-3. However  3.1.7-1 works fine, though more slowly.

Comment 6 Tobi Vollebregt 2012-02-26 13:03:14 UTC
You're, right, the problem is still present, with the note that with kernel 3.2.2 a hang seemed to be much more likely than with 3.2.5 and 3.2.6.

Past week, with 3.2.6, I had some 4 succesful hibernates, 5th time froze. Next boot the first hibernate froze.

Comment 7 aaronsloman 2012-02-27 00:53:50 UTC
I don't know if it will help anyone, but I took a couple of (not very clear) pictures of my laptop screen the last time it froze during pm_hibernate.

The lower half, showing the hibernate frozen at 69% is here: http://www.cs.bham.ac.uk/~axs/pics/laptop-screen-lower-closeup.jpg

The upper half, which seems to include some of the output of the last resume from hibernate is here: http://www.cs.bham.ac.uk/~axs/pics/laptop-screen-upper.jpg

The "acpi cmd .... rejected..." lines are presumably there because I found (after following a tip on the internet somewhere) that before resume from hibernate I need to add the flag 'acpi=off' to the boot command otherwise resume tended to fail and produced a reboot.

That was also necessary with 3.1 kernels in F16. It used not to be necessary in earlier versions of Fedora.

(I can't include the acpi=off command when doing a full reboot as that stops some things from working, e.g. getting information about battery state.) So I have two grub2 entries, one for boot, used rarely, and one for resume, the default. I tried removing that from my resume option and it did not prevent pm_hibernate failing.

I don't use virtualbox, so I suspect the apparent link mentioned above was just a coincidence.

My machine is a Dell Latitude E6410 with 4GB mem 500GB drive, 1440x900 screen using intel graphics (i9150) and intel integrated wireless. I use 32bit fedora 16.

The messages: "Dell_wmi: Received unknown WMI event" visible on the screen have been appearing with no known cause and no known harmful effects for many months, including, I think when I was running Fedora 13 and fedora 15. (I skipped F14 because I could not get the screen to work properly.)

Comment 8 Mike Heller 2012-02-27 21:25:51 UTC
I'm having exactly the same behaviour since kernel 3.2 that during hibernation system hang at " PM: Compressing and saving image data nn%" !

Comment 9 aaronsloman 2012-02-27 23:35:12 UTC
See similar bugreport here:
https://bugzilla.redhat.com/show_bug.cgi?id=789708

Comment 10 aaronsloman 2012-02-28 11:49:01 UTC
Using 3.2.7-1.fc16.i686
I found at first that pm-hibernate froze as previously. 

However, on a hunch I removed my SD card from the card-reader on the Dell E6410 (previously left in whether mounted or not, as a backup device). Since then I have hibernated and resumed 7 or 8 times without pm-hibernate freezing.

Later I'll try re-inserting the card and see if the freezing returns. For now I would prefer to see how long it works without freezing.

Comment 11 aaronsloman 2012-02-29 04:22:56 UTC
I tried reinserting the SD card. That seemed to cause pm-hibernate to freeze.

Of course, that might have been a fluke. Is there anything I can do, as a fairly ignorant user, to provide more information to help the developers ?

Comment 12 Mike Heller 2012-02-29 11:00:08 UTC
I can't imagine that only the SD card is the root of the problem.
because my desktop at home is malfunctioning without any additional cards such the harddisk which is a SSD.

Comment 13 aaronsloman 2012-02-29 14:36:58 UTC
Thanks for the comment. It sounds as if my experience with the SD card is a red-herring.

As I understand the situation, the main difference between hibernate in 3.1 and 3.2 kernels is that in 3.2 multiple threads are used for compression (on a multi-core cpu), making hibernation very much faster.

So maybe this bug does not affect anyone using a single core cpu? That might explain why so few people have joined this bug report?

Comment 14 Mike Heller 2012-02-29 21:14:27 UTC
the multi thread usage sounds much more plausible...

But for example on my netbook hibernation works flawless although it is also a "dual core cpu" and my desktop in the office works either.

Comment 15 Mike Heller 2012-03-01 19:37:18 UTC
so, I played arround a little bit to find out if it is the multi threaded hibernate or not.
So for this case I switched off my dual core cpu to a single core cpu via bios and it's the same behaviour... freezes during compressing the image with only one thread. aaaarrrghhh

now I will try to unload all modules manually...

Comment 16 aaronsloman 2012-03-01 20:46:01 UTC
Just to confirm that I got the freeze on hibernate without my SD card plugged in. So the earlier sequence of several non-freezes without card followed by freeze with card was just a coincidence.

Another thing I've wondered about: I don't use NetworkManager because I've never been able to get it to do what I want, whereas wicd (and wicd-client) have proved a lot more usable (also for several of my colleagues, who mostly use Ubuntu). Unfortunately it seems that some linux developers write code assuming that NM is used for connectivity (e.g. for a while firefox had that flaw). Is it possible that there's some troublesome interaction between wicd and pm-hibernate, or absence of NM and pm-hibernate?

If others reporting this bug are using NM, that conjecture is ruled out.

Comment 17 aaronsloman 2012-03-02 14:34:34 UTC
Alas, something has changed and I am now unable to get 3.1.7-1 to work properly: after hibernate+resume I get system errors.

I also tried the latest tuxonice kernel for F16. I previously used the tuxonice kernel until F15, after which it started freezing on resume for me. But that problem still remains.

I have today noticed that there's an update for kernel-tools for 3.2.7-1. So far have managed to pm-hibernate and resume and pm-hibernate again after installing that. Is there a place to find out what's changed in the new kernel-tools and why?

Comment 18 Dave Jones 2012-03-02 15:04:04 UTC
kernel-tools is a sub-package of the kernel (which is what changed).

The 3.2.6 build had a patch included that may have improved hibernate in some situations, but there's still a lot of work to do, as it's still broken for many users, including situations where it randomly corrupts memory.

Comment 19 aaronsloman 2012-03-02 22:46:55 UTC
Thanks very much for the feedback. It's very helpful to know this is being actively investigated by experts.

Comment 20 aaronsloman 2012-03-05 23:37:30 UTC
I've gone back to using 3.1.7-1.fc16.i686 which for a short time after one of the upgrades caused errors. But for several days it has been working fine for me
after I un-installed  3.2.7-1

The discussion about abandoning hibernate, in https://bugzilla.redhat.com/show_bug.cgi?id=781749 , is very alarming:

Comment 21 Per Olofsson 2012-03-13 16:47:30 UTC
I'm running Debian and also hit this bug, reported here:
http://bugs.debian.org/659363

If I leave my computer on for a while after hibernate has frozen, I get error messages from the kernel about hung tasks. I made a photo of them; will attach it.

To me it seems that it could be an I/O issue?

Bojan Smojver says that he made hibernate I/O "smarter" in 3.2:
https://bugzilla.redhat.com/show_bug.cgi?id=781749#c43

I have e-mailed him and asked him to take a look at this bug.

Comment 22 Per Olofsson 2012-03-13 16:48:20 UTC
Created attachment 569728 [details]
photo of kernel messages after hibernate hangs

Comment 23 Bojan Smojver 2012-03-13 20:56:52 UTC
(In reply to comment #21)
> I'm running Debian and also hit this bug, reported here:
> http://bugs.debian.org/659363
> 
> If I leave my computer on for a while after hibernate has frozen, I get error
> messages from the kernel about hung tasks. I made a photo of them; will attach
> it.
> 
> To me it seems that it could be an I/O issue?
> 
> Bojan Smojver says that he made hibernate I/O "smarter" in 3.2:
> https://bugzilla.redhat.com/show_bug.cgi?id=781749#c43
> 
> I have e-mailed him and asked him to take a look at this bug.

Looked at this and it doesn't seem like an I/O related bug from the screenshots. lzo_compress_threadfn() is doing compression in memory, not I/O. crc32_threadfn is calculating CRC32 in memory, so not I/O again.

Anyhow, you can use more or less the same I/O by passing hibernate=nocompress to the kernel. Threading and compression are not used in such a case.

However, when i915 corrupts memory on thaw, there is no telling what will get broken next. So, try doing you tests with nomodeset passed to the kernel. If you can reproduce the same thing, then i915 is not involved.

PS. The improvements in I/O of hibernation in 3.2 are related to better batching of pages that are read/written (more of them are queued up to be used by async I/O at any given time). Underlying I/O otherwise works the same.

Comment 24 Per Olofsson 2012-03-15 17:18:38 UTC
(In reply to comment #23)
> Looked at this and it doesn't seem like an I/O related bug from the
> screenshots. lzo_compress_threadfn() is doing compression in memory, not I/O.
> crc32_threadfn is calculating CRC32 in memory, so not I/O again.

I thought that, perhaps, they were waiting on input or output. If they were not waiting for anything, how could the kernel possibly detect them as "hung"?

> Anyhow, you can use more or less the same I/O by passing hibernate=nocompress
> to the kernel. Threading and compression are not used in such a case.

OK, thanks. Will try.

> However, when i915 corrupts memory on thaw, there is no telling what will get
> broken next. So, try doing you tests with nomodeset passed to the kernel. If
> you can reproduce the same thing, then i915 is not involved.

Will do. Note however that the bug reporter is using an Nvidia GPU. Also, I have never seen any segfaults the way others have when i915 corrupts memory.

> PS. The improvements in I/O of hibernation in 3.2 are related to better
> batching of pages that are read/written (more of them are queued up to be used
> by async I/O at any given time). Underlying I/O otherwise works the same.

I'm sure it's better. I'm just thinking that maybe it interacts badly with something else on my system (such as sleep/wake of SATA devices).

Comment 25 Bojan Smojver 2012-03-15 21:04:47 UTC
(In reply to comment #24)
 
> I thought that, perhaps, they were waiting on input or output. If they were not
> waiting for anything, how could the kernel possibly detect them as "hung"?

These threads are started when image saving starts and are kept alive until saving is done (they are signalled when data is ready for them, at which point they run their work function and sleep again). If anything else hangs the kernel in the meantime, they will be there, running. So, they will be reported as unfinished threads.

> Will do. Note however that the bug reporter is using an Nvidia GPU.

Proprietary driver there, so I doubt anyone will want to touch this with a ten foot pole.

> Also, I
> have never seen any segfaults the way others have when i915 corrupts memory.

Yeah, not everyone is seeing segfaults. Some are seeing OOPSes. Some kernel panics. I've seen all of those on my machine.

And, of course, the hang could be caused by things other than i915. And it could be caused by hibernation improvement code - I'm just not seeing evidence for that yet.

If you want to verify whether other kernel versions also hang your box, you can try this:

echo -n reboot > /sys/power/disk; for (( i=0; i<125; i++)); do echo $i; pm-hibernate; chvt 2; sleep 1; chvt 1; sleep 2; done

This will exercise hibernation/thaw repeatedly. On my system, I usually get in trouble after about 20 cycles. Sometimes after 50.
 
> I'm sure it's better. I'm just thinking that maybe it interacts badly with
> something else on my system (such as sleep/wake of SATA devices).

Possibly. Although none of the device code has been touched by threading and I/O improvements of hibernate in 3.2. We just keep deeper queues (i.e. more pages ready be written or read into) so that async I/O is fed more efficiently.

Comment 26 Bojan Smojver 2012-03-15 21:32:43 UTC
(In reply to comment #25)
 
> And it
> could be caused by hibernation improvement code - I'm just not seeing evidence
> for that yet.

If you feel like building the kernel, you can revert threading and I/O improvements, by reverting this commit: 081a9d043c983f161b78fdc4671324d1342b86bc.

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=081a9d043c983f161b78fdc4671324d1342b86bc

If it doesn't hang after that, then the problem is most definitely somewhere in there.

Comment 27 Bojan Smojver 2012-03-17 00:35:17 UTC
Created attachment 570742 [details]
Half the number of pages used by hibernation I/O

If anyone would like to play by compiling the kernel, this patch will reduce the number of pages used by hibernation code in half, so that hopefully other parts of the kernel have enough and do not hang waiting for free pages. Well, that's this flimsy theory at least.

Let me know whether it helps.

Comment 28 Per Olofsson 2012-03-19 15:54:22 UTC
I have now been able to reproduce this bug without i915 loaded, and without any virtualbox modules. I blacklisted i915 and turned off the script which loaded the vbox* modules. I think I can now safely say that this bug is not due to i915 corruption.

Here is what I do to reproduce the bug:

1. Pass hibernate=nocompress to the kernel.
2. Create a big file in shared memory. I used:
dd if=/dev/urandom of=/dev/shm/bigfile bs=1M count=750
3. echo disk >/sys/power/state
4. Watch the counter freeze.

This works even without i915 and X.

I will try to build the kernel with Bojan's commit reverted when I have time.

Comment 29 Bojan Smojver 2012-03-19 20:12:04 UTC
Created attachment 571226 [details]
Better watermark the number of pages used by hibernation I/O

Just a slightly better patch to try. This one tracks the number of free pages and flushes I/O when we fall below half of available pages.

Comment 30 Bojan Smojver 2012-03-19 21:26:38 UTC
(In reply to comment #28)
> I have now been able to reproduce this bug without i915 loaded, and without any
> virtualbox modules. I blacklisted i915 and turned off the script which loaded
> the vbox* modules. I think I can now safely say that this bug is not due to
> i915 corruption.

Thanks. Obviously a different bug and not the garden variety i915 corruption.

> Here is what I do to reproduce the bug:
> 
> 1. Pass hibernate=nocompress to the kernel.
> 2. Create a big file in shared memory. I used:
> dd if=/dev/urandom of=/dev/shm/bigfile bs=1M count=750
> 3. echo disk >/sys/power/state
> 4. Watch the counter freeze.

Tried this on my ThinkPad T510:
---------------------------
[root@shrek ~]# df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           3.9G   76K  3.9G   1% /dev/shm
[root@shrek ~]# dd if=/dev/urandom of=/dev/shm/bigfile bs=1M count=4000
dd: writing `/dev/shm/bigfile': No space left on device
3892+0 records in
3891+0 records out
4080844800 bytes (4.1 GB) copied, 301.019 s, 13.6 MB/s
[root@shrek ~]# df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           3.9G  3.9G     0 100% /dev/shm
[root@shrek ~]# echo -n reboot > /sys/power/disk;  for (( i=0; i<10; i++)); do echo $i; pm-hibernate; done
0
1
2
3
4
5
6
7
8
9
[root@shrek ~]# df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           3.9G  3.9G     0 100% /dev/shm
[root@shrek ~]# uname -r
3.2.10-3.fc16.x86_64
[root@shrek ~]# grep linux /boot/grub2/grub.cfg
### BEGIN /etc/grub.d/10_linux ###
menuentry 'Fedora (3.2.10-3.fc16.x86_64)' --class gnu-linux --class gnu --class os {
	linux	/vmlinuz-3.2.10-3.fc16.x86_64 root=/dev/mapper/vg00-lv00 ro SYSFONT=latarcyrheb-sun16 LANG=en_AU.UTF-8 KEYTABLE=us hibernate=nocompress nomodeset
---------------------------

So, cannot replicate it readily. How many tries did it take you to get a hang? I'm having a feeling this may be related to your particular hardware combo...

PS. I did the above test with i915 loaded as well. After 10th thaw, I got segfaults, followed by a kernel panic. Just FYI.

Comment 31 Per Olofsson 2012-03-19 21:48:07 UTC
(In reply to comment #30)
 > So, cannot replicate it readily. How many tries did it take you to get a hang?
> I'm having a feeling this may be related to your particular hardware combo...

I only need to hibernate once, it always happens the first time.

I also think it has something to do with the particular hardware. My guess is that there is a bug somewhere which is triggered by the deeper I/O queues. Maybe some ACPI weirdness with the SATA disks. When I hibernate, the kernel writes several messages related to ACPI sending commands to SATA disks:

[ 8391.960027] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 8392.015413] ata1.00: ACPI cmd ef/03:0c:00:00:00:a0 (SET FEATURES) filtered out
[ 8392.015417] ata1.00: ACPI cmd ef/03:45:00:00:00:a0 (SET FEATURES) filtered out
[ 8392.015450] ata1.00: ACPI cmd c6/00:10:00:00:00:a0 (SET MULTIPLE MODE) succeeded
[ 8392.015487] ata1.00: ACPI cmd e3/00:00:00:00:00:a0 (IDLE) succeeded
[ 8392.015490] ata1.00: ACPI cmd b1/c1:00:00:00:00:a0 (DEVICE CONFIGURATION OVERLAY) filtered out
[ 8392.015494] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out
[ 8392.015497] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out
[ 8392.016631] ata1.00: ACPI cmd ef/03:0c:00:00:00:a0 (SET FEATURES) filtered out
[ 8392.016636] ata1.00: ACPI cmd ef/03:45:00:00:00:a0 (SET FEATURES) filtered out
[ 8392.016665] ata1.00: ACPI cmd c6/00:10:00:00:00:a0 (SET MULTIPLE MODE) succeeded
[ 8392.016698] ata1.00: ACPI cmd e3/00:00:00:00:00:a0 (IDLE) succeeded
[ 8392.016702] ata1.00: ACPI cmd b1/c1:00:00:00:00:a0 (DEVICE CONFIGURATION OVERLAY) filtered out
[ 8392.016706] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out
[ 8392.016710] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out

Comment 32 Bojan Smojver 2012-03-19 22:08:32 UTC
(In reply to comment #31)

> I only need to hibernate once, it always happens the first time.

Right. So, very deterministic. At least some good news. Kind of... :-(

> I also think it has something to do with the particular hardware. My guess is
> that there is a bug somewhere which is triggered by the deeper I/O queues.
> Maybe some ACPI weirdness with the SATA disks. When I hibernate, the kernel
> writes several messages related to ACPI sending commands to SATA disks:

Possibly. Although this could also be a red herring.

Here is the suspend (hibernate) workflow from http://www.kernel.org/doc/Documentation/power/swsusp.txt:
============================
Suspend part
~~~~~~~~~~~~
running system, user asks for suspend-to-disk

user processes are stopped

suspend(PMSG_FREEZE): devices are frozen so that they don't interfere
                with state snapshot

state snapshot: copy of whole used memory is taken with interrupts disabled

resume(): devices are woken up so that we can write image to swap

write image to swap

suspend(PMSG_SUSPEND): suspend devices so that we can power off

turn the power off
============================

So, there is a resume just before the image is written to disk. If there is a bug in anything that's being resumed (i.e. various modules), it may hang the kernel at any time during the image writing phase.

Of course, once you revert my commit that changed buffering, we'll know for sure. In relation to that, the only thing I can think of is that some device that was resumed just ran out of pages (because we now use more for hibernation queues) and the kernel hangs waiting for more. But, then again, it's very hard to find bugs in your own code. :-(

Comment 33 Mike Heller 2012-03-20 07:55:31 UTC
(In reply to comment #32)
> > I only need to hibernate once, it always happens the first time.

same for me. happens every first time.
On the other hand if I add acpi=off as kernel parameter it fails on the second hibernate.

Comment 34 Mike Heller 2012-03-22 07:43:16 UTC
since updated to kernel 3.3 no more hangs during hibernation.

Comment 35 Per Olofsson 2012-03-22 15:47:58 UTC
OK, now I have done some more testing. All tests were done from the console, without compression, without i915 loaded, and with a big file in tmpfs.

Linux 3.2.12: Hibernate hangs.
Linux 3.2.12 with commit 081a9d043c983f161b78fdc4671324d1342b86bc reverted: Hibernate works fine
Linux 3.3.0: Hibernate works, but during image writing (when it usually hangs) the kernel prints an error message/oops related to memory pages. The hibernation continues, though.

The error messages flashes through too quickly for me to read. It also isn't present in dmesg after resuming. Is it possible to pause hibernation after writing the image but before shutting down the system?

(On a sidenote: when compression is disabled, one can use /dev/zero instead of /dev/urandom to write a big file on a tmpfs. It is much faster.)

Comment 36 Bojan Smojver 2012-03-22 20:56:30 UTC
Created attachment 572081 [details]
Better watermark the number of pages used by hibernation I/O

Same patch, v3.

Comment 37 Bojan Smojver 2012-03-22 21:00:51 UTC
(In reply to comment #35)
> OK, now I have done some more testing. All tests were done from the console,
> without compression, without i915 loaded, and with a big file in tmpfs.
> 
> Linux 3.2.12: Hibernate hangs.
> Linux 3.2.12 with commit 081a9d043c983f161b78fdc4671324d1342b86bc reverted:
> Hibernate works fine
> Linux 3.3.0: Hibernate works, but during image writing (when it usually hangs)
> the kernel prints an error message/oops related to memory pages. The
> hibernation continues, though.

Aha, so it does look like memory pressure causes this. I'm guessing some piece of hardware you have needs more pages, can't get them because they are used by hibernation buffers, which then causes the kernel to hang.

> The error messages flashes through too quickly for me to read. It also isn't
> present in dmesg after resuming. Is it possible to pause hibernation after
> writing the image but before shutting down the system?

Only if we insert some explicit code to sleep. I can get you a patch for that.

> (On a sidenote: when compression is disabled, one can use /dev/zero instead of
> /dev/urandom to write a big file on a tmpfs. It is much faster.)

Yeah, pages containing zeros compress pretty good. :-)

Anyhow, could you please put back commit 081a9d043c983f161b78fdc4671324d1342b86bc to 3.2.12, apply the patch that I attached here and try with that?

Thanks for testing!

Comment 38 Bojan Smojver 2012-03-22 22:31:14 UTC
Created attachment 572093 [details]
Sleep 10 seconds after image write, so that photo can be taken

Per,

You can try this patch for pausing after the image has been written. Let me know if it doesn't work for you and we can try something else.

Comment 39 aaronsloman 2012-03-22 22:36:19 UTC
Yesterday I installed vmlinuz-3.3.0-2.fc16.i686 on Dell Latitude D6410 (with
core i5 cpu and intel graphics: 1440x900).

First resume after hibernate caused a reboot, so I tried again using acpi=off
for resume. Since then it has hibernated and resumed several times without any
problem. (Single thread compression for pm-hibernate?) Used much of the day,
without any problems.

Tonight noticed that 3.3.0-4 was available, so installed it and tested.
Hibernates very fast (three threads for compression)

First hibernate+resume worked. Second one did a reboot after resume failed, so
inserted acpi=off for resume. And tried again. Resume crashed with segfaults
reported, but just in case I had done something wrong, re-started.

Since then have managed four or five hibernate+resume cycles (with acpi=off for
resume) and it seems to be working fine. (Could the earlier crash have been due
to something left over from earlier kernel. Perhaps I should clear swap are for
each new kernel?)

During boot and resume I get this, just before screen resolution changes:
kernel: [    7.038582] ata2: link is slow to respond, please be patient
(ready=0)

It pauses for a second or two before printing that then continues after another
second or so.

Will go back to 3.3.0-2 if I have any more problems, but so far looks fine.

Comment 40 Bojan Smojver 2012-03-22 22:42:36 UTC
(In reply to comment #39)
 
> During boot and resume I get this, just before screen resolution changes:
> kernel: [    7.038582] ata2: link is slow to respond, please be patient
> (ready=0)

Hmm, this looks like some kind of hardware problem (or driver problem) related to SATA.

> Could the earlier crash have been due to something left over from earlier kernel. Perhaps I should clear swap are for each new kernel?

This should never be the case. You should never need to manually clean anything.

Comment 41 aaronsloman 2012-03-23 00:18:01 UTC
(In reply to comment #40)

> > Could the earlier crash have been due to something left over from earlier kernel. Perhaps I should clear swap are for each new kernel?
> 
> This should never be the case. You should never need to manually clean
> anything.

I've done several more hibernate-resume cycles without any problem, using 
3.3.0-4

Perhaps on the occasion it went wrong I inadvertently resumed using the wrong kernel.

3.3. looks like a major advance. If only it was not necessary to have two grub.cfg entries, one with acpi=off for resume, that would be excellent. I expect most people would not tolerate having to edit grub.cfg.

Comment 42 Bojan Smojver 2012-03-23 00:28:02 UTC
(In reply to comment #41)

> 3.3. looks like a major advance.

Given that threading and buffering that I submitted in 3.2 did not change with 3.3 (apart from lack of required free space calculation when compression is used, which is before image writing starts, so irrelevant), I do not think your problem is the same as Per's problem.

> If only it was not necessary to have two
> grub.cfg entries, one with acpi=off for resume, that would be excellent. I
> expect most people would not tolerate having to edit grub.cfg.

The fact that turning ACPI off on resume is making a difference on your system would suggest that you are having hardware/driver issues. You should never need two different grub entries to hibernate/thaw.

PS. The reason hibernate/thaw works faster for you now is that Fedora patch to reduce the number of threads to 1 has been reverted. So, not related to 3.3.

Comment 43 Bojan Smojver 2012-03-23 00:58:43 UTC
Created attachment 572114 [details]
Better watermark the number of pages used by hibernation I/O

A more aggressive patch - required that 3/4 of pages be free at all times.

Comment 44 Mike Heller 2012-03-26 20:13:21 UTC
hibernating will not hang anymore with kernel 3.3.x but still a lot of troubles.

I saw several failures:

most of the time I go a "PM: LZO decompression failed/"Invalid LZO uncompressed length" and the machine booted instead of resuming.

on the other hand I got some kernel panics which I cannot post here. ( could not find anywhere on logs )

and sometimes call traces like this:

Mar 26 19:25:49  kernel: [ 8858.103137] Call Trace:
Mar 26 19:25:49  kernel: [ 8858.103154]  [<c058f530>] ? dquot_quota_off+0x20/0x20
Mar 26 19:25:49  kernel: [ 8858.103178]  [<c056dce7>] __sync_filesystem+0x87/0x90
Mar 26 19:25:49  kernel: [ 8858.103202]  [<c056dd07>] sync_one_sb+0x17/0x20
Mar 26 19:25:49  kernel: [ 8858.103226]  [<c0549c38>] iterate_supers+0xb8/0xc0
Mar 26 19:25:49  kernel: [ 8858.103249]  [<c056dcf0>] ? __sync_filesystem+0x90/0x90
Mar 26 19:25:49  kernel: [ 8858.103274]  [<c056dd9f>] sys_sync+0x3f/0x60
Mar 26 19:25:49  kernel: [ 8858.103299]  [<c0951cdf>] sysenter_do_call+0x12/0x28
Mar 26 19:25:49  kernel: [ 8858.103323] Code: 00 00 00 00 39 c3 8d b0 68 ff ff ff 75 1a eb 6c 8d 74 26 00 80 07 01 8b 86 98 00 00 00 39 45 bc 8d b0 68 ff ff ff 74 54 8d 7e 4c <8b> 5e 20 89 f8 e8 47 30 3e 00 f6 46 64 38 75 d9 8b 43 3c 85 c0
Mar 26 19:25:49  kernel: [ 8858.103704] EIP: [<c0567aff>] sync_inodes_sb+0xbf/0x150 SS:ESP 0068:ee7f9f20
Mar 26 19:25:49  kernel: [ 8858.103743] CR2: 00000000ffffff89
Mar 26 19:25:49  kernel: [ 8858.103761] ---[ end trace 94ea8aff47ec29aa ]---



However, I have also a desktop in the office with not troubles at all!!

Comment 45 aaronsloman 2012-03-26 21:31:21 UTC
(In reply to comment #44)
> hibernating will not hang anymore with kernel 3.3.x but still a lot of
> troubles.

I added a response to this in comment 9 in bug #789708

Comment 46 Bojan Smojver 2012-03-26 21:45:20 UTC
(In reply to comment #44)

> most of the time I go a "PM: LZO decompression failed/"Invalid LZO uncompressed
> length" and the machine booted instead of resuming.

If you are seeing this, the image data that was saved to disk is corrupt. This should never happen. Meaning, you are probably having hardware or driver problems in your I/O path and the image data is not being saved properly.

This is why we now check CRC32 of the image, but your thaw attempt didn't even make it that far.

> However, I have also a desktop in the office with not troubles at all!!

Yeah, more than likely hardware related.

Comment 47 aaronsloman 2012-03-26 22:42:06 UTC
(In reply to comment #46)
> (In reply to comment #44)
> 
> > most of the time I go a "PM: LZO decompression failed/"Invalid LZO uncompressed
> > length" and the machine booted instead of resuming.
> 
> If you are seeing this, the image data that was saved to disk is corrupt. This
> should never happen. Meaning, you are probably having hardware or driver
> problems in your I/O path and the image data is not being saved properly.

I have a question about this. Suppose, as happened to me, 'yum update' (apparently using 'grubby') inserted the wrong root partition ID in the grub.cfg menu entry for the new kernel. Then what would happen when booting? It would presumably use the new files in the /boot partition for 3.3.0-4 but in combination with an older collection of files in the old root partition (in my case a fedora 15 root partition). 

It seemed to boot OK when that happened to me, but at a later stage I got a variety of bizarre errors, which disappeared after I rebooted into an older kernel, edited the grub.cfg file to use the right partition for new kernel, then rebooted into the new kernel.

Could the errors reported in comment #44 be a result of running the new startup files in the wrong root partition?

This would of course only be possible if the /boot partition was separate from the root partition.

> However, I have also a desktop in the office with not troubles at all!!

This could simply be a manifestation of the semi-random nature of the errors made by grubby in creating grub.cfg -- it could depend on whether another partition had previously been used as a root partition, e.g. for an earlier version of fedora. (as on my laptop).

There are several reports of the bug in grubby, e.g. bug #751875
The buggy grubby seems to produce slightly different problems for different users.

Comment 48 Bojan Smojver 2012-03-26 22:52:53 UTC
(In reply to comment #47)

> I have a question about this. Suppose, as happened to me, 'yum update'
> (apparently using 'grubby') inserted the wrong root partition ID in the
> grub.cfg menu entry for the new kernel. Then what would happen when booting? It
> would presumably use the new files in the /boot partition for 3.3.0-4 but in
> combination with an older collection of files in the old root partition (in my
> case a fedora 15 root partition).

The hibernation image data is not stored on the root partition, it is stored in swap. This swap is specially marked as a partition that has a hibernation image (S1SUSPEND is the signature). Once that swap partition is identified, the image is read from there and what is being read is verified after decompression (that's how we know we got the correct stuff).

So, very unlikely that these two would be related.

Comment 49 aaronsloman 2012-03-26 23:46:10 UTC
(In reply to comment #48)

Thanks for rapid response.

> The hibernation image data is not stored on the root partition, it is stored in
> swap. This swap is specially marked as a partition that has a hibernation image
> (S1SUSPEND is the signature). Once that swap partition is identified, the image
> is read from there and what is being read is verified after decompression
> (that's how we know we got the correct stuff).
> 
> So, very unlikely that these two would be related.

But long before any hibernate or resume is attempted the system will have booted using libraries from an old root partition (/usr/lib, kernel headers, and others?) could this not mean that by the time hibernation begins many things have been corrupted which lead to errors?

Or is the hibernate/resume code sufficiently encapsulated as to be impermeable to any such corruption by mixing versions?

Unfortunately I did not keep detailed records of all the bizarre things that I experienced before I discovered that grubby was putting the wrong root UUID in grub.cfg. I think some of them involved  hibernate or resume, and others involved programs simply generating errors and aborting. I am reluctant to try deliberately booting with the wrong root partition uuid to investigate this!

Comment 50 Bojan Smojver 2012-03-26 23:55:39 UTC
(In reply to comment #49)
 
> Or is the hibernate/resume code sufficiently encapsulated as to be impermeable
> to any such corruption by mixing versions?

Hibernation/thaw that we are talking about here is done entirely by kernel code - no user libraries or other code is ever used. The image that is being thawed is stored entirely on the swap partition. Each compressed chunk is verified against obvious decompression errors. If any are detected, thaw will fail. Also, a CRC32 of uncompressed data is calculated and verified against what's been calculated/stored at hibernation time. If CRC32 is not the same, again thaw will fail.

So, short of your kernel RPM being somehow corrupt (meaning your /boot/vmlinuz-`uname -r` being changed by something), the userspace will have zero impact on hibernate/thaw.

Try:

rpm -qvf -V /boot/vmlinuz-`uname -r`  | grep vmlinuz

It should print all dots before the file name.

Comment 51 aaronsloman 2012-03-27 00:07:58 UTC
(In reply to comment #50)

> Try:
> 
> rpm -qvf -V /boot/vmlinuz-`uname -r`  | grep vmlinuz
> 
> It should print all dots before the file name.

Yes.

So if by the time hibernation starts there's anything corrupt in memory because the wrong root partition was used, it will simply be compressed and stored on swap, and then during resume it will all be uncompressed and the same corrupt system restored. But there will not be evidence during hibernate or resume of the corruption. 

Thanks for the clarification.

Comment 52 Mike Heller 2012-03-27 05:50:14 UTC
(In reply to comment #46)
> (In reply to comment #44)
> If you are seeing this, the image data that was saved to disk is corrupt. This
> should never happen. Meaning, you are probably having hardware or driver
> problems in your I/O path and the image data is not being saved properly.
> 
> This is why we now check CRC32 of the image, but your thaw attempt didn't even
> make it that far.
> 
> > However, I have also a desktop in the office with not troubles at all!!
> 
> Yeah, more than likely hardware related.

Hmm... the main diffrence is the amount of RAM ( 2GB in the office and 6GB at home ) and radeon graphic driver instead of i915 at home.

However is there anything I can afford to find the issue faster? or do I just lean back and wait if a furhter update solve the problem?

Comment 53 Bojan Smojver 2012-03-27 06:18:33 UTC
(In reply to comment #52)

> Hmm... the main diffrence is the amount of RAM ( 2GB in the office and 6GB at
> home ) and radeon graphic driver instead of i915 at home.
> 
> However is there anything I can afford to find the issue faster? or do I just
> lean back and wait if a furhter update solve the problem?

Try looking for similar kernel traces reported by other folks. This may give you clues.

You may have different disk controllers in these two machine, which may be causing curruption problems. A guess only...

Comment 54 Mike Heller 2012-03-27 09:39:59 UTC
(In reply to comment #53)

> 
> You may have different disk controllers in these two machine, which may be
> causing curruption problems. A guess only...

yes, this is also true.
I'm havin an SSD at home and a standard sata disk at offices desktop.

is it helpful to try also with a sata disk on my failing computer?

Comment 55 Bojan Smojver 2012-03-27 11:59:00 UTC
(In reply to comment #54)
> (In reply to comment #53)
> 
> > 
> > You may have different disk controllers in these two machine, which may be
> > causing curruption problems. A guess only...
> 
> yes, this is also true.
> I'm havin an SSD at home and a standard sata disk at offices desktop.
> 
> is it helpful to try also with a sata disk on my failing computer?

I do not know for sure. It may make a difference.

Comment 56 Bojan Smojver 2012-03-27 15:17:42 UTC
(In reply to comment #48)

> So, very unlikely that these two would be related.

At the risk of pointing out the obvious, let me just qualify this a bit.

If you have two OSes "sharing" file systems and you hibernate one, then boot the other, you are risking file system corruption. Also, if you hibernate one and then the other somehow modifies the swap where hibernation image is written, all bets are off.

http://www.kernel.org/doc/Documentation/power/swsusp.txt

Comment 57 Mike Heller 2012-03-27 21:11:39 UTC
so, actually I removed 4 gigs of RAM ( only 2GB left ) and did some tests.

I hibernate/thaw 5 times my computer with the commandline loop described in post #25 with success.
then I worked again with my computer and did this loop again. also with success.

so, does this depend on the amount of RAM and/or the amount of RAM devices inserted? 
maybe it is also another red herring...

Comment 58 Bojan Smojver 2012-03-27 21:37:51 UTC
(In reply to comment #57)
 
> so, does this depend on the amount of RAM and/or the amount of RAM devices
> inserted?

The more RAM you have, the more is being used for buffering during hibernation, to improve performance. Of course, if more memory is used by the kernel, swap space needs to be bigger to hold a larger image as well.

I have 8 GB in my laptop and that never caused any particular problems for me.

So, yeah, it could be a red herring. Then again, does your machine pass memory stress tests with all the RAM?

Comment 59 aaronsloman 2012-03-28 08:48:12 UTC
(In reply to comment #58)

> I have 8 GB in my laptop and that never caused any particular problems for me.

Is there a recommended minimum ratio between swap size and RAM for hibernate to work? There used to be a general recommendation to have at least twice as much swap size as RAM for running unix, e.g. to allow forking of large processes.

I expect hibernate does not require as much as that. Presumably it will provide a warning and abort if there is not enough swap space. I have never seen that happen. The man file for pm-utils does not mention swap space, surprisingly.

Comment 60 Bojan Smojver 2012-03-28 09:47:57 UTC
(In reply to comment #59)
> (In reply to comment #58)
> 
> > I have 8 GB in my laptop and that never caused any particular problems for me.
> 
> Is there a recommended minimum ratio between swap size and RAM for hibernate to
> work? There used to be a general recommendation to have at least twice as much
> swap size as RAM for running unix, e.g. to allow forking of large processes.
> 
> I expect hibernate does not require as much as that. Presumably it will provide
> a warning and abort if there is not enough swap space. I have never seen that
> happen. The man file for pm-utils does not mention swap space, surprisingly.

In kernel hibernation can save at most half of RAM. Default I think is 2/5 (but check the source ob this). In 3.2, size of the swap is checked when compression is used. In 3.3, it is not - hibernation fails when space is short. When compression is not used, size is always checked.

Comment 61 Josh Boyer 2012-03-28 18:03:11 UTC
[Mass hibernate bug update]

Dave Airlied has found an issue causing some corruption in the i915 fbdev after a resume from hibernate.  I have included his patch in this scratch build:

http://koji.fedoraproject.org/koji/taskinfo?taskID=3940545

This will probably not solve all of the issues being tracked at the moment, but it is worth testing when the build completes.  If this seems to clear up the issues you see with hibernate, please report your results in the bug.

Comment 62 Per Olofsson 2012-03-28 21:14:19 UTC
Created attachment 573466 [details]
page allocation failure message photo (1)

Comment 63 Per Olofsson 2012-03-28 21:14:58 UTC
Created attachment 573467 [details]
page allocation failure message photo (2)

Comment 64 Per Olofsson 2012-03-28 21:18:20 UTC
Hi,

I have now finally had time to take photos. Sorry for the bad quality, my camera is not very good (nor the lighting).

I guess I would be able to capture the message through a serial console also. I do not seem to have a null modem cable, but I can get one if it helps.

Bojan, do you want me to try your latest patch also?

Comment 65 Bojan Smojver 2012-03-29 03:05:53 UTC
(In reply to comment #64)
 
> I have now finally had time to take photos. Sorry for the bad quality, my
> camera is not very good (nor the lighting).

Never mind the quality - they give precisely the information that is required.

> I guess I would be able to capture the message through a serial console also. I
> do not seem to have a null modem cable, but I can get one if it helps.

Nah, don't worry about that now. Images will do.

> Bojan, do you want me to try your latest patch also?

Yes, please. I will attach the latest version, which will land in 3.5.

Comment 66 Bojan Smojver 2012-03-29 03:06:49 UTC
Created attachment 573507 [details]
Better watermark the number of pages used by hibernation I/O

Version of the patch that is queued for 3.5 by Rafael.

Comment 67 Bojan Smojver 2012-04-03 05:51:43 UTC
Created attachment 574743 [details]
Better watermark the number of pages used by hibernation I/O

One more tweak to the patch: enable super-paranoia mode and flush BIO chain when number of free pages is equal or less than required number of free pages. Just in case required free pages (somehow) turns up being zero.

Comment 68 Per Olofsson 2012-04-05 10:03:20 UTC
I can confirm that your latest patch works. 3.2.12 no longer freezes, and 3.3 no longer prints an error.

Comment 69 Bojan Smojver 2012-04-05 10:11:16 UTC
(In reply to comment #68)
> I can confirm that your latest patch works. 3.2.12 no longer freezes, and 3.3
> no longer prints an error.

Thank you very much for testing. Truly appreciated!

I will ask Rafael to queue up for 3.4 and stable as well.

Comment 70 Per Olofsson 2012-04-05 10:15:14 UTC
(In reply to comment #69)
> Thank you very much for testing. Truly appreciated!

And thank you for fixing the problem for me! :-)

Comment 71 Dave Jones 2012-04-05 15:52:02 UTC
added this patch to all branches. will be in next updates.
Thanks Bojan.

Comment 72 Per Olofsson 2012-04-06 08:42:59 UTC
Sorry, I was too quick.

The problem still exists in 3.2 with the patch. It's just that my testcase no longer triggers the bug. The bug is still sometimes triggered during normal use. The only difference is that the counter goes up higher before freezing (over 90%). I also let the computer on for a while and I then received messages like those in the first attached photo.

Comment 73 Per Olofsson 2012-04-06 11:17:51 UTC
It happened again now, but this time the counter froze at 64%. The counter seems to advance slower with the patch applied, before it freezes.

On a side note: I am running a 32-bit kernel with PAE (CONFIG_HIGHMEM64G=y) and I have 4 GB physical memory. Do you think it matters? Should I try a 64-bit kernel?

Comment 74 Bojan Smojver 2012-04-06 11:34:20 UTC
(In reply to comment #72)
> Sorry, I was too quick.
> 
> The problem still exists in 3.2 with the patch. It's just that my testcase no
> longer triggers the bug. The bug is still sometimes triggered during normal
> use. The only difference is that the counter goes up higher before freezing
> (over 90%). I also let the computer on for a while and I then received messages
> like those in the first attached photo.

OK, thanks for the feedback.

What we know for sure is that kernel runs out of pages during hibernation. I see two possibilities:

1. Pages are not freed as they should be when BIO chain is flushed. So, change in buffering triggered this problem.

2. Something hardware specific is eating pages on your system during hibernation.

I will try to create a debugging patch, so that we can determine where the pages went.

Question: are you setting a specific image size on your system?

Also, it would be good to see output of free before the problematic hibernation starts.

Comment 75 Bojan Smojver 2012-04-06 11:36:22 UTC
(In reply to comment #73)
> It happened again now, but this time the counter froze at 64%. The counter
> seems to advance slower with the patch applied, before it freezes.
> 
> On a side note: I am running a 32-bit kernel with PAE (CONFIG_HIGHMEM64G=y) and
> I have 4 GB physical memory. Do you think it matters? Should I try a 64-bit
> kernel?

Each kernel will present a slightly different memory situation during hibernation, so it may make a difference. But, it should work with both (that is the theory, at least).

Comment 76 Bojan Smojver 2012-04-06 11:40:19 UTC
(In reply to comment #73)
> It happened again now, but this time the counter froze at 64%. The counter
> seems to advance slower with the patch applied, before it freezes.

One more note. That the counter slows down is the sign that less and less free pages are available for buffering. So, smaller and smaller chunks are being repeatedly given to underlying I/O, therefore making hibernation slower. That is how old code used to work and that is the reason it was slow.

Comment 77 aaronsloman 2012-04-06 11:50:34 UTC
I have been using kernel 3.3.0-8.fc16.i686 (32 bit, Fedora 16) on both a desktop PC and a Dell Latitude E6410 laptop for several days. Hibernate and resume work fine on both except that for resume to complete on the E6410 I need to use acpi=off in the kernel flags in the grub menu for resume only, not for booting.

The Desktop PC does not need this special mode for resume. Both have 4GB RAM and swap partitions at least twice the size. 3.3 seems to have fixed things that were broken in 3.2.

So it may be worth trying the latest 3.3 kernel.

Comment 78 Per Olofsson 2012-04-06 20:56:55 UTC
(In reply to comment #74)
> 2. Something hardware specific is eating pages on your system during
> hibernation.

Or maybe some running process?

> Question: are you setting a specific image size on your system?

I don't know what that means, so I guess not?

> Also, it would be good to see output of free before the problematic hibernation
> starts.

OK, I'll try to do that.

Comment 79 Bojan Smojver 2012-04-06 23:43:46 UTC
(In reply to comment #78)
> (In reply to comment #74)
> > 2. Something hardware specific is eating pages on your system during
> > hibernation.
> 
> Or maybe some running process?

This should not be the case. User processes should be frozen.

> > Question: are you setting a specific image size on your system?
> 
> I don't know what that means, so I guess not?

It is /sys/power/image_size knob of hibernation. It is set to 2/5 of your RAM by default, but can be changed (up to 1/2).

Comment 80 Bojan Smojver 2012-04-07 04:44:49 UTC
(In reply to comment #74)
 
> 1. Pages are not freed as they should be when BIO chain is flushed. So, change
> in buffering triggered this problem.

I just did a quick test on my ThinkPad T510 to see whether this is the case and I could not reproduce it. Essentially, after each BIO chain flush during hibernation, the number of free pages would come back to the previous number (give or take a few).

I did that with /dev/shm holding large amount of random data and with /sys/power/image_size set to half of RAM available on my box.

So, at least on my system, pages used for hibernation buffering appear to be returned back to the kernel when BIO chain is flushed.

Comment 81 Per Olofsson 2012-04-07 13:07:56 UTC
Created attachment 575923 [details]
hogmem.c

This is as small program which hogs a chosen amount of memory. Run it like

./hogmem 1000 &

to hog 1000 MB memory.

Comment 82 Bojan Smojver 2012-04-08 00:23:01 UTC
(In reply to comment #71)
> added this patch to all branches. will be in next updates.
> Thanks Bojan.

Feel free to remove it. As you can see from Per's responses, it only postponed the hang by a bit. Somehow, pages are being leaked, which then eventually causes hibernation to hang.

I will work on reverting the I/O changes for image saving. Or I will come up with an asynchronous approach, so that we do not go back to the slowness we had before 3.2.

Comment 83 aaronsloman 2012-04-08 06:37:12 UTC
I am somewhat mystified. I thought the problem of hibernate hanging had been solved in 3.3.0-8.fc16.i686

Using that kernel, pm-hibernate and resume now work very reliably for me on two different machines (one laptop, one desktop, both using i915) which previously repeatedly gave problems with hibernate freezing and resume rebooting. Both problems seem to be fixed (except that on the laptop I need acpi=off for resume to work reliably).

Doesn't this make work on kernel 3.2 redundant? Apologies if I've misunderstood something?

Comment 84 Bojan Smojver 2012-04-08 08:47:50 UTC
(In reply to comment #83)
> I am somewhat mystified. I thought the problem of hibernate hanging had been
> solved in 3.3.0-8.fc16.i686
> 
> Using that kernel, pm-hibernate and resume now work very reliably for me on two
> different machines (one laptop, one desktop, both using i915) which previously
> repeatedly gave problems with hibernate freezing and resume rebooting. Both
> problems seem to be fixed (except that on the laptop I need acpi=off for resume
> to work reliably).
> 
> Doesn't this make work on kernel 3.2 redundant? Apologies if I've misunderstood
> something?

It is my understanding that Per is having hangs with all versions of the kernel that have latest I/O "improvements" done by yours truly. As such, I will either fix the breakage or revert that particular change until enough is understood as to why it may be happening. This will make hibernation part slower, but I would rather have that than hangs.

Also, Per's troubles are not i915 related. We know that for sure too.

PS. I sent some more patches to Per privately and we already know that when the buffering part of the hibernation changes in 3.2 is reverted, no more hangs occur. Of course, more testing is welcome, but thus far that's what we know. I also prepared and sent to Per another patch that may help, but without feedback (and I am sure most folks have better things to do than test patches around Easter), it is impossible to tell. Anyhow, any new developments will be posted here.

Comment 85 Per Olofsson 2012-04-08 09:47:04 UTC
(In reply to comment #84)
> (In reply to comment #83)
> > I am somewhat mystified. I thought the problem of hibernate hanging had been
> > solved in 3.3.0-8.fc16.i686
> > 
> > Using that kernel, pm-hibernate and resume now work very reliably for me on two
> > different machines (one laptop, one desktop, both using i915) which previously
> > repeatedly gave problems with hibernate freezing and resume rebooting. Both
> > problems seem to be fixed (except that on the laptop I need acpi=off for resume
> > to work reliably).
> > 
> > Doesn't this make work on kernel 3.2 redundant? Apologies if I've misunderstood
> > something?
> 
> It is my understanding that Per is having hangs with all versions of the kernel
> that have latest I/O "improvements" done by yours truly. As such, I will either
> fix the breakage or revert that particular change until enough is understood as
> to why it may be happening. This will make hibernation part slower, but I would
> rather have that than hangs.

It doesn't hang on 3.3, but prints an error message about a page allocation failure (see attached photos). Hibernation continues though. If you don't see the error, that might be because your system is not configured to display any messages during hibernation. I don't know if the error message is a problem, but I would prefer things to work correctly without allocation failures.

I also care about 3.2 because Debian's next release (wheezy) will use 3.2. I know this is Fedora's bug tracker, but this report is what I found when I googled for the problem. I can move the discussion to bugzilla.kernel.org if you want.

Comment 86 aaronsloman 2012-04-08 10:11:40 UTC
(In reply to comment #85)

Thanks for the explanations. I was mystified regarding tests based on an
earlier kernel that had never worked properly for me, unlke 3.3. But you've
explained the need to fix 3.2.

> It doesn't hang on 3.3, but prints an error message about a page allocation
> failure (see attached photos). Hibernation continues though. If you don't see
> the error, that might be because your system is not configured to display any
> messages during hibernation. I don't know if the error message is a problem,
> but I would prefer things to work correctly without allocation failures.

I don't get much chance to read the messages printed during hibernation because
they go so fast. I am travelling now, but would be willing (if given detailed
instructions, since I am not a developer) to try to get more data after I get
home tomorrow.

In case it is relevant here's the contents of /var/log/pmsuspend.log, using
3.3.0-8.fc16.i686 on Dell Latitude E6410:

:
Sun Apr  8 08:25:38 BST 2012: Running hooks for hibernate.
Running hook /usr/lib/pm-utils/sleep.d/00logging hibernate hibernate:
Linux lape 3.3.0-8.fc16.i686 #1 SMP Thu Mar 29 18:33:55 UTC 2012 i686 i686 i386
GNU/Linux
Module                  Size  Used by
cdc_ncm                17952  0
usbnet                 25791  1 cdc_ncm
mii                    13311  1 usbnet
usb_storage            42728  0
xts                    12687  8
gf128mul               13980  1 xts
dm_crypt               22308  1
lp                     17486  0
lockd                  73543  0
fcoe                   22673  0
libfcoe                41950  1 fcoe
libfc                 101889  2 fcoe,libfcoe
scsi_transport_fc      47165  2 fcoe,libfc
scsi_tgt               18993  1 scsi_transport_fc
8021q                  23401  0
garp                   13744  1 8021q
stp                    12719  1 garp
llc                    13770  2 garp,stp
be2iscsi               62864  0
iscsi_boot_sysfs       15121  1 be2iscsi
bnx2i                  49424  0
cnic                   57653  1 bnx2i
uio                    14374  1 cnic
cxgb4i                 32063  0
cxgb4                  96268  1 cxgb4i
cxgb3i                 28014  0
libcxgbi               50487  2 cxgb4i,cxgb3i
cxgb3                 130826  1 cxgb3i
mdio                   13214  1 cxgb3
ib_iser                32863  0
rdma_cm                36864  1 ib_iser
ib_cm                  36679  1 rdma_cm
iw_cm                  13715  1 rdma_cm
ib_sa                  23625  2 rdma_cm,ib_cm
ib_mad                 37189  2 ib_cm,ib_sa
ib_core                61900  6 ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
ib_addr                13473  1 rdma_cm
iscsi_tcp              18015  0
libiscsi_tcp           19333  4 cxgb4i,cxgb3i,libcxgbi,iscsi_tcp
libiscsi               44809  8
be2iscsi,bnx2i,cxgb4i,cxgb3i,libcxgbi,ib_iser,iscsi_tcp,libiscsi_tcp
scsi_transport_iscsi    45361  8
be2iscsi,bnx2i,libcxgbi,ib_iser,iscsi_tcp,libiscsi
ip6t_REJECT            12782  2
ip6t_ipv6header        12505  2
nf_conntrack_ipv6      13885  6
nf_defrag_ipv6         13678  1 nf_conntrack_ipv6
ip6table_filter        12711  1
ip6_tables             17844  2 ip6t_ipv6header,ip6table_filter
nf_conntrack_netbios_ns    12585  0
nf_conntrack_broadcast    12487  1 nf_conntrack_netbios_ns
nf_conntrack_ipv4      14182  2
nf_defrag_ipv4         12601  1 nf_conntrack_ipv4
xt_state               12514  8
nf_conntrack           70512  5
nf_conntrack_ipv6,nf_conntrack_netbios_ns,nf_conntrack_broadcast,nf_conntrack_ipv4,xt_state
snd_hda_codec_hdmi     31469  1
snd_hda_codec_idt      63835  1
arc4                   12473  2
ppdev                  17363  0
dell_wmi               12601  0
sparse_keymap          13342  1 dell_wmi
snd_hda_intel          32323  2
snd_hda_codec         102795  3
snd_hda_codec_hdmi,snd_hda_codec_idt,snd_hda_intel
snd_hwdep              13236  1 snd_hda_codec
uvcvideo               66834  0
videobuf2_core         27177  1 uvcvideo
snd_seq                54637  0
videodev               86829  1 uvcvideo
media                  19719  2 uvcvideo,videodev
videobuf2_vmalloc      12839  1 uvcvideo
snd_seq_device         13817  1 snd_seq
videobuf2_memops       13086  1 videobuf2_vmalloc
dell_laptop            13671  0
dcdbas                 14364  1 dell_laptop
snd_pcm                81170  3 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec
microcode              18642  0
iwlwifi               332511  0
mac80211              427444  1 iwlwifi
joydev                 17124  0
cfg80211              169437  2 iwlwifi,mac80211
rfkill                 20417  2 dell_laptop,cfg80211
snd_timer              23896  2 snd_seq,snd_pcm
snd                    62853  13
snd_hda_codec_hdmi,snd_hda_codec_idt,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer
soundcore              14116  1 snd
snd_page_alloc         13709  2 snd_hda_intel,snd_pcm
iTCO_wdt               17652  0
iTCO_vendor_support    13243  1 iTCO_wdt
intel_ips              17920  0
e1000e                175665  0
parport_pc             27442  0
i2c_i801               17485  0
parport                39179  3 lp,ppdev,parport_pc
sunrpc                196934  2 lockd
mmc_block              26446  0
firewire_ohci          39585  0
firewire_core          55025  1 firewire_ohci
crc_itu_t              12523  1 firewire_core
sdhci_pci              18140  0
sdhci                  32642  1 sdhci_pci
mmc_core               97466  3 mmc_block,sdhci_pci,sdhci
wmi                    18273  1 dell_wmi
i915                  413229  7
drm_kms_helper         30903  1 i915
drm                   199418  3 i915,drm_kms_helper
i2c_algo_bit           12980  1 i915
i2c_core               28151  6
videodev,i2c_i801,i915,drm_kms_helper,drm,i2c_algo_bit
video                  18500  1 i915
             total       used       free     shared    buffers     cached
Mem:       3548884    1312292    2236592          0     123572     410108
-/+ buffers/cache:     778612    2770272
Swap:     10489064      24324   10464740

/usr/lib/pm-utils/sleep.d/00logging hibernate hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/00powersave hibernate hibernate:

/usr/lib/pm-utils/sleep.d/00powersave hibernate hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/01grub hibernate hibernate:

/usr/lib/pm-utils/sleep.d/01grub hibernate hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/49bluetooth hibernate hibernate:

/usr/lib/pm-utils/sleep.d/49bluetooth hibernate hibernate: not applicable.
Running hook /usr/lib/pm-utils/sleep.d/55NetworkManager hibernate hibernate:
Having NetworkManager put all interfaces to sleep...Done.

/usr/lib/pm-utils/sleep.d/55NetworkManager hibernate hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/55wicd hibernate hibernate:

/usr/lib/pm-utils/sleep.d/55wicd hibernate hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/56atd hibernate hibernate:

/usr/lib/pm-utils/sleep.d/56atd hibernate hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/56dhclient hibernate hibernate:

/usr/lib/pm-utils/sleep.d/56dhclient hibernate hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/75modules hibernate hibernate:
Unloading kernel module vboxdrv...Done.

/usr/lib/pm-utils/sleep.d/75modules hibernate hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/90clock hibernate hibernate:

/usr/lib/pm-utils/sleep.d/90clock hibernate hibernate: not applicable.
Running hook /usr/lib/pm-utils/sleep.d/94cpufreq hibernate hibernate:

/usr/lib/pm-utils/sleep.d/94cpufreq hibernate hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/95led hibernate hibernate:

/usr/lib/pm-utils/sleep.d/95led hibernate hibernate: not applicable.
Running hook /usr/lib/pm-utils/sleep.d/95packagekit hibernate hibernate:

/usr/lib/pm-utils/sleep.d/95packagekit hibernate hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/98video-quirk-db-handler hibernate
hibernate:
Kernel modesetting video driver detected, not using quirks.

/usr/lib/pm-utils/sleep.d/98video-quirk-db-handler hibernate hibernate:
success.
Running hook /usr/lib/pm-utils/sleep.d/99video hibernate hibernate:

/usr/lib/pm-utils/sleep.d/99video hibernate hibernate: success.
Sun Apr  8 08:25:40 BST 2012: performing hibernate
Sun Apr  8 10:17:40 BST 2012: Awake.
Sun Apr  8 10:17:40 BST 2012: Running hooks for thaw
Running hook /usr/lib/pm-utils/sleep.d/99video thaw hibernate:

/usr/lib/pm-utils/sleep.d/99video thaw hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/98video-quirk-db-handler thaw hibernate:

/usr/lib/pm-utils/sleep.d/98video-quirk-db-handler thaw hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/95packagekit thaw hibernate:
method return sender=:1.248 -> dest=:1.247 reply_serial=2

/usr/lib/pm-utils/sleep.d/95packagekit thaw hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/95led thaw hibernate:

/usr/lib/pm-utils/sleep.d/95led thaw hibernate: not applicable.
Running hook /usr/lib/pm-utils/sleep.d/94cpufreq thaw hibernate:

/usr/lib/pm-utils/sleep.d/94cpufreq thaw hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/90clock thaw hibernate:

/usr/lib/pm-utils/sleep.d/90clock thaw hibernate: not applicable.
Running hook /usr/lib/pm-utils/sleep.d/75modules thaw hibernate:
Reloaded unloaded modules.

/usr/lib/pm-utils/sleep.d/75modules thaw hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/56dhclient thaw hibernate:

/usr/lib/pm-utils/sleep.d/56dhclient thaw hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/56atd thaw hibernate:

/usr/lib/pm-utils/sleep.d/56atd thaw hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/55wicd thaw hibernate:

/usr/lib/pm-utils/sleep.d/55wicd thaw hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/55NetworkManager thaw hibernate:
Having NetworkManager wake interfaces back up...Done.

/usr/lib/pm-utils/sleep.d/55NetworkManager thaw hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/49bluetooth thaw hibernate:

/usr/lib/pm-utils/sleep.d/49bluetooth thaw hibernate: not applicable.
Running hook /usr/lib/pm-utils/sleep.d/01grub thaw hibernate:

/usr/lib/pm-utils/sleep.d/01grub thaw hibernate: not applicable.
Running hook /usr/lib/pm-utils/sleep.d/00powersave thaw hibernate:

/usr/lib/pm-utils/sleep.d/00powersave thaw hibernate: success.
Running hook /usr/lib/pm-utils/sleep.d/00logging thaw hibernate:

/usr/lib/pm-utils/sleep.d/00logging thaw hibernate: success.
Sun Apr  8 10:17:42 BST 2012: Finished.

Note: the resume option in my grub.cfg uses acpi=off
Otherwise it's the same as my boot command:
##
menuentry 'Fedora (3.3.0-8.fc16.i686)RESUME' --class fedora --class gnu-linux
--class gnu --class os {
 load_video
 set gfxpayload=keep
 insmod gzio
 insmod part_msdos
 insmod ext2
 set root='(hd0,msdos6)'
 search --no-floppy --fs-uuid --set=root edcc4306-8eab-4f2e-990e-57520dfa2ab5
 echo 'Loading Fedora (3.3.0-8.fc16.i686)'
 linux /vmlinuz-3.3.0-8.fc16.i686
root=UUID=29cae70a-ba5d-4178-8d93-30208331e510 ro rd.md=0 rd.lvm=0 rd.dm=0
SYSFONT=latarcyrheb-sun16 KEYTABLE=uk rd.luks=0 LANG=en_US.UTF-8 norhgb
acpi=off
 echo 'Loading initial ramdisk ...'
 initrd /initramfs-3.3.0-8.fc16.i686.img
}

Comment 87 Bojan Smojver 2012-04-08 11:23:43 UTC
(In reply to comment #85)

> It doesn't hang on 3.3, but prints an error message about a page allocation
> failure (see attached photos). Hibernation continues though.

I know it may sound silly, but that is not an error. The code  expects to see an allocation failure there from time to time and it is supposed to continue. In fact, in the last patch I sent to you privately, this warning would not even appear on allocation failure, because we can live without that page, albeit slower.

The hibernation code (including buffering) did not change substantially in 3.3, as compared to 3.2. So, something else, 3.2 specific, is triggering the page shortage. The last patch I sent privately does two things to avoid shortage:

1. Does not use memory from emergency reserves for buffering.

2. Does not retry failed allocations for buffering.

This should help with getting around any shortage that may happen. Hopefully...

Comment 88 Udo Richter 2012-04-08 16:12:10 UTC
Just giving my hello here too, I'm also from the corresponding Debian bug http://bugs.debian.org/659363 .

I'm using debian backports linux-image-3.2.0-0.bpo.1-686-pae, 3.2.4-1~bpo60+1. However, I'm on an nVidia nForce 570 chipset (AMD A64-x2, 4G RAM), esp. using the sata_nv driver. Original symptoms were:
- properly, fast working hibernate about 50% of the time
- typical fail scenario, hibernating fast until 20-40%, then very abrupt and massive slowdown of writing speed, giving another 1% every 30s or 1min. This continues for several minutes, until endless sequences of allocation failures appear. (see netconsole output in debian bug.)


I've applied the 2012-04-03 patch to my kernel last Friday, and it worked for the first 10 hibernates in a row, setting a new record for kernel 3.2. Since then I had another 7 hibernates without reboot, until the bug returned once yesterday, just as before. Image size is set to default 2/5, but the real image size got bigger after 2 days of usage, so maybe thats why it worked 17 times before.

I'll try hogmem now.

Comment 89 Per Olofsson 2012-04-08 16:54:29 UTC
(In reply to comment #79)
> (In reply to comment #78)
> > (In reply to comment #74)
> > > 2. Something hardware specific is eating pages on your system during
> > > hibernation.
> > 
> > Or maybe some running process?
> 
> This should not be the case. User processes should be frozen.

Regarding hardware, there is a message from e1000e printed during image writing.

> 
> > > Question: are you setting a specific image size on your system?
> > 
> > I don't know what that means, so I guess not?
> 
> It is /sys/power/image_size knob of hibernation. It is set to 2/5 of your RAM
> by default, but can be changed (up to 1/2).

OK. I have not touched it.

Comment 90 Per Olofsson 2012-04-08 20:35:43 UTC
(In reply to comment #87)
> (In reply to comment #85)
> 
> > It doesn't hang on 3.3, but prints an error message about a page allocation
> > failure (see attached photos). Hibernation continues though.
> 
> I know it may sound silly, but that is not an error. The code  expects to see
> an allocation failure there from time to time and it is supposed to continue.
> In fact, in the last patch I sent to you privately, this warning would not even
> appear on allocation failure, because we can live without that page, albeit
> slower.
> 
> The hibernation code (including buffering) did not change substantially in 3.3,
> as compared to 3.2. So, something else, 3.2 specific, is triggering the page
> shortage.

But there is still page shortage in 3.3, right? It just doesn't hang waiting for a page. And your patch makes it do so in 3.2 as well.

There are some MM changes in 3.3, maybe that's why there's a difference.

Comment 91 Per Olofsson 2012-04-08 20:39:16 UTC
(In reply to comment #76)
> (In reply to comment #73)
> > It happened again now, but this time the counter froze at 64%. The counter
> > seems to advance slower with the patch applied, before it freezes.
> 
> One more note. That the counter slows down is the sign that less and less free
> pages are available for buffering. So, smaller and smaller chunks are being
> repeatedly given to underlying I/O, therefore making hibernation slower. That
> is how old code used to work and that is the reason it was slow.

But it was not *that* slow. At this instance, the counter paused for 10 seconds or so between 63% and 64%.

Comment 92 Bojan Smojver 2012-04-08 23:15:26 UTC
Created attachment 576087 [details]
Better watermark the number of pages used by hibernation I/O

Cease use of kernel emergency reserves for buffering and do not retry failed allocations.

Comment 93 Bojan Smojver 2012-04-08 23:21:51 UTC
(In reply to comment #90)
 
> But there is still page shortage in 3.3, right? It just doesn't hang waiting
> for a page. And your patch makes it do so in 3.2 as well.

We do not know where the hang happens. You should be getting exactly the same warning in 3.3 and 3.2 when pages cannot be allocated for hibernation buffering and the kernel should continue. So, the hang is likely somewhere else.

What is surprising to me is that we ever actually get in such a tight situation with pages. I cannot replicate this on my system at all, because once we get down to three quarters of free pages, hibernation buffers get flushed (and pages freed). Somehow, that doesn't seem to be the case on your system - something else is taking pages up to the hilt.

> There are some MM changes in 3.3, maybe that's why there's a difference.

Maybe.

Anyhow, I attached a patch similar to what I sent to you privately here. It should help, because it doesn't reach for emergency reserves any more. It also won't bother retrying failed page allocations.

Comment 94 Bojan Smojver 2012-04-08 23:25:39 UTC
(In reply to comment #91)
 
> But it was not *that* slow. At this instance, the counter paused for 10 seconds
> or so between 63% and 64%.

Yes, when memory gets super tight, flushing out all that I/O will take a long time. The theory is that we should never get ourselves into such a tight situation (in fact, hibernation buffers are flushed and pages returned to the kernel periodically), but obviously something else is allocating up the the hilt and then we get in trouble.

Comment 95 Bojan Smojver 2012-04-08 23:27:16 UTC
(In reply to comment #88)
 
> I've applied the 2012-04-03 patch to my kernel last Friday, and it worked for
> the first 10 hibernates in a row, setting a new record for kernel 3.2.

Please try the patch I attached now. It should be a lot more relaxed about memory allocation for buffering on hibernation.

Comment 96 Per Olofsson 2012-04-09 12:48:17 UTC
Created attachment 576205 [details]
photo of 3.2 hang backtrace

Backtrace made with magic sysrq.

Comment 97 Per Olofsson 2012-04-09 12:50:19 UTC
Sorry Bojan, it failed again, even with your latest patch :-(

I attached a photo of the backtrace made with magic sysrq. As you can see, it hangs somewhere in the slab code when trying to free pages.

I'm still using 3.2.12, but I'll try with 3.2.14 to see if any related bug has been fixed. I will also try with a 64-bit kernel.

Comment 98 Bojan Smojver 2012-04-09 12:58:46 UTC
(In reply to comment #97)
> Sorry Bojan, it failed again, even with your latest patch :-(

OK, thanks for the feedback. I think I will simply ask Rafael to revert that part of the patch for now.

Comment 99 Per Olofsson 2012-04-09 13:11:39 UTC
I'm no kernel hacker but... are you sure that nr_free_pages() is the right function to use? It seems to return the global number of free pages in all zones. But you don't use the high zone, right?

Grep'ing through the kernel source, it seems that nr_free_pages() isn't used much, except for printing statistics.

Comment 100 Bojan Smojver 2012-04-09 14:47:50 UTC
(In reply to comment #99)
> I'm no kernel hacker

Neither am I, actually. Just a user that got annoyed enough by slowness of in-kernel hibernation and decided to do something about it, given that suspend2 will probably never get merged (unfortunately).

> but... are you sure that nr_free_pages() is the right
> function to use? It seems to return the global number of free pages in all
> zones. But you don't use the high zone, right?
> 
> Grep'ing through the kernel source, it seems that nr_free_pages() isn't used
> much, except for printing statistics.

It is a macro defined in power.h. But you could be right. This did not occur to me at all.

See, the previous code had a hard upper limit of between 512 or 1024 pages (depending on the size of the sector_t) that would get allocated for buffering. It never used nr_free_pages() for this purpose.

I will have a look at some other VM info that is available.

Comment 101 Bojan Smojver 2012-04-09 14:53:11 UTC
Sorry, in swap.h.

Comment 102 Bojan Smojver 2012-04-09 16:16:14 UTC
Created attachment 576257 [details]
Better watermark the number of pages used by hibernation I/O

Here is a patch that only takes into account pages that are free in ZONE_NORMAL when calculating when to start flushing block I/O.

Comment 103 Per Olofsson 2012-04-09 17:00:34 UTC
(In reply to comment #102)
> Created attachment 576257 [details]
> Better watermark the number of pages used by hibernation I/O
> 
> Here is a patch that only takes into account pages that are free in ZONE_NORMAL
> when calculating when to start flushing block I/O.

Note that on e.g. x86_64 there is a big ZONE_DMA32 up to 4 GB. So I don't think that ZONE_NORMAL is what you want here. I think you need to sum the free pages in all zones except those which are is_highmem() or something like that.

It would also be nice to have a patch which only changes nr_free_pages() and nothing else. It is harder to trigger the bug with the other changes (1/4 pages used and such). The other changes might not be needed either.

Comment 104 Per Olofsson 2012-04-09 17:04:50 UTC
(In reply to comment #100)
> (In reply to comment #99)
> > I'm no kernel hacker
> 
> Neither am I, actually. Just a user that got annoyed enough by slowness of
> in-kernel hibernation and decided to do something about it, given that suspend2
> will probably never get merged (unfortunately).

OK :-)

> > but... are you sure that nr_free_pages() is the right
> > function to use? It seems to return the global number of free pages in all
> > zones. But you don't use the high zone, right?
> > 
> > Grep'ing through the kernel source, it seems that nr_free_pages() isn't used
> > much, except for printing statistics.
> 
> It is a macro defined in power.h. But you could be right. This did not occur to
> me at all.

Yes, and it reads from vm_stat, which seems to always be updated for all zones:

http://lxr.linux.no/linux+v3.2.14/include/linux/vmstat.h#L90

Comment 105 Per Olofsson 2012-04-09 17:25:30 UTC
Created attachment 576266 [details]
count only low memory pages

Comment 106 Per Olofsson 2012-04-09 18:52:12 UTC
My patch applied on 3.2.14 passes my testcase. I will run with it during normal use for a while also and see what happens.

Comment 107 Bojan Smojver 2012-04-09 22:07:46 UTC
(In reply to comment #103)
 
> Note that on e.g. x86_64 there is a big ZONE_DMA32 up to 4 GB. So I don't think
> that ZONE_NORMAL is what you want here. I think you need to sum the free pages
> in all zones except those which are is_highmem() or something like that.

That worked on my x86_64 box, but I see your point.

> It would also be nice to have a patch which only changes nr_free_pages() and
> nothing else. It is harder to trigger the bug with the other changes (1/4 pages
> used and such). The other changes might not be needed either.

Yeah, for proving the concept, yes. However, other changes do other things that I want to give to Rafael, because they were not quite right in the original version of my patch.

Comment 108 Bojan Smojver 2012-04-09 22:08:32 UTC
(In reply to comment #106)
> My patch applied on 3.2.14 passes my testcase. I will run with it during normal
> use for a while also and see what happens.

Excellent. Let me know when you know for sure and I'll ask Rafael to queue up a patch based on that.

Comment 109 Per Olofsson 2012-04-09 22:25:56 UTC
(In reply to comment #107)
> Yeah, for proving the concept, yes. However, other changes do other things that
> I want to give to Rafael, because they were not quite right in the original
> version of my patch.

OK. But I do think it would be good to use half of the free pages rather than a
quarter of them. Especially for 32-bit kernels where low memory is limited to
~850 MB.

Comment 110 Bojan Smojver 2012-04-09 22:34:42 UTC
(In reply to comment #109)
 
> OK. But I do think it would be good to use half of the free pages rather than a
> quarter of them. Especially for 32-bit kernels where low memory is limited to
> ~850 MB.

Yes. The original intention of a quarter was to limit initial pressure, because I thought something else may have eaten the pages at some point. With that being not true, I will leave it at half, as in my original patch. I will change >> 1 to / 2 though, because Rafael pointed out to me that such micro optimisations are really not required - compiler will do it anyway - and it affects readability.

Thanks for staying on this and for finding the root cause of the problem!

Comment 111 Bojan Smojver 2012-04-09 22:43:29 UTC
(In reply to comment #105)
> Created attachment 576266 [details]
> count only low memory pages

@@ -1130,7 +1135,7 @@ static int load_image_lzo(struct swap_map_handle *handle,
 	/*
 	 * Adjust number of pages for read buffering, in case we are short.
 	 */
-	read_pages = (nr_free_pages() - snapshot_get_image_size()) >> 1;
+	read_pages = (nonhigh_free_pages() - snapshot_get_image_size()) >> 1;
 	read_pages = clamp_val(read_pages, LZO_CMP_PAGES, LZO_READ_PAGES);
 
 	for (i = 0; i < read_pages; i++) {

BTW, the above is probably not what you want. The snapshot image also contains high pages.

This calculation is relevant to reading of the image, so it won't affect the hibernation hang, but still...

Comment 112 Bojan Smojver 2012-04-09 23:04:11 UTC
(In reply to comment #111)

> BTW, the above is probably not what you want. The snapshot image also contains
> high pages.

Ah, ignore me. They all eventually have to fit into non-high memory.

Comment 113 Bojan Smojver 2012-04-09 23:26:19 UTC
Created attachment 576333 [details]
Better watermark the number of pages used by hibernation I/O

Includes proper calculation of low free pages, as suggested by Per.

Comment 114 Bojan Smojver 2012-04-10 03:04:28 UTC
Hopefully final version of the patch submitted to Rafael here:

http://marc.info/?l=linux-kernel&m=133402615419774&w=2

Comment 115 Bojan Smojver 2012-04-12 22:33:24 UTC
One more:

http://marc.info/?l=linux-kernel&m=133426808115262&w=2

Comment 116 Udo Richter 2012-04-15 16:17:35 UTC
Just reporting back: I've been on the 2012-04-08 patch for a week now, no issues to report. No serious slowdowns of hibernation. 
I've switched to the v11-patch now, and continue to test. I also have set up permanent netconsole logging, in case anything happens.

Comment 117 Bojan Smojver 2012-04-15 23:58:13 UTC
(In reply to comment #116)
> Just reporting back: I've been on the 2012-04-08 patch for a week now, no
> issues to report. No serious slowdowns of hibernation. 
> I've switched to the v11-patch now, and continue to test. I also have set up
> permanent netconsole logging, in case anything happens.

Thank you for testing!

V11 patch only changes the calculation of pages on thaw, so it should not affect hibernation hangs. So, if v10 worked for you, there is a very good chance v11 will too.

Comment 118 Dave Jones 2012-07-16 18:23:17 UTC
Tobi, is the problem you originally reported here fixed in the 3.4 kernel update?

Comment 119 Tobi Vollebregt 2012-07-16 18:45:46 UTC
I do not know right now, since I started always using STR as a workaround when I encountered this bug.

I'll switch to STD again for a while and keep you posted.

Comment 120 John Schmitt 2012-07-16 19:29:09 UTC
This issue is not fixed in the 3.4 kernel.  3.4.2-1.fc16.x86_64

Comment 121 Bojan Smojver 2012-07-16 21:30:22 UTC
(In reply to comment #120)
> This issue is not fixed in the 3.4 kernel.  3.4.2-1.fc16.x86_64

Given this is your first post in this bug, could you please tell us more about your particular problem. This information would be useful:

- a photo of your screen when hibernation hangs
- what kind of machine you have

If I understand things correctly, most people that encountered this were on 32-bit kernels and the problem was incorrect calculation of free pages - something that has subsequently been fixed.

You may also want to try the latest F-16 kernel: kernel-3.4.4-4.fc16.

PS. In order to get a good photo of the hang, you may want to create a 00verbosity file in /etc/pm/sleep.d directory and put this in it, then hibernate as usual:
-------------------------
#!/bin/sh
case "$1" in
	hibernate)
		echo 7 > /proc/sys/kernel/printk
		;;
	thaw)
		;;
	*)
esac
-------------------------

Comment 122 John Schmitt 2012-07-17 02:02:53 UTC
My machine is an Asus Maximus motherboard with a core i7.  I'll be happy to provide more details and logs and to try things as needed.  Please forgive me if this is the wrong bug, I'll be happy to provide details to a different ticket.

I tried pm-suspend and pm-hibernate just now. Both seem to power the machine down correctly and I can wake the computer by pressing the spacebar (as set in my bios).

On wake up from pm-suspend, I don't see anything on the screen, it remains black.  After waiting for some time, I press the reset button.  When the machine boots it does an fsck as if the file system had been forcibly unmounted.  Waking this machine from pm-suspend used to work perfectly.  

On wake up from pm-hibernate, I can see the bios messages, a short kernel message screen and then I can see the thaw process count down.  After the count down process finishes, before my desktop appears, the computer reboots normally.

Comment 123 Bojan Smojver 2012-07-17 02:28:27 UTC
(In reply to comment #122)

From what you described, I think you are having a different problem than the one handled in this bug. This bug is about hangs during hibernation (caused by incorrect calculation of free pages) only.

Comment 124 John Schmitt 2012-07-17 04:09:18 UTC
I guess https://bugzilla.redhat.com/show_bug.cgi?id=654107 would be the bug report with the symptoms that match mine.  That bug seems to have been unchanged since 2010.

Comment 125 Tobi Vollebregt 2012-07-18 11:17:42 UTC
> I'll switch to STD again for a while and keep you posted.

I've tested hibernation now some 5 times; it worked flawlessly each time. So this bug can be considered fixed. To everyone involved: thanks for the effort!