Created attachment 880051 [details] /var/log/messages for rare successful boot of 3.13.7-100.fc19.x86_64 Description of problem: On my f19 and f20 installs, all tested 3.13 kernels frequently hang and force automatic reboot with no message of any form left anywhere whether on console or in dmesg, /var/log/messages, etc. Version-Release number of selected component (if applicable): 3.13 in general, examples here with 3.13.7-100.fc19.x86_64 How reproducible: Just boot machine and choose a 3.13 kernel from grub2 Steps to Reproduce: 1. As above. 2. 3. Actual results: Machine may boot regularly, but more likely to just present the one line "Booting ..." and then, either immediately or after up to about 5 seconds, the screen goes blank, and the machine then reboots automatically through POST and back to grub2. Expected results: Regular boot (at least to a virtual terminal... I note many people describe things as saying "didn't boot", when it did, just not into a graphics mode. Not getting anywhere near that far with this). Additional info: Machine is Intel Core i3 540, Asus p7H55-M Pro motherboard, NVidia 9800GT graphics (using stock Nouveau driver), it has run earlier Fedoras and their stock kernels for 3 years. Never seen this major of a fail before. (Or on earlier machines since RedHat 4.0). Tested many forms of 3.13, all seem to fail. Compiled my own from kernel.org and it also failed. Tried to compile redhat debug kernel, and it also failed. ALso tested rawhide 3.14 and they fail. But all 3.11 and 3.12 all boot reliably on F19 and F20 (which I installed specially just to test this). It may take ten tries to boot these kernels (some I think did not boot at all). Sometimes they do boot, as often as three times in a row. The debug kernel 3.13 kernels suppied BY redhat, however, DO ALL BOOT reliably. (Someone perhaps can say how to build these. I followed the instructions in editing the spec file to produce debug kernel, and it names it that way, but it is not the same as the distributed one since its size is much smaller, more akin to the regular kernels). I attach /var/log/messages for two boots. One rare successful boot of 3.13.7-100.fc19.x86_64, and one of the reliable booting 3.13.7-100.fc19.x86_64.debug. Also list of modules, pci and usb bus for the first (I think more or less the same for both). Can someone perhaps mention how it is that the debug kernel would boot regularly when the other does not. Is this a true "Heisenbug", or is it that as well as more messages, the debug kernel does various checks, for example, for buffer overruns or something, and tries to correct them? I don't see anything in the logs that gives a clue on this. Also, can someone perhaps mention if this is at all common, that a kernel will sometimes boot, sometimes (far more often) not boot? I have not experiened this. I might think it was attributable to hardware, except for the fact that ever other kernel version boots okay, and the debug kernels also boot. Also mention what else I can do. I would prefer to use f19 if possible, I have not yet installed f20 fully yet (maybe will not, if cannot get a new kernel on it. I had to fetch the 3.12 kernel not from yum but an archive, it has only 3.11 and 3.13).
Created attachment 880052 [details] /var/log/messages for boot of 3.13.7-100.fc19.x86_64.debug
Created attachment 880053 [details] lsmod for above 3.13.7 boot
Created attachment 880054 [details] lspci brief and details for above 3.13.7 boot
Created attachment 880055 [details] lsusb brief and details for above 3.13.7 boot Oops, I see each of these generates an email. Perhaps it is more correct to tar up all these files? I just though that it is easier to read online this way.
Can you try removing 'rhgb' and 'quiet' from your kernel command line and seeing if you get console output when you boot? If that doesn't work alone, you might try adding "earlyprintk=vga". If the issue is a race in the kernel somewhere, the debug options enabled in the debug kernels could change the timing enough to not hit the race. If it's something else, like an oddity in grub loading the kernel into memory, the debug kernel is also a different size than the normal kernel.
As I did not make clear in my original post, but which is evident in the /var/log/messages files I posted, rhgb and quiet are not on the kernel command line. However I did not know about earlyprintk=vga until I found https://fedoraproject.org/wiki/Common_kernel_problems#crash_hang (through extensive googling and not actually just browsing the fedora site). When I tried this, of about 30 boot attempts, I got the following. First, in all cases I did get some messages, then a pause, either very short (too short to read any messages) or longer (up to 5 seconds? Anyway, same as before), and in this case the last message consistently said "No AGP aperture found". I don't know why it would need anything related to AGP and why this would cause a problem, so I tried to disable that by booting with iommu=noaperture (found from google). Of the many boots I tried in that form, the first one actually booted (the only of about 30 attempts total) so I thought that might be something, but subsequent boots did not work. And the only thing different was that it did not check for AGP aperture. So it seems that what is causing the hang takes place just after that. One important thing: of the 30 boots, one of them produced a stack trace printed on the screen. I was ready to take a photo but it disappeared too fast. Of the photos I did take, I can see that the messages printed up to that point, those that remain on the screen, are exactly the same as the messages recorded in /var/log/messages for a good boot except that strangely, the good boot does not report anything in /var/log/messages about checking for AGP aperture. However, everything else is the same (obviating the need to post a blurry had to read photo). The last line seen when I boot with iommu=noaperture is then: PID hash table entries: 4096 (order: 3, 32768 bytes) Then it pauses and crashes. The next line in the successful boot for both regular and debug kernel is: Memory: 16356488K/16709688K available (6736K kernel code, 1056K rwdata, 2976K rodata, 1468K init, 1604K bss, 353200K reserved) So it is something happening between (what should be) the printing of these two messages, I presume. That narrows done the location a little more. Regarding the debug kernel, you are saying that it could not prevent the crash by finding some bug via range checks and the like, but the only way it could conceivably do that is by changing the timing (inadvertently? I guess that must be right, or otherwise it would have reported detecting some range violations. I noticed also that the debug kernel was a substantially different size, when I pointed out that this led me to believe that I had not successfully build my own version of the debug kernel (since what the build called debug has a size close to the regular kernel size).
Have you run memtest86+ overnight on this machine? If you haven't, it might be a good idea to do so.
Yes, I have run the memtest Fedora provides in the default grub2.cfg for a full day, a few months ago, and it did not find anything. Should I do it again? Do you think it could be a hardware problem? It seems the the crash is happening in a pretty regular location with a pretty specific piece of code (3.13 kernels, nothing else). Is there no other plausible idea as to what could be causing it? It seems that memory problems should cause more erratic kinds of crash than this. Do you have any other ideas at current?
No ideas at present. There's not a lot of concrete information to go on as of yet. You might try passing initcall_debug in addition to earlyprintk=vga to see if you can get it narrowed down to a specific initcall. If you do get an oops, you can also pass pause_on_oops=60 to make it wait for 60 seconds so you are able to take a photo.
I added initcall debug but nothing was any different, no new messages on the console. After trying seven more times, I got another successful boot, and there are no initcall's in /var/log/messages. Apparently they only go into dmesg, and there are a bunch there, but I guess that's not helpful since they are for a successful boot? I though you might see a sign of a problem in the successful boot messages, but I guess not. You did not answer if you think it is likely to be a memory problem. Wouldn't such a problem cause other serious symptoms, and also less predictably? Also, do you think I should run memtest again? I ran it a few months ago for a long time with no incidents.
The first initcall in dmesg occurs at t=0.15 or so, which is long after the spot where the kernel crashes (when it does). As I mentioned before, it crashes here between "No AGP bridge" and "Memory" (this from the dmesg file of the current, rare, successful boot): [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] Checking aperture... [ 0.000000] No AGP bridge found [ 0.000000] Memory: 16355776K/16709688K available (6679K kernel code, 1020K rwdata, 2976K rodata, 1440K init, 1552K bss, 353912K reserved) Here is the first initcall: [ 0.157947] initcall set_real_mode_permissions+0x0/0x9a returned 0 after 0 usecs
I looked in the code to find the place where the last message printed, containing the phrase "hash table entries", was printed, and without understanding any of the code neverthless traced it laboriously upwards until reaching init. It was called in the call to the function pidhash_init so this function returns. Then I did the same for the first message not printed, but printed in a regular successful boot, containing the phrase "Memory ... available" It was called in the call to the function mm_init. So the crash happens in that function or before it was called. This is the section of code. pidhash_init(); vfs_caches_init_early(); sort_main_extable(); trap_init(); mm_init(); I hope that is helpful, it took quite a while to do.
Whoops "It was called in the call to the function pidhash_init so this function returns". Well it is called anyway. The crash seems to happen after pidhash_init is called and before mm_init returns, assuming the prints happen synchronously, which I suppose they might not, so it's not clear what this all means. It the mm_init print is buffered up and for some reason it returns but the print is not done, I guess it doesn't really say that much after all. I don't know what is legitimate to assume about the timing of these print statements. Still, it seems like this is a very regular occurence.
I have no idea what the problem is at the moment. If it's a memory problem, it could be within a certain range that only gets touched during boot by 3.13.y due to size issues or different memory allocation ordering. You can try memtest86+ again if you'd like; it can't hurt.
I will run memtest again when I get the chance, but this is my only machine and it is used throughout the entire day so it hard to find the time. What else can I do at this point? Does it make sense to put more printk's in the kernel to try to track the point where it crashes down to a smaller and smaller region, without having any understanding of what the code is doing? Will this information help you? From what I have so far, would you agree it seems to crash at or very near the same point each time, and that this point is well before any initcalls? Or is it possible that it prints out a bunch of messages, then stops printing out messages for a comparatively long time even though it has accumulated a large number of them? I am assuming that the fact it hasn't printed out the "Memory... allocated" message means it has not progressed far behind that part in the code where this printk is issued.
I did a few passes with memtest86+, it found no errors. Then trying to reboot into a 3.13, took ten tries with a crash in the same place every time. It seems to me that if this was a memory problem, the kernel does a so much better job of detecting it, since it detects it nearly each time, whereas despite testing each byte with eight different tests, memtest86+ failed to find it. (Although memtest86+ it seems, does not test all memory, there is a small part at the beginning it seems to skip, although the kernel would also skip it, I believe). So I don't believe it is a memory problem, although it could be some other hardware problem. But if it is a hardware problem, I think it would have to be a problem where the hardware just does not implement some spec correctly, and not failing hardware, since it is so consistent (I had a "hardware problem" like this once, in 2002, and ended up talking to Alan Cox himself who was in the process of making the kernel work around it). I also compiled a stock 3.13.7 kernel from kernel.org, and put in it some printk's after each function call listed in my message above. It got through all of them okay, and then booted fine, ten times in a row. So it does not appear to be a problem with a combination of my hardware and the basic kernel. Something specific to the redhat kernel running on my hardware (whether defective or not).
Still cannot boot these kernels, and now yum update wants to delete my only working kernel. Since I did not get any advice on what to do next, which I hopefully asked about on Apr 1, I downloaded one of the non-working 3.13 Fedora kernel sources and put lots of printks in it. After laboriously tracing out the code again, not knowing what any of it does, I found that this modified kernel (modified only with printks, which of course could produce something different as the code is located in slightly different spots now) crashes on boot uniformly (assuming it does crash at all: only crashes nine times out of ten or so) in free_all_bootmem, called by mem_init, called by mm_init, called by start_kernel. This is a manually detected stack trace, in effect, found by tracing backwards through the code and putting in lots of printks and getting someone with better reaction time than me to watch them on the screen. I don't know if there is any point going further to try to narrow things down, because the place where it crashes may not be where the error is actually caused. However, I would hope this information would be helpful to those who actually work on and understand the kernel, and maybe they can say, oh yeah, it could be this or that given the specific hardware configuration. I can keep going to try to narrow it down, but the code starts to get complicated. Based on the bdebug statements in there, there have been problems found free_all_bootmen_core in the past. I also found some tracebacks via google which indicate that spin_lock is called in this part of the code also. I don't understand why there have to be these locks at an early point when interrupts are disabled and presumably there is only one process running, but if there are, I guess there could be a timing problem? It may not be of any use to track down the location where the problem is realized, if it was actually caused earlier? So what can I do now? The normal yum update I do now wants to totally remove the only kernel for f19 that actually works. How long can I continue to use Fedora if it tries to delete the only working kernel? I keep the rpms around and forcefully install them, but presumably this will only work so long (I assume the kernel is fairly free standing but it does build up a big initramfs of new software. Another thing I don't understand, why such a large initramfs is needed, just more things that make linux very hard to understand and thus use the instant something goes wrong).
What am I supposed to do now? I am trying to learn by myself for days without any guidance how to make it so I can actually continue to run Linux on my computer. Searching around I see Josh Boyer recommends in 2012 and then 2013 that we use "koji bisect" to find the last good kernel and the first bad kernel. His example has kernels with the same major version number so I don't know if this works in this case, where the good kernels are 3.12 (all of them) and the bad ones 3.13 (all of them). In any case I don't need the koji bisect since the choices available are so limited that a search among them is quite easy. I already reported all Fedora 3.13 available work and all Fedora 3.13 do not. So a fortiori, the last good kernel in koji is kernel-3.12.11-200.fc19 and the first bad one is kernel-3.13.4-100.fc19 (there is a a 3.13.3 but it has been deleted). So as Josh Boyor says "You then dutifully tell the Fedora kernel team that kernel-[xxx] caused your regression and we go off scratching our heads.". I don't know if that is what is happened since I have heard nothing, but I assume not. I gather from searching around that I should try to do a git bisect at this point, but I cannot figure out how to do that. I tried what is given at https://fedoraproject.org/wiki/User:Ignatenkobrain/Kernel/Bisection for quite a while but cannot understand it. It seems to require that I give only v3.12 and v3.13 as good and bad, and reports 6500 search points. If I try to be more precise and say "git bisect bad kernel-3.13.4-100.fc19" or even "git bisect bad v3.13.4" it says cryptically "fatal" Needed a single revision". I don't understand why it cannot be narrowed down further than that. And I don't understand if this includes the fedora patches as well, it is not clear from the minimal and cryptic output. Since the stock kernels work fine it would seem foolish to search in 6500 search points all of which actually work. One has to be present and keep checking when it might be ready to proceed, after waiting while it stops at "Start: Outputting list of available packages" for a very long time (it has not yet returned from this even once). I understand how this could work fairly simple with kernel.org kernels (although I cannot see where they have a patch from a 3.12 to a 3.12. All the patches for 3.13.x seem to be back to 3.13 original release or 3.13.(x-1). But I guess there is something that allows one to search back from a 3.13 to a 3.12 even though it is not evident in the last of downloadable patches). But I don't understand how to do it for Fedora kernels with their own list of patches. I see in the source for e.g. 3.13.6 there is the kernel source for 3.13 and then patches to go up to 3.13.6 and then a list of Fedora patches on top of that. But I don't see how any semi-automated bisect would work to go back to 3.12 kernels. Moreover, it seems it would be best if I could get the earliest 3.13 kernel which I would bet fails too, but there isn't one in koji that I could find to test. On top of that, Josh's instructions on kobi bisect and other instructions on git bisect tell us to get the latest working kernel, but when going from 3.12 to 3.13 I understand that the latest 3.12 kernel is not the ancesor via patches of a 3.13 kernel, since 3.12 was further developed after 3.13 was released. I don't know what this means for what kernels I should use in a semi-automated git bisect assuming I can even get that started. This is very frustrating. I have tried very hard for three weeks to look around without guidance and figure out how to simply get a working kernel for Fedora, aside from explictly downloading an old one and forcing it to be installed and preventing yum update from deleting it, and wondering when this will no longer work anymore, which is now what I have had to do.
Regarding memtest86+, I now have run 8 straight tests of memtest86+, taking 21 hours to do so. No problems were detected. This is the third time I have done this degree of testing on this identical hardware. Actually, not quite identical. I also updated my motherboard BIOS a couple of days ago, so this system was slightly different. No problems detected in any case. However, all the 3.13 kernels still crash at boot most of the time. From the printk's I insered, they seem all to be doing so at basically the same place, as observed above, in free_all_bootmem. Still don't know what to do next. (I configured yum to not install any more kernels, but I don't know how long this will last. I won't be able to do updates to a newer Fedora when F19 reachs EOL once they are all using 3.13 kernels).
I have the same problem but mine sometimes gets to the graphical login screen. Just tried a dracut --force --kver 3.13.10-200.fc20.x86_64 and that made no difference. Stuck in reboot loop. Also tried init 1 on the kernel line but it won't even boot to single user mode. Nothing in logs. The only kernel that works is 3.11.10-301.fc20.x86_64 that last 3 don't?
I assume you mean when it gets to the login screen, all is then fine and you and log in and use the system indefinitely? That was my experience with the kernels (although they don't get to a login screen since my dm is configured to skip that). If your problem is the same as mine, I don't think altering the initramfs with dracut or using init 1 will have any effect. The crash occurs very very early before any of this stuff comes into play. I wonder if it is "the same thing", thought since I still don't know what it is. How far do you get in the boot? If you boot with rhgb and quiet removed, do you get messages anywhere when it crashes? I did not, so if you do, you might have something to go on. What happens when you earlyprintk=vga? In that case, as described above, I get a few messages, and it crashes in apparently the same place each time. Can you check that? Since no one else has apparently reported what appears to be the same bug that I have reported, Red Hat apparently has no interest in trying to find it, so I also have given up and am using old kernels. They suggest it is hardware, which I doubt since the evidence suggests it crashes in the same place each time (although that place perhaps could just be catching a hardware error) and 20 continuous hours of memtest86+ show no problems. If you can see if you get the crash in the same place that would be useful. As you see above I put printk's in the kernel source of a crashing kernel and found it is crashing in free_all_bootmem (A place that has been riddled with crash prone behavior in the past). I hope it is the same place and then there will be better evidence for Red Hat that it is a problem. 3.12 kernels will probably also work for you though. f20 doesn't have any in the standard repos I believe, but you can download the rpm for one from the koji archives and try that one.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs. Fedora 19 has now been rebased to 3.14.4-100.fc19. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 20, and are still experiencing this issue, please change the version to Fedora 20. If you experience different issues, please open a new bug report for those.
"Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel." I am not going to test this kernel right now. This was a very traumatic experience, in all, I tried booting 100's of times to do testing with various kernels, and having the machine constantly crash on 3.13 and 3.14 (rawhide at the time) kernels was very disconcerting. Compiling many versions, putting debug statements in, etc., and no one seemed interested in this, INCLUDING the person above who suggested he had the same problem but in many weeks could not provide even minimal extra information that would confirm that. So I went back to the 3.12.9 kernel and am just going to use that as long as I possibly can, and then when I cannot any longer get along with it (manually installing it even after any other software upgrades), I will try whatever is suggested to be the current stable kernel at that time. This has been a horrible experience, and having my machine up now for a month I do not want to go back to seeing this constant crashing when it serves ABSOLUTELY no purpose by to cause anxiety.
I have the exact same problem. The kernel-debug 3.14 kernel works but any regular 3.13 or 3.14 kernel boots, and deep into the booting process suddenly hangs and dies without displaying an oops. nothing is logged to disk either. this is an ASUS P8P67pro main board with UEFI AMI version 2.00.1201 (C) 2010 cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 42 model name : Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz stepping : 7 microcode : 0x29 cpu MHz : 1627.750 cache size : 8192 KB physical id : 0 siblings : 8 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid bogomips : 6822.16 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: ( 8 cpus after hyperthreading) I can reliably reproduce this. It seems silly to request a new bug number for this.
Paul, it is not necessarily "exactly" the same problem. It could be something very different, anything in the kernel at boot. In fact I gather the Fedora people here seem to think mine is faulty memory, which could be true even though 20 hours of memtest86+ find no error and it happens at the same place each time. Actually, it sounds like yours is not the same, if it goes "deep into boot". I notice you did not say that it put nothing onto the console, like mine without the earlyprintk=vga. Could you try yours with the earlyprintk=vga on the command line (and quite rhgb removed of course) to see what you get on the console. Compare to message 11 above and see if it seems to be crashing at the same, or nearly the same, place? If it is not at the same place, it probably is a different bug and needs a different bug report.
This message is a notice that Fedora 19 is now at end of life. Fedora has stopped maintaining and issuing updates for Fedora 19. It is Fedora's policy to close all bug reports from releases that are no longer maintained. Approximately 4 (four) weeks from now this bug will be closed as EOL if it remains open with a Fedora 'version' of '19'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 19 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.