Description of problem: System will do emergency restart when the brightness settings have not changed and does not boot unless I put systemd.restore_state=0 as a kernel parameter, or if I unplug my laptop briefly after kernel has started booting and plug it back in after the backlight settings have changed. Also acpi=rsdt, acpi=off and mem=4G make the system bootable, but are not preferred as they introduce instabilities in other areas. Version-Release number of selected component (if applicable): From late Fedora 18 until now (Fedora 20). How reproducible: Always Steps to Reproduce: 1. Boot without Grub2 Theme. 2. Boot without systemd.restore_state=0 3. Boot without brightness settings changing. Actual results: Emergency restart. Expected results: Clean boot. Additional info: Booting with Gentoo's latest install CD did not produce any problems which has lead me to suspect that this is a systemd-backlight bug and not a kernel bug.
That sounds like a bug in the backlight driver in the kernel. Reassigning.
Hi Steven, Can you please try to collect some kernel logs from when this happens ? IE plug in a usb->serial convertor, and do serial console logging to another machine ? Thanks, Hans
Hi Hans, You may need to help walk me through that. I haven't done much with tracking issues down in the kernel until this problem. I had tried using USB to serial to capture kernel output, but for some reason I wouldn't get anything on the machine I had hooked up. The equipment I have works because I was able to send plain text messages over the cable once the machine had booted.
(In reply to Steven Michael Williams > The equipment I have works because I was able to send plain text messages over the cable once the machine had booted. Good, that means you're nearly there. You will need to add "console=ttyUSB0,115200" to the kernel cmdline to get the kernel to send it boots message to the (usb) serial port, with that you should see kernel messages on the other machine, and you will hopefully get some kernel error messages at the time of the freeze, giving us a hint on what is going on. Thanks, Hans
I have been able to start getting serial output from the kernel, but I have been having trouble reproducing the crash. The brightness needs to be fully up for it to crash, and in the serial console mode I have to try to set it manually quickly. Is there anyway I can set full brightness automatically? Also it seems when it does crash, that the console messages do not come through, so I started getting earlyprintk to work, but I haven't had time to do more with it yet.
Created attachment 903323 [details] Capture of Boot Output I tried today to capture crashes with earlyprintk=serial,ttyUSB0,115200 console=ttyUSB0,115200 debug as kernel parameters. It seems like the crashes never made it to serial output, but there may be some information of use here anyway.
Steven, Thanks for the capture, unfortunately you're right and it does not seem to contain any useful info about the crash. Still there are some things we can do, for starters can you do: ls /sys/class/backlight And: ls /var/lib/systemd/backlight And also cat each file in /var/lib/systemd/backlight, and let us know the output of all of these commands ? Thanks & Regards, Hans
Created attachment 904668 [details] ls /sys/class/backlight output This is the output from "ls /sys/class/backlight".
Created attachment 904669 [details] ls /var/lib/systemd/backlight output
I ran cat on each of the acpi_video* files "ls /var/lib/systemd/backlight" showed with all of them having an output of 10.
Interesting, what model system / laptop is this ? For some reason the acpi-video driver finds 5 different video busses, which is quite funky. Can you try booting with video.use_native_backlight=1 on the kernel cmdline and then do "ls /sys/class/backlight" again ? If that helps, and your system is a laptop, can you also check to see if you can still control your backlight with the hotkeys on the keyboard ? If that does not help, can you try with acpi_backlight=vendor on the kernel cmdline ?
I have a Lenovo G530 (it is from 2008). Using video.use_native_backlight=1 didn't change anything as far as the 5 video buses being under /sys/class/backlight, but acpi_backlight=vendor didn't leave so much as one there. Also acpi_backlight=vendor enabled me to boot with full brightness whereas before that simply wouldn't happen. I can do more extensive testing (20 boots) with acpi_backlight=vendor to see if it will crash at all later today.
(In reply to Steven Michael Williams from comment #13) > I have a Lenovo G530 (it is from 2008). OK. > Using video.use_native_backlight=1 didn't change anything as far as the 5 > video buses being under /sys/class/backlight, but acpi_backlight=vendor > didn't leave so much as one there. Also acpi_backlight=vendor enabled me to > boot with full brightness whereas before that simply wouldn't happen. I can > do more extensive testing (20 boots) with acpi_backlight=vendor to see if it > will crash at all later today. Can you also control the backlight, e.g. by using the slider in the upper right corner menu of gnome3 when using acpi_backlight=vendor, and do the brigthness up / down hotkeys on the keyboard work ? Regards, Hans
(In reply to Hans de Goede from comment #14) > Can you also control the backlight, e.g. by using the slider in the upper > right corner menu of gnome3 when using acpi_backlight=vendor, and do the > brigthness up / down hotkeys on the keyboard work ? No I am not able to use the hotkeys at all. Since I am using KDE I also do not have a slider for brightness control, and I could not find an equivalent widget or similar control so far.
Hi, (In reply to Steven Michael Williams from comment #15) > (In reply to Hans de Goede from comment #14) > > > Can you also control the backlight, e.g. by using the slider in the upper > > right corner menu of gnome3 when using acpi_backlight=vendor, and do the > > brigthness up / down hotkeys on the keyboard work ? > > No I am not able to use the hotkeys at all. Since I am using KDE I also do > not have a slider for brightness control, and I could not find an equivalent > widget or similar control so far. Hmm, ok. What is the output of: "ls /sys/class/backlight" when you boot with acpi_backlight=vendor? And when you boot without acpi_backlight=vendor can you then control the backlight with the hotkeys? Regards, Hans
(In reply to Hans de Goede from comment #16) > What is the output of: "ls /sys/class/backlight" when you boot with > acpi_backlight=vendor? Nothing. > And when you boot without acpi_backlight=vendor can you then control the > backlight with the hotkeys? Yes I am able to.
(In reply to Steven Michael Williams from comment #17) > > And when you boot without acpi_backlight=vendor can you then control the > > backlight with the hotkeys? > > Yes I am able to. Hmm, so it seems that acpi_backlight=vendor is not the ideal solution, since it breaks backlight control. This machine only has built-in intel graphics, and no discrete gpu, right ? Can you boot without acpi_backlight=vendor and then for each dir under /sys/class/backlight cd into that dir and do (as root): cat max_brightness echo "half of max_brightness" > brightness echo "max_brightness" > brightness Where the strings between "" are to be replaced with the values of the string (ie 127 and 255) And see what happens with the backlight ? Please write down max_brightness for each of the acpi_video# dirs, and also write down per dir if echoing values to brightness controls the backlight or not.
Created attachment 908112 [details] Results of my testing in the acpi_video* directories. I ran the commands you told me to and the results are attached. To summarize all of the directories have a max brightness of 10 and all of them responded to echo half and max brightness > backlight.
Thanks, so non of them caused the emergency restart you're seeing when systemd is using them, weird. Can you try as root to execute this line: echo 10 > /sys/class/backlight/acpi_video0/brightness && echo 10 > /sys/class/backlight/acpi_video1/brightness && echo 10 > /sys/class/backlight/acpi_video2/brightness && echo 10 > /sys/class/backlight/acpi_video3/brightness && echo 10 > /sys/class/backlight/acpi_video4/brightness And see if that triggers the reboot ? Also you said that with acpi_backlight=vendor you're getting a higher / more bright backlight then without it. Does this mean that the brightness with echo 10 > brightness is less bright then when booting with acpi_backlight=vendor ?
(In reply to Hans de Goede from comment #20) > Thanks, so non of them caused the emergency restart you're seeing when > systemd is using them, weird. > > Can you try as root to execute this line: > > echo 10 > /sys/class/backlight/acpi_video0/brightness && echo 10 > > /sys/class/backlight/acpi_video1/brightness && echo 10 > > /sys/class/backlight/acpi_video2/brightness && echo 10 > > /sys/class/backlight/acpi_video3/brightness && echo 10 > > /sys/class/backlight/acpi_video4/brightness > > And see if that triggers the reboot ? I ran those commands, and they do not trigger a reboot. > Also you said that with acpi_backlight=vendor you're getting a higher / more > bright backlight then without it. > Does this mean that the brightness with echo 10 > brightness is less bright > then when booting with acpi_backlight=vendor ? Sorry if that is what came across. With acpi_backlight=vendor I could boot with full brightness without a crash whereas otherwise I have to dim it as soon as the kernel starts booting or it will reboot when the Fedora icon is about 30 to 40 percent full.
(In reply to Steven Michael Williams from comment #21) > (In reply to Hans de Goede from comment #20) > > Thanks, so non of them caused the emergency restart you're seeing when > > systemd is using them, weird. > > > > Can you try as root to execute this line: > > > > echo 10 > /sys/class/backlight/acpi_video0/brightness && echo 10 > > > /sys/class/backlight/acpi_video1/brightness && echo 10 > > > /sys/class/backlight/acpi_video2/brightness && echo 10 > > > /sys/class/backlight/acpi_video3/brightness && echo 10 > > > /sys/class/backlight/acpi_video4/brightness > > > > And see if that triggers the reboot ? > > I ran those commands, and they do not trigger a reboot. The intend was for you to run that command as a single long line, so as to write to the different devices in quick succession. Hmm, while at can you also try echoing 10 to acpi_video0/brightness twice ? Maybe writting the same value as is already there makes it reboot ? > > Also you said that with acpi_backlight=vendor you're getting a higher / more > > bright backlight then without it. > > Does this mean that the brightness with echo 10 > brightness is less bright > > then when booting with acpi_backlight=vendor ? > > Sorry if that is what came across. With acpi_backlight=vendor I could boot > with full brightness without a crash whereas otherwise I have to dim it as > soon as the kernel starts booting or it will reboot when the Fedora icon is > about 30 to 40 percent full. I see thanks for clarifying. So we do want to keep at least 1 acpi_video# backlight interface around so that you can control your backlight. For starters lets see if we can bring the amount of registered interfaces down from 5 to 1, hopefully that will also fix the reboot issue. I've started a scratch build of a kernel with some extra debugging: http://koji.fedoraproject.org/koji/taskinfo?taskID=7042440 Can you please download and install this ("sudo rpm -ivh kernel...rpm") , then boot into it without any special kernel cmdline parameters, and after booting do: dmesg | grep acpi_video And copy and paste the output here? Thanks, Hans
(In reply to Hans de Goede from comment #22) > > > Can you try as root to execute this line: > > > > > > echo 10 > /sys/class/backlight/acpi_video0/brightness && echo 10 > > > > /sys/class/backlight/acpi_video1/brightness && echo 10 > > > > /sys/class/backlight/acpi_video2/brightness && echo 10 > > > > /sys/class/backlight/acpi_video3/brightness && echo 10 > > > > /sys/class/backlight/acpi_video4/brightness > > > > > > And see if that triggers the reboot ? > > > > I ran those commands, and they do not trigger a reboot. > > The intend was for you to run that command as a single long line, so as to > write to the different devices in quick succession. Sorry again for miscommunication. I did run it as a single line using &&. > Hmm, while at can you also try echoing 10 to acpi_video0/brightness twice ? > Maybe writting the same value as is already there makes it reboot ? Did it 5 times over and no crash. Downloading and installing the kernel now as well.
(In reply to Hans de Goede from comment #22) > For starters lets see if we can bring the amount of registered interfaces > down from 5 to 1, hopefully that will also fix the reboot issue. I've > started a scratch build of a kernel with some extra debugging: > > http://koji.fedoraproject.org/koji/taskinfo?taskID=7042440 > > Can you please download and install this ("sudo rpm -ivh kernel...rpm") , > then boot into it without any special kernel cmdline parameters, and after > booting do: > > dmesg | grep acpi_video > > And copy and paste the output here? This is what I got: [ 3.395050] acpi_video: registering backlight device ffff880036a95360 on bus ffff8800b6ed59c0, crt 1, lcd 0 tvout 0 dvi 0 unknown 0 [ 3.411743] acpi_video: registering backlight device ffff880036a95300 on bus ffff8800b6ed59c0, crt 0, lcd 1 tvout 0 dvi 0 unknown 0 [ 3.428565] acpi_video: registering backlight device ffff880036a952a0 on bus ffff8800b6ed59c0, crt 0, lcd 0 tvout 0 dvi 0 unknown 1 [ 3.444863] acpi_video: registering backlight device ffff880036a95240 on bus ffff8800b6ed59c0, crt 0, lcd 0 tvout 0 dvi 0 unknown 1 [ 3.459941] acpi_video: registering backlight device ffff880036a951e0 on bus ffff8800b6ed59c0, crt 0, lcd 0 tvout 0 dvi 0 unknown 1
Thanks for the input, here is a kernel build which should reduce the number of backlight interfaces on your system to 1: http://koji.fedoraproject.org/koji/taskinfo?taskID=7043168 Note this is still building atm, once it is done building please install it, and do ls /sys/class/backlight to confirm that there only is one backlight interface after this. If this indeed reduces the backlight interface count to 1, please also check if it fixes your reboot problem. p.s. I'm not sure the fix I've come up with will be accepted by the upstream Linux kernel community, but first lets check if it actually works.
$ uname -a Linux localhost.localdomain 3.14.7-200.rhbz1094512.2.fc20.x86_64 #1 SMP Fri Jun 13 14:20:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux $ ls /sys/class/backlight/ acpi_video0 It looks like the fix for the number of backlights worked, but I crashed two times upon attempting to boot.
(In reply to Steven Michael Williams from comment #26) > It looks like the fix for the number of backlights worked, but I crashed two > times upon attempting to boot. Bummer, I was hoping getting rid of the bogus backlight interfaces would fix the crash on boot :| Without that the fix is actually pretty useless, since most userspace apps will just pick the first interface, and since they all work that is fine. It is a bit weird, but it cannot hurt. I'll attach the patch I did for this here, in case we decide we do want to use it afterall later. So I wonder what systemd-backlight is doing that causes the crash. You should have one or more files under /var/lib/systemd/backlight, can you cat a file there, see what value is in there and try echoing that ? Can you also try echoing 0 to the brightness file ? Last can you try booting into text mode, add "3" at the end of the kernel cmdline, then you should boot straight into a text mode login, then try echoing some values to the brightness there, and see if that maybe triggers the crash ? Thanks, Hans
Created attachment 908679 [details] PATCH to only show one backlight interface For future reference.
(In reply to Hans de Goede from comment #27) Sorry for taking so long to get back with you on this. > Bummer, I was hoping getting rid of the bogus backlight interfaces would fix > the crash on boot :| Without that the fix is actually pretty useless, since > most userspace apps will just pick the first interface, and since they all > work that is fine. It is a bit weird, but it cannot hurt. I'll attach the > patch I did for this here, in case we decide we do want to use it afterall > later. > > So I wonder what systemd-backlight is doing that causes the crash. > > You should have one or more files under /var/lib/systemd/backlight, can you > cat a file there, see what value is in there and try echoing that ? This is what is currently sitting in the directory with the kernel that you modified. $ ls /var/lib/systemd/backlight/ acpi_video0 acpi_video2 acpi_video4 pci-0000:00:02.0:backlight:acpi_video1 pci-0000:00:02.0:backlight:acpi_video3 acpi_video1 acpi_video3 pci-0000:00:02.0:backlight:acpi_video0 pci-0000:00:02.0:backlight:acpi_video2 pci-0000:00:02.0:backlight:acpi_video4 The values for all those files are 10. > Can you also try echoing 0 to the brightness file ? I did, and it works. No crash. > Last can you try booting into text mode, add "3" at the end of the kernel > cmdline, then you should boot > straight into a text mode login, then try echoing some values to the > brightness there, and see if that maybe triggers the crash ? Did that and the brightness changed, but the kernel did not crash.
Hi, Hmm, so it seems that we cannot figure out which sequence of events exactly causes the reboot, making this quite hard to fix. I think it is best if you just use systemd.restore_state=0 as a workaround for now. The kernels acpi support does get various improvements and bugfixes regularly. So it would be good if you can try without the argument whenever we bump the kernel, e.g. from 3.14.x to 3.15.x. If such a kernel upgrade fixes things please let us know. Regards, Hans
3.14.9 seems to work fine, but 3.15.3 still has the same problem.