Bug 1509515 - tuned cmdline variable in grub.cfg corrupted after installing kernel-core
Summary: tuned cmdline variable in grub.cfg corrupted after installing kernel-core
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: grubby
Version: 29
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1618684 (view as bug list)
Depends On:
Blocks: 1561919
TreeView+ depends on / blocked
 
Reported: 2017-11-04 10:50 UTC by Kees de Jong
Modified: 2019-11-27 22:26 UTC (History)
29 users (show)

Fixed In Version:
Clone Of:
: 1561919 (view as bug list)
Environment:
Last Closed: 2019-11-27 22:26:41 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Kees de Jong 2017-11-04 10:50:54 UTC
Description of problem:
After upgrading/installing kernel-core my system becomes unbootable due to configuration errors in my grub.cfg. Running a `grub2-mkconfig` or re-enable the tuned configuration after the upgrade corrects it, so my guess is that something in the kernel-core RPM triggers this problem in combination with tuned.


Version-Release number of selected component (if applicable):
# rpm -qa kernel tuned                                                                                                                                                
kernel-4.13.9-200.fc26.x86_64                
kernel-4.13.8-200.fc26.x86_64                
tuned-2.8.0-3.fc26.noarch                    
kernel-4.13.10-200.fc26.x86_64

How reproducible:
Create a tuned profile in /etc/tuned by creating a directory there e.g. custom-powersave and there place the config file named tuned.conf with the following content (full path /etc/tuned/custom-powersave/tuned.conf) shown below. More information about this can be found here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_enterprise_linux-performance_tuning_guide-performance_monitoring_tools-tuned_and_tuned_adm



[main]
summary=Customized low-power usage profile
include=powersave

[bootloader]
cmdline=pcie_aspm=force



After this start and enable tuned with `systemctl enable --now tuned`. Then activate the profile with `tuned-adm profile custom-powersave`. Then inspect /boot/efi/EFI/fedora/grub.cfg and search for this line: set tuned_params="pcie_aspm=force"

Now reinstall the following package with `dnf reinstall kernel-core`. Then inspect /boot/efi/EFI/fedora/grub.cfg again and then you'll see the following: set tuned_params="pcie_aspm=force"=force"=force"=force"=force"=force"=force"=force"=force"=force"=force"=force"=force"=force"=force"=force"=force"=force"=force"=force"

Which after a reboot renders your system unbootable. Running either `grub2-mkconfig` or `tuned-adm profile custom-powersave` fixes the grub.cfg again.


Steps to Reproduce:
1. Create a custom tuned profile with cmdline options.
2. Enable the tuned service.
3. Enable the tuned profile.
4. Inspect grub.cfg, see that "tuned_params=" is set without errors.
5. Reinstall kernel-core.
6. Inspect grub.cfg, see that "tuned_params=" is set with errors.
6. Re-enable the custom tuned profile again to restore grub.cfg to a workable syntax.

Actual results:
System doesn't boot anymore.

Expected results:
"tuned_params=" to be updated like `grub2-mkconfig` or `tuned-adm profile custom-powersave` does in grub.cfg.

Additional info:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_enterprise_linux-performance_tuning_guide-performance_monitoring_tools-tuned_and_tuned_adm

Comment 1 Kees de Jong 2017-11-04 10:56:20 UTC
On #fedora-devel it was suggested to switch this rapport to the grubby package.

# rpm -qa grubby     
grubby-8.40-4.fc26.x86_64

Comment 2 Kees de Jong 2018-01-24 07:37:44 UTC
I still have to be very careful with kernel updates, is someone already looking into to this? I might have some time to spare next month, best case in the summer..

Comment 3 Ondřej Lysoněk 2018-03-29 07:54:32 UTC
Reproducer without tuned or kernel scriptlets involvement:

[root@f27-clean ~]# echo 'set foo="bar=1"' >> /etc/grub2.cfg 
[root@f27-clean ~]# grep 'set foo' /etc/grub2.cfg 
set foo="bar=1"
[root@f27-clean ~]# grubby --update-kernel $(grubby --default-kernel)
[root@f27-clean ~]# grep 'set foo' /etc/grub2.cfg 
set foo="bar=1"=1"
[root@f27-clean ~]# rpm -q grubby
grubby-8.40-8.fc27.x86_64

Comment 4 Ondřej Lysoněk 2018-03-29 09:12:45 UTC
Ha, I knew I'd seen this before :). I see you fixed it upstream some years ago, but for some reason the fix is not in the Fedora's grubby:
https://github.com/rhboot/grubby/commit/a6649d6268f5e1fa8e0afe7c713be53fda2ef8b7

Comment 5 Christopher Tubbs 2018-03-29 22:12:16 UTC
I saw this on F27, and can provide more details, as needed.

See this devel post for more details on my experience: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/CKRJZKQESEMLE2LOOGXRA5KYR55X7HNN/#CKRJZKQESEMLE2LOOGXRA5KYR55X7HNN

Comment 6 Ted 2018-04-27 15:39:52 UTC
I am running into this bug on Fedora 27 when setting the network-latency profile in tuned then installing a new kernel via grubby.  Result is the dreaded grub> prompt after rebooting.

Comment 7 Fedora End Of Life 2018-05-03 08:24:38 UTC
This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 8 Dan Siemon 2018-08-09 20:57:26 UTC
This bug is hurting us pretty badly on Fedora 28 (and 26 and 27). We have a large number of appliances running Fedora and use tuned to apply a power profile. This results in the mess below in grub.cfg which causes the boot to fail at the grub> prompt.

### BEGIN /etc/grub.d/00_tuned ###                                              
set tuned_initrd=""                                                             
set tuned_params="skew_tick=1"=1"=1"=1"=1"=1"=1"=1"=1"=1"=1"=1"=1"=1"=1"=1"=1"= 
1"=1"=1"                                                                        
### END /etc/grub.d/00_tuned ###

Comment 9 Jaroslav Škarvada 2018-08-09 21:00:10 UTC
Could we get the fix linked from comment 4 to fedora please?

Comment 10 Sebastian 2018-09-13 15:36:50 UTC
Same problem here and a fix would be really cool, especially as there is already a solution.

Comment 11 Sebastian 2018-09-13 15:41:04 UTC
*** Bug 1618684 has been marked as a duplicate of this bug. ***

Comment 12 Dan Siemon 2018-10-17 14:05:18 UTC
This started in Fedora 26, is currently tagged as 27 and is still a problem in 28.

Is there anything we can do to help get this fixed? It's pretty painful for us.

Comment 13 lm 2018-11-04 15:06:47 UTC
Today i attempted to install a new fedora 29 on a toshiba satellite and i met this bug. 

i download Fedora-Workstation-Live-x86_64-29-1.2.iso, i used unetbootin to produce a bootable pendrive, and after i turned on the pc i met this bug.

Comment 14 Joe Mario 2018-11-11 21:15:43 UTC
I just hit this on Fedora 28 by doing a simple "dnf update".

Is there an update on when this will be pulled into Fedora?

Comment 15 Phea Duch 2018-11-14 23:16:58 UTC
Lost my grub2 menu after doing a "dnf update" with kernel-4.20.0-0.rc2.git0.1.fc30.x86_64.

Comment 16 Ted 2018-11-15 19:54:51 UTC
Still broken in Fedora 29.

Comment 17 Ben Cotton 2018-11-27 13:34:01 UTC
This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 18 Jaroslav Škarvada 2019-04-02 08:57:51 UTC
FYI the problem should go away with the switch to the BLS which is planned to be the default option for f30:
https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault

The BLS support is there even in f29, it's just not enabled by default and can be switched on manually. Switching to BLS should resolve this problem.

Comment 19 Jesse Brandeburg 2019-04-30 23:32:44 UTC
That's nice that BLS support is there, doesn't help me very much that I just did a dnf update and reboot after running tuned-adm, and had an endless "=1" string from tuned, causing boot failure.  I manually fixed my grub.cfg, but my system still won't boot.

Comment 20 Jaroslav Škarvada 2019-05-03 09:09:01 UTC
(In reply to Jesse Brandeburg from comment #19)
> That's nice that BLS support is there, doesn't help me very much that I just
> did a dnf update and reboot after running tuned-adm, and had an endless "=1"
> string from tuned, causing boot failure.  I manually fixed my grub.cfg, but
> my system still won't boot.

Did you enabled BLS? AFAIK the problem is caused by grubby and with BLS the grubby is not used. It worked for me on the testing system.

Comment 21 Jesse Brandeburg 2019-05-03 19:58:45 UTC
So, I had presumed that I just needed to install an RPM to switch to BLS, so I had installed grubby-bls, but that wasn't what I needed.  Once I read the whole BootLoaderSpecByDefault, I realized all I had to do was try grub2-switch-to-blscfg.
I did that and rebooted with the 5.0.9-200 kernel and things are working so far.

I see that after running tuned-adm, /boot/efi/EFI/fedora/grubenv has the tuned_params=skew_tick=1, but not the recursive, buggy set tuned_params="skew_tick=1"=1"=1"=1" ad-nauseum.

So my conclusion is that the suggestion is a good workaround, but not really a bug fix for whatever causes the =1"=1"=1" thing.

Comment 22 Ben Cotton 2019-10-31 19:24:07 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 23 Ben Cotton 2019-11-27 22:26:41 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.