Bug 1872356

Summary: Leapp upgrade may lead to the grub prompt
Product: Red Hat Enterprise Linux 7 Reporter: Christophe Besson <cbesson>
Component: leapp-repositoryAssignee: Dominik Rehák <drehak>
Status: CLOSED ERRATA QA Contact: Alois Mahdal <amahdal>
Severity: high Docs Contact:
Priority: high    
Version: 7.8CC: drehak, fkrska, mkluson, pstodulk
Target Milestone: rcKeywords: Reproducer
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: leapp-repository-0.13.0-1.el7_9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-16 08:59:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1818077, 1818088    

Description Christophe Besson 2020-08-25 14:35:51 UTC
Description of problem:
After the "leapp upgrade" phase, all looks good, the system is rebooted on the specific initramfs used for the upgrade, updates are done, and then the system fails to reboot. It ends at the grub prompt.

Version-Release number of selected component (if applicable):
leapp-0.10.0-2.el7_8.noarch
leapp-repository-0.10.0-2.el7_8.noarch

How reproducible:
100%

Steps to Reproduce:
1. Use a custom /etc/default/grub with blank lines between each variable (or with comment after the line containing GRUB_CMDLINE_LINUX=)
2. Proceed to the leapp upgrade and observe after the (second) reboot.

Actual results:
It ends at the grub cli prompt, as if grub.cfg doesn't exist.

Expected results:
Provide the standard grub menu with the RHEL 8 entry (with old entries as well).

Additional info:
After rescuing the el8 system, we can notice /etc/default/grub contains:
~~~
GRUB_CMDLINE_LINUX="elevator=noop
~~~

It lacks the misleading double-quote.

Here is the original file used by the customer, it was valid:
~~~
# WARNING: This file maintained by Puppet.
# Editing is no use unless you avoid running the Puppet agent

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
##GRUB_HIDDEN_TIMEOUT=0
##
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"

GRUB_CMDLINE_LINUX_RECOVERY="true"

GRUB_CMDLINE_LINUX="elevator=noop"

# Uncomment to disable graphical terminal (grub-pc only)

#GRUB_TERMINAL="console"

GRUB_GFXMODE=640x480
GRUB_DISABLE_RECOVERY="true"


# Set a background image
# Image must respect various specifications
GRUB_BACKGROUND="(hd0,0)/grub/splash.xpm.gz"

# Disable OS prober
GRUB_DISABLE_OS_PROBER="true"
~~~

It seems the pattern used in /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/detectgrubconfigerror/libraries/scanner.py is wrong:
~~~
pattern = r'GRUB_CMDLINE_LINUX="[^"]+"(?!(\s*$)|(\s+GRUB))'
~~~

Looks like it expects the next line begins with GRUB.

Maybe we can use "grub2-mkconfig -o /tmp/test" to check if the file has a good syntax.

For now, the suggested workaround is to remove blank lines and any comments before launching the upgrade:
~~~
# sed -i -e '/^#/d' -e '/^$/d' /etc/default/grub
~~~

Comment 2 Dominik Rehák 2020-09-14 14:01:19 UTC
Hi Christopher,

the problem seems to be in the comment following GRUB_CMDLINE_LINUX, rather than the newlines. 
This pattern works for the file you provided:

~~~
pattern = r'GRUB_CMDLINE_LINUX="[^"]+"(?!(\s*$)|(\s+GRUB+#))'
~~~

I'm still wondering about some edge cases - can the variable values be multiline? What if there are comments in the middle of them - can those be ignored or are they part of the value?

Comment 3 Christophe Besson 2020-09-14 14:19:52 UTC
Hmm strange, I'm almost sure it didn't work, that's why I suggested the customer to remove blank lines and comments and it worked.

> can the variable values be multiline? What if there are comments in the middle of them - can those be ignored or are they part of the value?

I think it's one line per variable, like in shell. 
If value contains a space, double-quotes are needed. 
If there is a # outside of a double-quote, everything on the right is stripped out.

To my opinion, using grub2-mkconfig is the best way to validate that thing.

Comment 10 Martin Klusoň 2021-02-15 11:59:18 UTC
The bug was verified on versions:
leapp-0.12.0-1.el7_9.noarch
leapp-repository-0.13.0-2.el7_9.noarch

Tests: 
[tmeta.tc.@bootloader/edgjunk-blockcmt_post]
Test run (x86_64 only): https://beaker.engineering.redhat.com/recipes/9543951 - PASS

[tmeta.tc.@bootloader/edgjunk-blockcmt_pre]
Test run (x86_64 only): https://beaker.engineering.redhat.com/recipes/9543952 - PASS

[tmeta.tc.@bootloader/edgjunk-inlinecmt_post]
Test run (x86_64 only): https://beaker.engineering.redhat.com/recipes/9543955 - PASS

[tmeta.tc.@bootloader/edgjunk-inlinews_pre]
Test run (x86_64 only): https://beaker.engineering.redhat.com/recipes/9543958 - PASS

[tmeta.tc.@bootloader/edgjunk-inlinews_post]
Test run (x86_64 only): https://beaker.engineering.redhat.com/recipes/9543957 - PASS

[tmeta.tc.@bootloader/edgjunk-emptyline_pre]
Test run (x86_64 only): https://beaker.engineering.redhat.com/recipes/9543954 - PASS

[tmeta.tc.@bootloader/edgjunk-emptyline_post]
Test run (x86_64 only): https://beaker.engineering.redhat.com/recipes/9543953 - PASS

Comment 12 errata-xmlrpc 2021-02-16 08:59:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (leapp and leapp-repository bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0569