Bug 1827397

== Comment: #0 - Murilo Opsfelder Araujo <muriloo.com> - 2020-04-22 15:57:42 ==
Fedora Rawhide installation is broken on ppc64le because grub2-mkconfig returns non-zero status code.

Anaconda displays an installation error:

Installing boot loader
================================================================================
================================================================================

Question

The following error occurred while installing the boot loader. The system will
not be bootable. Would you like to ignore this and continue with installation?

failed to write boot loader configuration

Please respond 'yes' or 'no':


On /tmp/storage.log, there is:

DEBUG:anaconda.bootloader.installation:Installing the boot loader.
INFO:anaconda.bootloader.installation:boot loader stage1 target device is sda2
INFO:anaconda.bootloader.installation:boot loader stage2 target device is sda1
INFO:anaconda.bootloader.grub2:bootloader.py: used boot args: debug rd.lvm.lv=fedora_ltcmihawk60/root rd.lvm.lv=fedora_ltcmihawk60/swap
INFO:program:Running in chroot '/mnt/sysroot'... grub2-set-default 814a7471023b4e5abdc80b8f8426d953-5.7.0-0.rc2.1.fc33.ppc64le
DEBUG:program:Return code: 0
INFO:program:Running in chroot '/mnt/sysroot'... grub2-mkconfig -o /boot/grub2/grub.cfg
INFO:program:Generating grub configuration file ...
DEBUG:program:Return code: 1
ERROR:anaconda.modules.storage.bootloader.installation:Bootloader installation has failed: failed to write boot loader configuration
INFO:anaconda.threading:Thread Failed: AnaTaskThread-InstallBootloaderTask-1 (140736001667456)
ERROR:anaconda.modules.common.task.task:Thread AnaTaskThread-InstallBootloaderTask-1 has failed: Traceback (most recentcall last):
File "/usr/lib64/python3.8/site-packages/pyanaconda/threading.py", line 280, in run
threading.Thread.run(self)
File "/usr/lib64/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.8/site-packages/pyanaconda/modules/common/task/task.py", line 95, in _task_run_callback
self._set_result(self.run())
File "/usr/lib64/python3.8/site-packages/pyanaconda/modules/storage/bootloader/installation.py", line 106, in run
raise BootloaderInstallationError(str(e)) from None
pyanaconda.modules.common.errors.installation.BootloaderInstallationError: failed to write boot loader configuration


Investigating this, the error comes from a "read" command in /etc/grub.d/10_linux script:

191    arch="$(uname -m)"
192    if [ "x${arch}" = "xppc64le" ] && [ -d /sys/firmware/opal ]; then
193
194        petitboot_path="/sys/firmware/devicetree/base/ibm,firmware-versions/petitboot"
195
196        if test -e ${petitboot_path}; then
197            read -a petitboot_version < ${petitboot_path}
198            petitboot_version="$(echo ${petitboot_version//v})"
199            major_version="$(echo ${petitboot_version} | cut -d . -f1)"
200            minor_version="$(echo ${petitboot_version} | cut -d . -f2)"
201
202            if test -z ${petitboot_version} || test ${major_version} -lt 1 || \
203                    test ${major_version} -eq 1 -a ${minor_version} -lt 8; then
204                BLS_POPULATE_MENU="true"
205            fi
206        else
207            BLS_POPULATE_MENU="true"
208        fi
209    fi

The line 197 above returns non-zero because "read" finds end-of-file.  See:

[anaconda root@ltcmihawk60 /]# read -a petitboot_version < /sys/firmware/devicetree/base/ibm,firmware-versions/petitboot
[anaconda root@ltcmihawk60 /]# echo $?
1
[anaconda root@ltcmihawk60 /]# cat /sys/firmware/devicetree/base/ibm,firmware-versions/petitboot
v1.11[anaconda root@ltcmihawk60 /]#

From bash(1) manpage:

.              If no names are supplied, the line read is assigned to the
.              variable REPLY.  The exit status is zero, unless end-of-file
.              is encountered, read times out (in which case the status is
.              greater than 128), a variable assignment error (such as
.              assigning to a readonly variable) occurs, or an invalid file
.              descriptor is supplied as the argument to -u.

Since /etc/grub.d/10_linux has "set -v", it exits immediately, propagating the error up to grub2-mkconfig script.

This issue seems to have been introduced by commit:

https://src.fedoraproject.org/rpms/grub2/c/7509e59c4a99ff3b1cd479bcb102e5ab41123abf?branch=master

(In reply to IBM Bug Proxy from comment #1)
> == Comment: #0 - Murilo Opsfelder Araujo <muriloo.com> - 2020-04-22

[snip]

> The line 197 above returns non-zero because "read" finds end-of-file.  See:
> 
> [anaconda root@ltcmihawk60 /]# read -a petitboot_version <
> /sys/firmware/devicetree/base/ibm,firmware-versions/petitboot
> [anaconda root@ltcmihawk60 /]# echo $?
> 1

Sigh, sorry about the breakage. I missed that subtlety of the read command...

Could you please give a try to the following test build?

https://koji.fedoraproject.org/koji/taskinfo?taskID=43727721

------- Comment From muriloo.com 2020-04-24 11:10 EDT-------
> (In reply to IBM Bug Proxy from comment #1)
> > == Comment: #0 - Murilo Opsfelder Araujo <muriloo.com> - 2020-04-22
> [snip]
>
> > The line 197 above returns non-zero because "read" finds end-of-file.  See:
> >
> > [anaconda root@ltcmihawk60 /]# read -a petitboot_version <
> > /sys/firmware/devicetree/base/ibm,firmware-versions/petitboot
> > [anaconda root@ltcmihawk60 /]# echo $?
> > 1
>
> Sigh, sorry about the breakage. I missed that subtlety of the read command...
>
> Could you please give a try to the following test build?
>
> https://koji.fedoraproject.org/koji/taskinfo?taskID=43727721

This build seems to work despite the some error messages, but at the end,
grub2-mkconfig returned zero.

After installing packages and re-running grub2-mkconfig, I entered "yes" on the
Anaconda panel and installation proceeded successfully.

[anaconda root@ltcmihawk60 /]# rpm -Uvh ./grub2-*.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
1:grub2-common-1:2.04-15.fc33      ################################# [ 10%]
2:grub2-ppc64le-modules-1:2.04-15.f################################# [ 20%]
3:grub2-tools-1:2.04-15.fc33       ################################# [ 30%]
4:grub2-tools-minimal-1:2.04-15.fc3################################# [ 40%]
5:grub2-ppc64le-1:2.04-15.fc33     ################################# [ 50%]
Cleaning up / removing...
6:grub2-ppc64le-1:2.04-13.fc33     ################################# [ 60%]
7:grub2-ppc64le-modules-1:2.04-13.f################################# [ 70%]
8:grub2-tools-1:2.04-13.fc33       ################################# [ 80%]
9:grub2-tools-minimal-1:2.04-13.fc3################################# [ 90%]
10:grub2-common-1:2.04-13.fc33      ################################# [100%]
Running in chroot, ignoring request: daemon-reload

[anaconda root@ltcmihawk60 /]# rpm -qa | grep -i grub
grub2-ppc64le-modules-2.04-15.fc33.noarch
grub2-common-2.04-15.fc33.noarch
grub2-tools-2.04-15.fc33.ppc64le
grub2-ppc64le-2.04-15.fc33.ppc64le
grubby-8.40-41.fc33.ppc64le
grub2-tools-minimal-2.04-15.fc33.ppc64le

[anaconda root@ltcmihawk60 /]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
/etc/grub.d/10_linux: line 197: warning: command substitution: ignored null byte in input
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
done
0

Cheers!

-Murilo

(In reply to IBM Bug Proxy from comment #3)
> ------- Comment From muriloo.com 2020-04-24 11:10 EDT-------

[snip]

> 
> This build seems to work despite the some error messages, but at the end,
> grub2-mkconfig returned zero.
>

Thanks a lot for testing!

[snip]
 
> 
> [anaconda root@ltcmihawk60 /]# grub2-mkconfig -o /boot/grub2/grub.cfg
> Generating grub configuration file ...
> /etc/grub.d/10_linux: line 197: warning: command substitution: ignored null

Hmm, I wasn't able to get access to a ppc64le OPAL machine and that's why I couldn't test all these corner cases.

Could you please share the output of:

$ hexdump -C /sys/firmware/devicetree/base/ibm,firmware-versions/petitboot

> byte in input
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> logger: symbol lookup error: /lib64/libgcrypt.so.20: undefined symbol: dlopen
> done
> 0
>

Weird, I don't know that's the issue with libgcrypt in Rawhide.
 
> Cheers!
> 
> -Murilo

Best regards

------- Comment From muriloo.com 2020-04-27 10:15 EDT-------
I've collected two POWER9 samples from different machines with different firmware levels:

[root@ltcmihawk60 ~]# hexdump -C /sys/firmware/devicetree/base/ibm,firmware-versions/petitboot
00000000  76 31 2e 31 31 00                                 |v1.11.|
00000006
[root@ltcmihawk60 ~]# cat /sys/firmware/devicetree/base/ibm,firmware-versions/petitboot
v1.11[root@ltcmihawk60 ~]#

[root@ltc-hab1 ~]# hexdump -C /sys/firmware/devicetree/base/ibm,firmware-versions/petitboot
00000000  76 31 2e 31 30 2e 33 00                           |v1.10.3.|
00000008
[root@ltc-hab1 ~]# cat /sys/firmware/devicetree/base/ibm,firmware-versions/petitboot
v1.10.3[root@ltc-hab1 ~]#

Thanks a lot for the information, I've fixed this in https://koji.fedoraproject.org/koji/buildinfo?buildID=1499982

Please test with that build and let me know if there are issues remaining.

------- Comment From muriloo.com 2020-05-11 17:47 EDT-------
Javier,

Just to let you know that today, 2020-05-11, I was able to install Fedora Rawhide on the same machine used to report this bug.

This issue can be closed.

Thank you!

Murilo

(In reply to IBM Bug Proxy from comment #7)
> ------- Comment From muriloo.com 2020-05-11 17:47 EDT-------
> Javier,
> 
> Just to let you know that today, 2020-05-11, I was able to install Fedora
> Rawhide on the same machine used to report this bug.
> 
> This issue can be closed.
> 
> Thank you!
> 
> Murilo

Thanks a lot for testing and for providing all the needed information.

------- Comment From muriloo.com 2020-05-11 17:48 EDT-------
For the record:

[root@ltcmihawk60 ~]# uname -a
Linux ltcmihawk60.aus.stglabs.ibm.com 5.7.0-0.rc4.20200508git79dede78c057.1.fc33.ppc64le #1 SMP Fri May 8 14:59:29 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux

[root@ltcmihawk60 ~]# rpm -qa | grep grub | sort
grub2-common-2.04-15.fc33.noarch
grub2-ppc64le-2.04-15.fc33.ppc64le
grub2-ppc64le-modules-2.04-15.fc33.noarch
grub2-tools-2.04-15.fc33.ppc64le
grub2-tools-minimal-2.04-15.fc33.ppc64le
grubby-8.40-44.fc33.ppc64le