Bug 1783542
| Summary: | In-place upgrade fails to convert the zipl configuration to BLS for RHEL-ALT qemu-kvm guest on s390x | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | smitterl |
| Component: | leapp-repository | Assignee: | Michal Hečko <mhecko> |
| Status: | CLOSED ERRATA | QA Contact: | smitterl |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.6 | CC: | amike, bugproxy, dhorak, dzheng, fkrska, fmartine, jcastran, lagordon, leapp-notifications, mbocek, mdeng, ngu, podvody, pstodulk, qzhang, smitterl, tstaudt |
| Target Milestone: | rc | Keywords: | Reopened, Upgrades |
| Target Release: | 7.9 | Flags: | pm-rhel:
mirror+
|
| Hardware: | s390x | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | leapp-repository-0.15.0-1.el7_9 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-11-10 09:13:09 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1663362, 1729246, 1818088, 1857216 | ||
| Attachments: | |||
|
Description
smitterl
2019-12-13 21:34:20 UTC
(In reply to smitterl from comment #0) [snip] > [ 52.796396] localhost zipl[870]: Boot loader written to vda (0000) - fd:00 This output comes from zipl and it doesn't give an indication of what failed. > [ 158.443139] localhost upgrade[553]: 2019-12-14 04:58:10.523 ERROR PID: > 560 leapp.workflow.Preparation.zipl_convert_to_blscfg: Failed to execute > zipl-switch-to-blscfg to convert the zipl configuration to BLS > [ 158.443139] localhost upgrade[553]: Traceback (most recent call last): > [ 158.443139] localhost upgrade[553]: File > "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/ > ziplconverttoblscfg/actor.py", line 51, in process > [ 158.443139] localhost upgrade[553]: > context.call(['/usr/sbin/zipl-switch-to-blscfg']) > [ 158.443139] localhost upgrade[553]: File > "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/ > mounting.py", line 156, in call > [ 158.443139] localhost upgrade[553]: return > run(self.type.make_command(cmd), *args, **kwargs) > [ 158.443139] localhost upgrade[553]: File > "/usr/lib/python2.7/site-packages/leapp/libraries/stdlib/__init__.py", line > 187, in run > [ 158.443139] localhost upgrade[553]: result=result > [ 158.443139] localhost upgrade[553]: CalledProcessError: Command > ['systemd-nspawn', '--register=no', '--quiet', '-D', > u'/var/lib/leapp/el8userspace', > '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/el7toel8/ > tools', > '--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/el7toel8/ > files', '--setenv=LEAPP_HOSTNAME=localhost.localdomain', > '--setenv=LEAPP_EXECUTION_ID=3d04221f-2b21-4dce-959c-44422274ab30', > '/usr/sbin/zipl-switch-to-blscfg'] failed with exit code 1. So the zipl-switch-to-blscfg script exited with status 1. The script prints an error message stderr, can we get the output of the script to figure out what went wrong? So regarding the latest info, the issue is not possible to reproduce anymore. As we are not aware what happens before, I am closing the bug rather with insufficient_data status as it could happen the issue could appear again in future. In case the issue appears again, please reopen the bug with all required info. Thank you guys for cooperation. (In reply to Petr Stodulka from comment #15) > So regarding the latest info, the issue is not possible to reproduce > anymore. As we are not aware what happens before, I am closing the bug > rather with insufficient_data status as it could happen the issue could > appear again in future. In case the issue appears again, please reopen the > bug with all required info. Thank you guys for cooperation. Petr, unfortunately, this is still reproducible on the RHEL ALT 7.6 guest. My last info was about RHEL 7.6. However, this doesn't seem RHEL ALT 7.6 specific but might be due to an unexpected zipl.conf. There might be a workaround, removing entry from /etc/zipl.conf and running zipl before rebooting. I added more info to the actor log: https://github.com/oamg/leapp-repository/pull/463 1. The error output from zipl-swtich-to-blscfg: ... [ 53.502244] upgrade[554]: BLS file /boot/loader/entries/b9085c88059317bdf6f241658a1ec532-0-rescue.conf already exists ... 2. The zipl.conf before reboot: [root@localhost libraries]# cat /etc/zipl.conf [defaultboot] defaultauto prompt=1 timeout=5 default=RHEL-Upgrade-Initramfs target=/boot [RHEL-Upgrade-Initramfs] image=/boot/vmlinuz-upgrade.s390x parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap LANG=en_US.UTF-8 enforcing=0 rd.plymouth=0 plymouth.enable=0" ramdisk=/boot/initramfs-upgrade.s390x.img [Red_Hat_Enterprise_Linux_Server_7.6_Rescue_b9085c88059317bdf6f241658a1ec532] image=/boot/vmlinuz-0-rescue-b9085c88059317bdf6f241658a1ec532 parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap LANG=en_US.UTF-8" ramdisk=/boot/initramfs-0-rescue-b9085c88059317bdf6f241658a1ec532.img [4.14.0-115.17.1.el7a.s390x] image=/boot/vmlinuz-4.14.0-115.17.1.el7a.s390x parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap LANG=en_US.UTF-8" ramdisk=/boot/initramfs-4.14.0-115.17.1.el7a.s390x.img [linux-0-rescue-55d8db5ca0b34b9e8c38668de7307410] image=/boot/vmlinuz-0-rescue-55d8db5ca0b34b9e8c38668de7307410 ramdisk=/boot/initramfs-0-rescue-55d8db5ca0b34b9e8c38668de7307410.img parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap" 3. Remove last entry ([linux-0-rescue-55d8db5ca0b34b9e8c38668de7307410]) and run zipl before reboot. Result: 1. The upgrade process finishes with "Complete" and boots into RHEL 8.1. 2. I can log in and have network. (In reply to smitterl from comment #16) > (In reply to Petr Stodulka from comment #15) > > So regarding the latest info, the issue is not possible to reproduce > > anymore. As we are not aware what happens before, I am closing the bug > > rather with insufficient_data status as it could happen the issue could > > appear again in future. In case the issue appears again, please reopen the > > bug with all required info. Thank you guys for cooperation. > > Petr, > > unfortunately, this is still reproducible on the RHEL ALT 7.6 guest. My last > info was about RHEL 7.6. However, this doesn't seem RHEL ALT 7.6 specific > but might be due to an unexpected zipl.conf. > > There might be a workaround, removing entry from /etc/zipl.conf and running > zipl before rebooting. > I added more info to the actor log: > https://github.com/oamg/leapp-repository/pull/463 > > 1. The error output from zipl-swtich-to-blscfg: > > ... > [ 53.502244] upgrade[554]: BLS file > /boot/loader/entries/b9085c88059317bdf6f241658a1ec532-0-rescue.conf already I think this is expected if there are BLS snippets already created. Did you install the rhel8 kernels before calling the zipl-swtich-to-blscfg script? > exists > ... > > 2. The zipl.conf before reboot: > [root@localhost libraries]# cat /etc/zipl.conf > [defaultboot] > defaultauto > prompt=1 > timeout=5 > default=RHEL-Upgrade-Initramfs > target=/boot > [RHEL-Upgrade-Initramfs] > image=/boot/vmlinuz-upgrade.s390x > parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 > reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root > rd.lvm.lv=rhel/swap LANG=en_US.UTF-8 enforcing=0 rd.plymouth=0 > plymouth.enable=0" > ramdisk=/boot/initramfs-upgrade.s390x.img > [Red_Hat_Enterprise_Linux_Server_7.6_Rescue_b9085c88059317bdf6f241658a1ec532] > image=/boot/vmlinuz-0-rescue-b9085c88059317bdf6f241658a1ec532 > parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 > reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root > rd.lvm.lv=rhel/swap LANG=en_US.UTF-8" > ramdisk=/boot/initramfs-0-rescue-b9085c88059317bdf6f241658a1ec532.img > [4.14.0-115.17.1.el7a.s390x] > image=/boot/vmlinuz-4.14.0-115.17.1.el7a.s390x > parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 > reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root > rd.lvm.lv=rhel/swap LANG=en_US.UTF-8" > ramdisk=/boot/initramfs-4.14.0-115.17.1.el7a.s390x.img > [linux-0-rescue-55d8db5ca0b34b9e8c38668de7307410] > image=/boot/vmlinuz-0-rescue-55d8db5ca0b34b9e8c38668de7307410 > ramdisk=/boot/initramfs-0-rescue-55d8db5ca0b34b9e8c38668de7307410.img > parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 > reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root > rd.lvm.lv=rhel/swap" > > 3. Remove last entry ([linux-0-rescue-55d8db5ca0b34b9e8c38668de7307410]) and > run zipl before reboot. > > Result: > 1. The upgrade process finishes with "Complete" and boots into RHEL 8.1. > 2. I can log in and have network. What's the content of the zipl.conf file after the upgrade is complete? > > [ 53.502244] upgrade[554]: BLS file
> > /boot/loader/entries/b9085c88059317bdf6f241658a1ec532-0-rescue.conf already
>
> I think this is expected if there are BLS snippets already created. Did you install the rhel8 kernels before calling the zipl-swtich-to-blscfg script?
If I remember right, first is called the script before the installation of the rhel-8 kernel, based on your instructions before. That's why it is (or was?) working on machines we tried. Including some RHEL-ALT s390x machines as well. Not sure what exact difference is here in this case or whether something has been changed.
Hello Petr, (In reply to Petr Stodulka from comment #18) > > > [ 53.502244] upgrade[554]: BLS file > > > /boot/loader/entries/b9085c88059317bdf6f241658a1ec532-0-rescue.conf already > > > > I think this is expected if there are BLS snippets already created. Did you install the rhel8 kernels before calling the zipl-swtich-to-blscfg script? > > If I remember right, first is called the script before the installation of > the rhel-8 kernel, based on your instructions before. That's why it is (or > was?) working on machines we tried. Including some RHEL-ALT s390x machines > as well. Not sure what exact difference is here in this case or whether > something has been changed. Yes, I remember having some conversations about this with you. Although in this case it seems to be the opposite, that the script is being called after the kernel has been installed? Because after executing zipl-swtich-to-blscfg the zipl.conf file should not have any boot entry sections, and in this case it does according to Comment 16. I can reliably reproduce this issue with a kvm guest. I'm listing here the zipl.conf of the ALT 7.6 system at several stages of the upgrade process. ======================= zipl.conf - base system ======================= [defaultboot] defaultauto prompt=1 timeout=5 default=linux target=/boot [linux] image=/boot/vmlinuz-4.14.0-115.el7a.s390x ramdisk=/boot/initramfs-4.14.0-115.el7a.s390x.img parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap LANG=en_US.UTF-8" [linux-0-rescue-55d8db5ca0b34b9e8c38668de7307410] image=/boot/vmlinuz-0-rescue-55d8db5ca0b34b9e8c38668de7307410 ramdisk=/boot/initramfs-0-rescue-55d8db5ca0b34b9e8c38668de7307410.img parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap" =============================== zipl.conf - after yum update -y =============================== [defaultboot] defaultauto prompt=1 timeout=5 default=4.14.0-115.36.1.el7a.s390x target=/boot [Red_Hat_Enterprise_Linux_Server_7.6_Rescue_b9085c88059317bdf6f241658a1ec532] image=/boot/vmlinuz-0-rescue-b9085c88059317bdf6f241658a1ec532 parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap LANG=en_US.UTF-8" ramdisk=/boot/initramfs-0-rescue-b9085c88059317bdf6f241658a1ec532.img [4.14.0-115.36.1.el7a.s390x] image=/boot/vmlinuz-4.14.0-115.36.1.el7a.s390x parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap LANG=en_US.UTF-8" ramdisk=/boot/initramfs-4.14.0-115.36.1.el7a.s390x.img [linux] image=/boot/vmlinuz-4.14.0-115.el7a.s390x ramdisk=/boot/initramfs-4.14.0-115.el7a.s390x.img parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap LANG=en_US.UTF-8" [linux-0-rescue-55d8db5ca0b34b9e8c38668de7307410] image=/boot/vmlinuz-0-rescue-55d8db5ca0b34b9e8c38668de7307410 ramdisk=/boot/initramfs-0-rescue-55d8db5ca0b34b9e8c38668de7307410.img parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap" ========================================================================= zipl.conf - after rebooting into latest kernel and yum remove old kernels ========================================================================= [defaultboot] defaultauto prompt=1 timeout=5 default=4.14.0-115.36.1.el7a.s390x target=/boot [Red_Hat_Enterprise_Linux_Server_7.6_Rescue_b9085c88059317bdf6f241658a1ec532] image=/boot/vmlinuz-0-rescue-b9085c88059317bdf6f241658a1ec532 parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap LANG=en_US.UTF-8" ramdisk=/boot/initramfs-0-rescue-b9085c88059317bdf6f241658a1ec532.img [4.14.0-115.36.1.el7a.s390x] image=/boot/vmlinuz-4.14.0-115.36.1.el7a.s390x parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap LANG=en_US.UTF-8" ramdisk=/boot/initramfs-4.14.0-115.36.1.el7a.s390x.img [linux-0-rescue-55d8db5ca0b34b9e8c38668de7307410] image=/boot/vmlinuz-0-rescue-55d8db5ca0b34b9e8c38668de7307410 ramdisk=/boot/initramfs-0-rescue-55d8db5ca0b34b9e8c38668de7307410.img parameters="root=/dev/mapper/rhel-root console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap" Created attachment 1790194 [details]
leapp-preupgrade.log after rebooting back into 7 after error during leapp upgrade
*** Bug 1969278 has been marked as a duplicate of this bug. *** ------- Comment From thorsten.diehl.com 2021-06-08 12:49 EDT------- (In reply to comment #11) > Are there more details stored somewhere why the zipl-switch-to-blscfg tool > failed? Hmmm. Where should I search for it? Created attachment 1790206 [details]
console_output_after_reboot
Created attachment 1790207 [details]
console_output_in_emergency_shell
Created attachment 1790208 [details]
console_output_after_2nd_reboot
Created attachment 1790209 [details]
leapp-preupgrade output after fixing inhibitors
Created attachment 1790210 [details]
output of leapp-upgrade with errors
Created attachment 1790211 [details]
output of leapp-upgrade after fixing error
(In reply to IBM Bug Proxy from comment #37) > Created attachment 1790210 [details] > output of leapp-upgrade with errors Here you seem to have hit a known issue bz1783248, (details below from attachment leapp-upgrade.out). The workaround is to create an empty config file. This doesn't change the system 'touch /etc/dasd.conf'. Known issues - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/upgrading_from_rhel_7_to_rhel_8/index#known-issues_troubleshooting IBM BZ - https://bugzilla.linux.ibm.com/show_bug.cgi?id=185878 RH BZ - https://bugzilla.redhat.com/show_bug.cgi?id=1783248 " context.copy_to('/etc/dasd.conf', '/etc/dasd.conf') File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/mounting.py", line 210, in copy_to shutil.copy2(src, self.full_path(dst)) File "/usr/lib64/python2.7/shutil.py", line 130, in copy2 copyfile(src, dst) File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile with open(src, 'rb') as fsrc: IOError: [Errno 2] No such file or directory: '/etc/dasd.conf'" ------- Comment From thorsten.diehl.com 2021-06-23 09:53 EDT------- *** Bug 193104 has been marked as a duplicate of this bug. *** Pre-verified with:
leapp-repository-0.14.0-0.202108182050Z.ec19d1e.ec19d1e2_check_multiple_zipl_rescue_entries.PR700.el7_9
Expectation: multiple rescue entries are considered critical blocker
Steps as in description
==>
Preupgrade reports error
"Upgrade has been inhibited due to the following problems:
1. Inhibitor: Multiple rescue boot entries present in the bootloader configuration.
2. Inhibitor: Multiple kernels installed
3. Inhibitor: Possible problems with remote login using root account
4. Inhibitor: Missing required answers in the answer file
Consult the pre-upgrade report for details and possible remediation.
"
After solving other issues:
"
Upgrade has been inhibited due to the following problems:
1. Inhibitor: Multiple rescue boot entries present in the bootloader configuration.
Consult the pre-upgrade report for details and possible remediation.
"
After removing some rescue entry process can continue and upgrade succeeds (-> 8.4, kernel 4.18.0-305.12.1.el8_4.s390x).
Thanks Sebastian! I am setting th POST state for now. We will set the MODIFIED state once we have changes applied in the dist git / the brew build is prepared. Verified with: compose id EXTRAS-7.9-RHEL-7-20211104.n.0 leapp-upgrade-el7toel8-0.15.0-6.el7_9.noarch (~ leapp-repository) Details like in comment#41 Target system could be booted. Target RHEL 8. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (leapp, leapp-repository, and cockpit-leapp bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4603 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |