Description of problem: The "/boot/grub2/grubenv" in UEFI RHV host is no longer pointing to the UEFI grubenv. However, it's an independent file and the changes in this file will not have any effect on the boot since UEFI will only refer "/boot/efi/EFI/redhat/grubenv" ==== ~]# ls -l /boot/grub2/ |grep grubenv -rw-r--r--. 1 root root 1024 Dec 4 12:06 grubenv lrwxrwxrwx. 1 root root 25 Sep 20 16:38 grubenv.rpmnew -> ../efi/EFI/redhat/grubenv ~]# ls -l /boot/efi/EFI/redhat/grubenv ls: cannot access /boot/efi/EFI/redhat/grubenv: No such file or directory ===== Here instead of grubenv, the "grubenv.rpmnew" is pointing to the UEFI grubenv and it's a dangling symlink. The squashfs image in the 4.3.6 ISO is also having a similar structure. ==== # unsquashfs redhat-virtualization-host-4.3.6-20191108.0.el7_7.x86_64.liveimg.squashfs Parallel unsquashfs: Using 8 processors 1 inodes (81912 blocks) to write [=========================================|] 81912/81912 100% LiveOS]# mount rootfs.img /tmp/4.3.6_rootfs LiveOS]# ls -l /tmp/4.3.6_rootfs/boot/grub2/ |grep grubenv -rw-r--r--. 1 root root 1024 Nov 8 09:55 grubenv lrwxrwxrwx. 1 root root 25 Nov 8 09:58 grubenv.rpmnew -> ../efi/EFI/redhat/grubenv ==== However, the earlier version of ISO was having the correct structure. ==== SOURCES]# unsquashfs redhat-virtualization-host-4.3-20190409.0.el7_6.x86_64.liveimg.squashfs Parallel unsquashfs: Using 8 processors 1 inodes (81912 blocks) to write ======================================================================\] 81912/81912 100% SOURCES]# mount squashfs-root/LiveOS/rootfs.img /tmp/4.3.0_rootfs SOURCES]# ls -l /tmp/4.3.0_rootfs/boot/grub2/ |grep grubenv lrwxrwxrwx. 1 root root 25 Apr 10 2019 grubenv -> ../efi/EFI/redhat/grubenv ==== This same incorrect structure will be copied by Anaconda from squashfs to the newly installed OS and hence every UEFI RHV hosts will be having this issue. ==== Dec 03 20:03:00 localhost program[1678]: Running... mount -t auto -o ro /mnt/install/image/LiveOS/rootfs.img /mnt/install/source Dec 03 20:03:00 localhost kernel: EXT4-fs (loop4): mounted filesystem with ordered data mode. Opts: (null) Dec 03 20:03:00 localhost program[1678]: Return code: 0 Dec 03 20:03:00 localhost program[1678]: Running... rsync -pogAXtlHrDx --exclude /dev/ --exclude /proc/ --exclude /sys/ --exclude /run/ --exclude /boot/*rescue* --exclude /etc/machine-id /mnt/install/source/ /mnt/sysimage ==== I don't know how the squashfs is created however I think the changes in the grub2-efi package as per bug 1497918 seem to be causing this issue. This changed the grubenv to config(noreplace). ==== diff --git a/grub.macros b/grub.macros index cd95489..ba7d8f2 100644 --- a/grub.macros +++ b/grub.macros @@ -494,7 +494,7 @@ cd .. \ %attr(0700,root,root)/boot/efi/EFI/%{efidir}/%{2} \ %attr(0700,root,root)/boot/efi/EFI/%{efidir}/fonts \ %ghost %config(noreplace) %attr(0700,root,root)/boot/efi/EFI/%{efidir}/grub.cfg \ -/boot/grub2/grubenv \ +%config(noreplace) /boot/grub2/grubenv \ %ghost %config(noreplace) %attr(0700,root,root)/boot/efi/EFI/%{efidir}/grubenv \ ==== So I think the grubenv was already available and hence the installation of grub2-efi created the grubenv as grubenv.rpmnew since it was already available. Because of this issue, the "rollback" operation of nodectl will not work since it sets the default option in "/boot/grub2/grubenv" which doesn't have any effect of UEFI machine. Version-Release number of selected component (if applicable): RHV-H 4.3.6 How reproducible: 100% Steps to Reproduce: 1. Install an RHV-H 4.3.6 on UEFI server. 2. Check if the "/boot/grub2/grubenv" is symlink to "/boot/efi/EFI/redhat/grubenv 3. The operations like "nodectl rollback" will not work. Actual results: grubenv file is broken in the UEFI RHV-H hosts. Expected results: "/boot/grub2/grubenv" should point to "/boot/efi/EFI/redhat/grubenv" in EFI machines. Additional info:
Relates to bug 1765250 (duplicate perhaps)
(In reply to Yuval Turgeman from comment #1) > Relates to bug 1765250 (duplicate perhaps) Probably. But I think the root cause of that issue is still unknown since we are not able to reproduce that one.
grub2-install (running in anaconda) creates /boot/grub2/grubenv, so grub2-efi-x64 installs the new grubenv (a symlink) as .rpmnew 16:45:04,430 INFO program: Installing : 1:grub2-efi-x64-2.02-0.80.el7.x86_64 251/545 16:45:04,431 INFO program: warning: /boot/grub2/grubenv created as /boot/grub2/grubenv.rpmnew after that we install grub2-efi-x64
I tried to reproduce this issue. Test steps: 1. Install RHVH-4.3-20191108.0-RHVH-x86_64-dvd1.iso on UEFI server 2. Check if the "/boot/grub2/grubenv" is symlink to "/boot/efi/EFI/redhat/grubenv # ls -l /boot/grub2/ | grep grubenv 3. Check image rollback # nodectl rollback Test results: 1. The result of step 2 is as follows: # ls -l /boot/grub2/ | grep grubenv -rw-r--r--. 1 root root 1024 Dec 19 03:22 grubenv lrwxrwxrwx. 1 root root 25 Nov 8 04:28 grubenv.rpmnew -> ../efi/EFI/redhat/grubenv 2. The result of step 3 is as follows: # nodectl rollback Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 872, in emit stream.write(fs % msg) TypeError: unicode argument expected, got 'str' Logged from file update.py, line 153 Status: Failure Reason: I also performed an upgrade test. Upgrade RHVH from one build (that "/boot/grub2/grubenv" is symlink to "/boot/efi/EFI/redhat/grubenv) to another build (that "grubenv.rpmnew" is symlink to "/boot/efi/EFI/redhat/grubenv"). The test steps are as follows: 1. Install RHVH-4.3-20190410.3-RHVH-x86_64-dvd1.iso and reboot 2. Login to RHVH and check if the "/boot/grub2/grubenv" is symlink to "/boot/efi/EFI/redhat/grubenv # ls -l /boot/grub2/ | grep grubenv 3. Set local repo and point to "redhat-virtualization-host-image-update-4.3.6-20191108.0.el7_7.noarch.rpm" 4. Upgrade RHVH # yum update 5. Reboot and login to RHVH new layer 6. Check if the "/boot/grub2/grubenv" is symlink to "/boot/efi/EFI/redhat/grubenv # ls -l /boot/grub2/ | grep grubenv 7. Check image rollback # nodectl rollback 8. Run command "imgbase rollback" Test results: 1. The result of step 2 is as follows: lrwxrwxrwx. 1 root root 25 Apr 9 2019 grubenv -> ../efi/EFI/redhat/grubenv 2. The result of step 6 is as follows: lrwxrwxrwx. 1 root root 25 Apr 9 2019 grubenv -> ../efi/EFI/redhat/grubenv 3. The result of step 7 is as follows: # nodectl rollback Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 872, in emit stream.write(fs % msg) TypeError: unicode argument expected, got 'str' Logged from file update.py, line 171 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 872, in emit stream.write(fs % msg) TypeError: unicode argument expected, got 'str' Logged from file update.py, line 172 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 872, in emit stream.write(fs % msg) TypeError: unicode argument expected, got 'str' Logged from file update.py, line 180 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 872, in emit stream.write(fs % msg) TypeError: unicode argument expected, got 'str' Logged from file update.py, line 181 Status: Success New layer: rhvh-4.3.0.5-0.20190409.0+1 4. The result of step 8 is correct, after rebooting, RHVH roll back to old layer. # imgbase rollback 2019-12-19 05:56:18,453 [INFO] (MainThread) You are on rhvh-4.3.6.5-0.20191108.0+1.. 2019-12-19 05:56:18,456 [INFO] (MainThread) Rollback to rhvh-4.3.0.5-0.20190409.0+1.. 2019-12-19 05:56:18,509 [INFO] (MainThread) Rollback was successful 2019-12-19 05:56:18,509 [INFO] (MainThread) This change will take effect after a reboot! So I think I have reproduced this bug.
Not sure I understand the versions you're testing, but the rollback issue is covered in bug 1764534 and the grubenv issue is reported in 4.3.6
I think peyu just wanted to reproduce '3. The operations like "nodectl rollback" will not work.' mentioned in c #0, but after upgrade from RHVH-4.3-20190410.3 to RHVH-4.3-20191108.0, /boot/grub2/grubenv was persisted as a symlink, so `imgbase rollback` could work. I tried the following steps: 1. install rhvh-4.3.6.5-0.20191108.0 2. upgrade to rhvh-4.3.8.0-0.20191211.0 3. run `imgbase rollback` on new layer, then reboot /boot/grub2/grubenv was changed to the following content after run `imgbase rollback`: [root@dell-per730-34 ~]# cat /boot/grub2/grubenv # GRUB Environment Block saved_entry=rhvh-4.3.6.5-0.20191108.0 (3.10.0-1062.4.2.el7.x86_64) but after reboot, the default boot entry was still rhvh-4.3.8.0-0.20191211.0, the system couldn't enter the old layer by itself. I think this is what '"3. The operations like "nodectl rollback" will not work.' means. If /boot/grub2/grubenv is a real file, any changes to it will not take effect on UEFI machine.
Waiting for redhat-virtualization-host-4.3.8-20191219.0.el7_7 iso file
This bug seems to have been resolved in "redhat-virtualization-host-4.3.8-20191219.0.el7_7" Test steps: 1. Install RHVH-4.3-20191108.0-RHVH-x86_64-dvd1.iso and reboot 2. Login to RHVH and check if the "/boot/grub2/grubenv" is symlink to "/boot/efi/EFI/redhat/grubenv" # ls -l /boot/grub2/ | grep grubenv 3. Set local repo and point to "redhat-virtualization-host-image-update-4.3.8-20191219.0.el7_7.noarch.rpm" 4. Upgrade RHVH # yum update 5. Reboot and login to RHVH new layer 6. Check if the "/boot/grub2/grubenv" is symlink to "/boot/efi/EFI/redhat/grubenv" # ls -l /boot/grub2/ | grep grubenv 7. Check image rollback # nodectl rollback # ls -l /boot/grub2/ | grep grubenv Test results: 1. The result of step 2 is as follows: # ls -l /boot/grub2/ | grep grubenv -rw-r--r--. 1 root root 1024 Dec 25 06:17 grubenv lrwxrwxrwx. 1 root root 25 Nov 8 04:28 grubenv.rpmnew -> ../efi/EFI/redhat/grubenv 2. Result of step 6: # ls -l /boot/grub2/ | grep grubenv lrwxrwxrwx. 1 root root 25 Dec 25 06:28 grubenv -> ../efi/EFI/redhat/grubenv 3. Result of step 7: System roll back to old layer "rhvh-4.3.6.5-0.20191108.0+1" correctly and "/boot/grub2/grubenv" is symlink to "/boot/efi/EFI/redhat/grubenv" [root@hp-dl380g10-05 ~]# ls -l /boot/grub2/ | grep grubenv lrwxrwxrwx. 1 root root 25 Dec 25 06:28 grubenv -> ../efi/EFI/redhat/grubenv -rw-r--r--. 1 root root 1024 Dec 25 06:32 grubenv.new I will install "rhvh-4.3.8.1-0.20191219.0+1" to check if the "/boot/grub2/grubenv" is symlink to "/boot/efi/EFI/redhat/grubenv, when I get the iso file. Then will move bug status to "VERIFIED".
This bug was resolved in "RHVH-4.3-20200105.0-RHVH-x86_64-dvd1.iso" Test steps: 1. Install "RHVH-4.3-20200105.0-RHVH-x86_64-dvd1.iso" on UEFI server 2. Check if the "/boot/grub2/grubenv" is symlink to "/boot/efi/EFI/redhat/grubenv # ls -l /boot/grub2/ | grep grubenv 3. Check image rollback # nodectl rollback Test results: 1. The result of step 2 is as follows: # ls -l /boot/grub2/ | grep grubenv lrwxrwxrwx. 1 root root 25 Jan 6 03:04 grubenv -> ../efi/EFI/redhat/grubenv 2. The result of step 3 is as follows: # nodectl rollback Status: Failure Reason: It's required to have at least two layers available to execute rollback operation! So I move the bug status to "VERIFIED".
If this bug does not require documentation, please set the "requires_doc_text" flag to "-". Otherwise, please supply doc text.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0503
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days