Bug 1809367 - Host update fails with environment block too small
Summary: Host update fails with environment block too small
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: imgbased
Version: 4.3.7
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ovirt-4.4.0
: 4.4.0
Assignee: Yuval Turgeman
QA Contact: peyu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-02 23:27 UTC by Siddhant Rao
Modified: 2020-08-04 16:22 UTC (History)
12 users (show)

Fixed In Version: imgbased-1.2.9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-04 16:22:08 UTC
oVirt Team: Node
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
imgbase.log (1.33 MB, text/plain)
2020-03-02 23:53 UTC, Siddhant Rao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 4927961 0 None None None 2020-03-25 12:21:24 UTC
Red Hat Product Errata RHEA-2020:3316 0 None None None 2020-08-04 16:22:28 UTC
oVirt gerrit 107823 0 master MERGED utils: verify grubenv size 2021-01-05 02:45:45 UTC

Internal Links: 2052835

Description Siddhant Rao 2020-03-02 23:27:41 UTC
Description of problem:

Updating RHV Hosts can fail with the grub error of "grub2-editenv: error: environment block too small."

~~~
2020-02-26 00:44:56,359 [DEBUG] (MainThread) Calling binary: (['grub2-editenv', '/boot/grub2/grubenv', 'set', u'saved_entry=rhvh-4.3.8.1-0.20200126.0 (3.10.0-1062.12.1.el7.x86_64)'],) {}
2020-02-26 00:44:56,359 [DEBUG] (MainThread) Calling: (['grub2-editenv', '/boot/grub2/grubenv', 'set', u'saved_entry=rhvh-4.3.8.1-0.20200126.0 (3.10.0-1062.12.1.el7.x86_64)'],) {'close_fds': True, 'stderr': -2}
2020-02-26 00:44:56,376 [DEBUG] (MainThread) Exception! grub2-editenv: error: environment block too small.


2020-02-26 00:44:56,377 [DEBUG] (MainThread) Calling binary: (['umount', '-R', '-l', u'/tmp/mnt.POHjk/var'],) {}
2020-02-26 00:44:56,377 [DEBUG] (MainThread) Calling: (['umount', '-R', '-l', u'/tmp/mnt.POHjk/var'],) {'close_fds': True, 'stderr': -2}
2020-02-26 00:44:56,420 [DEBUG] (MainThread) Returned:
2020-02-26 00:44:56,421 [DEBUG] (MainThread) Calling binary: (['umount', '-l', u'/tmp/mnt.POHjk'],) {}
2020-02-26 00:44:56,422 [DEBUG] (MainThread) Calling: (['umount', '-l', u'/tmp/mnt.POHjk'],) {'close_fds': True, 'stderr': -2}
2020-02-26 00:44:56,803 [DEBUG] (MainThread) Returned:
2020-02-26 00:44:56,804 [DEBUG] (MainThread) Calling binary: (['rmdir', u'/tmp/mnt.POHjk'],) {}
2020-02-26 00:44:56,804 [DEBUG] (MainThread) Calling: (['rmdir', u'/tmp/mnt.POHjk'],) {'close_fds': True, 'stderr': -2}
2020-02-26 00:44:56,817 [DEBUG] (MainThread) Returned:
2020-02-26 00:44:56,818 [ERROR] (MainThread) Failed to update OS
Traceback (most recent call last):
  File "/tmp/tmp.yKlneATYGc/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 260, in migrate_boot
    adjust_mounts_and_boot(imgbase, new_lv, previous_layer_lv)
  File "/tmp/tmp.yKlneATYGc/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 1188, in adjust_mounts_and_boot
    add_bootentry(newroot.target)
  File "/tmp/tmp.yKlneATYGc/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 1149, in add_bootentry
    loader.set_default(new_lv.lv_name)
  File "/tmp/tmp.yKlneATYGc/usr/lib/python2.7/site-packages/imgbased/bootloader.py", line 273, in set_default
    self._update_grubenv(entry)
  File "/tmp/tmp.yKlneATYGc/usr/lib/python2.7/site-packages/imgbased/bootloader.py", line 235, in _update_grubenv
    grub2_editenv("set", "saved_entry=%s" % entry.title)
  File "/tmp/tmp.yKlneATYGc/usr/lib/python2.7/site-packages/imgbased/utils.py", line 127, in grub2_editenv
    ExternalBinary().grub2_editenv([grubenv] + list(args), **kwargs)
  File "/tmp/tmp.yKlneATYGc/usr/lib/python2.7/site-packages/imgbased/utils.py", line 483, in grub2_editenv
    return self.call(["grub2-editenv"] + args, **kwargs)
  File "/tmp/tmp.yKlneATYGc/usr/lib/python2.7/site-packages/imgbased/utils.py", line 422, in call
    stdout = call(*args, **kwargs)
  File "/tmp/tmp.yKlneATYGc/usr/lib/python2.7/site-packages/imgbased/utils.py", line 186, in call
    return subprocess.check_output(*args, **kwargs).strip()
  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['grub2-editenv', '/boot/grub2/grubenv', 'set', u'saved_entry=rhvh-4.3.8.1-0.20200126.0 (3.10.0-1062.12.1.el7.x86_64)']' returned non-zero exit status 1
~~~


Checked https://bugzilla.redhat.com/show_bug.cgi?id=1625124#c17 and found out that if grubenv has a size lesser than 1024 bytes, we can face this error.

Simply recreated the grubenv and added the present layer to it using grub2-editenv,

~~~
# grub2-editenv create
# grub2-editenv /boot/grub2/grubenv set saved_entry='rhvh-4.3.7.1-0.20191211.0 (3.10.0-1062.9.1.el7.x86_64)' -vvv
~~~

Then update works fine.



Version-Release number of selected component (if applicable):
rhvh-4.3.7.1-0.20191211.0
imgbased-1.1.13-0.1.el7ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. Modify grubenv such that the size of the file is less than 1024
2. Run yum update

Actual results:
Host update fails.

Expected results:
Host update should not fail.

Additional info:

Comment 1 Siddhant Rao 2020-03-02 23:53:39 UTC
Created attachment 1667092 [details]
imgbase.log

Comment 4 Yuval Turgeman 2020-03-03 08:52:24 UTC
but grubenv should be exactly 1024 bytes, and never edited manually -

$ grub2-editenv /tmp/asdf create && ll /tmp/asdf
-rw-rw-r--. 1 yturgema yturgema 1024 Mar  3 10:51 /tmp/asdf

Comment 5 Sandro Bonazzola 2020-03-04 08:26:10 UTC
We'll add a check on the size of the file and eventually re-create if too small.
Looks like this issue is not specific to RHV-H, it has been reported to fedora too (bug #1625124 )

Comment 15 errata-xmlrpc 2020-08-04 16:22:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHV Host (redhat-virtualization-host) 4.4), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:3316


Note You need to log in before you can comment on or make changes to this bug.