Bug 2092273

> We have a customer noticing that the /boot/grub2/grubenv file is getting modified continuously, which raises an alert when executing aide monitoring tool (https://aide.github.io/).

Sounds like the monitoring tool needs to be fixed.

> 1. usually the latest kernel fails to boot because the initramfs is broken, which leads to a kernel panic and system sitting there

No, usually the latest kernel works :)  Otherwise it's a bug that needs to be handled.  Worth noting that the UX won't be better without this feature: the system still won't boot, will still panic and sit there, etc..  In both cases the admin will need to reboot the system again, and in both cases they'll get the menu.

> 3. the feature is not enabled by default since its implementation inside grub.cfg relies on a "boot_counter" variable which isn't set

Well, if it weren't a default, it wouldn't be a problem :)  That variable is controlled by grub itself - see 12_menu_auto_hide, 10_reset_boot_success, grub-set-bootflag.

> 5. it's unclear when a RHEL system should be considered as "booted fine": many systems are acting as services only (e.g. database server) with no user except root connecting to the system

We're not interested in whether user space works perfectly.  We're interested in whether we can *get* to user-space.  Boot success is defined as successfully running a session as a non-root user for > 2 minutes (or manually rebooted from gdm) - otherwise it's indeterminate, and we'll show the screen.

> 4. there may be a race if 2 users log in concurrently: after 2 minutes, both grub-boot-success.service units will execute concurrently, which may break the grubenv file content if no proper locking is performed (I didn't check the code to confirm/infirm this)

This race should be harmless - they'll both record a success state.

(In reply to Robbie Harwood from comment #1)
> > We have a customer noticing that the /boot/grub2/grubenv file is getting modified continuously, which raises an alert when executing aide monitoring tool (https://aide.github.io/).
> 
> Sounds like the monitoring tool needs to be fixed.

Nope, it's Grub that needs to be fixed: the /boot/grub2/grubenv file should be modified only in rare occasions (through kernel update or administration commands).

> 
> > 1. usually the latest kernel fails to boot because the initramfs is broken, which leads to a kernel panic and system sitting there
> 
> No, usually the latest kernel works :)  Otherwise it's a bug that needs to
> be handled.  Worth noting that the UX won't be better without this feature:
> the system still won't boot, will still panic and sit there, etc..  In both
> cases the admin will need to reboot the system again, and in both cases
> they'll get the menu.
> 
> > 3. the feature is not enabled by default since its implementation inside grub.cfg relies on a "boot_counter" variable which isn't set
> 
> Well, if it weren't a default, it wouldn't be a problem :)  That variable is
> controlled by grub itself - see 12_menu_auto_hide, 10_reset_boot_success,
> grub-set-bootflag.

I mean setting "boot_success" has no impact by default: with or without it with RHEL /dev/default/grub settings, nothing special happens while in the Grub menu:
having grub_success just sets "menu_hide_ok" which isn't taking action when we have a serial console (usually in use with servers) or we don't have GRUB_HIDDEN_TIMEOUT_* variables.

> 
> > 5. it's unclear when a RHEL system should be considered as "booted fine": many systems are acting as services only (e.g. database server) with no user except root connecting to the system
> 
> We're not interested in whether user space works perfectly.  We're
> interested in whether we can *get* to user-space.  Boot success is defined
> as successfully running a session as a non-root user for > 2 minutes (or
> manually rebooted from gdm) - otherwise it's indeterminate, and we'll show
> the screen.
> 

OK then it's not correct to rely on a user session to do this, this should be done by a "system service" pulled by multi-user.target, this service may be triggered by a timer.
Note that on server systems, having "systemd --user" instance is optional and we recommend our users to disable this service.

> > 4. there may be a race if 2 users log in concurrently: after 2 minutes, both grub-boot-success.service units will execute concurrently, which may break the grubenv file content if no proper locking is performed (I didn't check the code to confirm/infirm this)
> 
> This race should be harmless - they'll both record a success state.

At least the tool should check if flag is not already present and modify the file only if that's not the case.

Also it's not acceptable on servers to have grub2-set-bootflag be able to modify the timeout for the menu while logged in as a normal user, only administrators should be able to do this.

Hi Suresh,

It is working as expected.Reason - As Robbie said in comment#1 ,the grubenv file is used to store boot success and other grub information. grub-boot-success.timer will run two minutes after non-root user logins. This timer calls grub-boot-success.service which sets the boot success flag within the environment block.

Also possible that with monitoring tools like AIDE ,it is likely that the modification time of the "/boot/grub2/grubenv" file is being recorded even if none made any changes to it.May be the monitoring tool is designed to track changes to all files on the system, regardless of whether they have been modified by the user or by other processes?

Switching component to aide based on comment#1

IMHO the issue is not with "aide" but Grub2, which, as I already wrote previously, updates CONSTANTLY the file while it should be smarter and update only once ("check boot success flag is set? return, otherwise set it").

Hi Renaud,
yeah, I know. :) Since Robbie disagrees with you, perhaps the way forward is to engage with aide devels.

I'm switching this back to Grub, because the issue is clearly with the implementation of the "Boot Success" functionality.

The tool HAS to be smart, and do the update only when necessary, hence modify grubenv only if there is a change.

aide as such is not culprit and cannot be fixed, it depends on the aide configuration created by the user.
To me, it's completely legit to monitor changed in /boot/grub2/grubenv because this indicates a potential tampering.

I would like a decision to be taken about this bug, it's open for too long time.

Renaud.

Created attachment 1972823 [details]
patch to modify grubenv just once per bootflag

If it is enough to modify grubenv only once per bootflag when an user is logged in, see attached patch.

Just want to comment this:
> 4. there may be a race if 2 users log in concurrently: after 2 minutes, both
> grub-boot-success.service units will execute concurrently, which may break
> the grubenv file content if no proper locking is performed (I didn't check
> the code to confirm/infirm this)

According my understanding of code, it is possible that the grubenv will be changed twice in very short time period, but won't be corrupted. Every instance of service creates its own tempfile and then use rename syscall and it is atomic on Linux.

Created attachment 1972824 [details]
patch to modify grubenv just once per bootflag

wrong patch adding correct one