> We have a customer noticing that the /boot/grub2/grubenv file is getting modified continuously, which raises an alert when executing aide monitoring tool (https://aide.github.io/).

Sounds like the monitoring tool needs to be fixed.

> 1. usually the latest kernel fails to boot because the initramfs is broken, which leads to a kernel panic and system sitting there

No, usually the latest kernel works :)  Otherwise it's a bug that needs to be handled.  Worth noting that the UX won't be better without this feature: the system still won't boot, will still panic and sit there, etc..  In both cases the admin will need to reboot the system again, and in both cases they'll get the menu.

> 3. the feature is not enabled by default since its implementation inside grub.cfg relies on a "boot_counter" variable which isn't set

Well, if it weren't a default, it wouldn't be a problem :)  That variable is controlled by grub itself - see 12_menu_auto_hide, 10_reset_boot_success, grub-set-bootflag.

> 5. it's unclear when a RHEL system should be considered as "booted fine": many systems are acting as services only (e.g. database server) with no user except root connecting to the system

We're not interested in whether user space works perfectly.  We're interested in whether we can *get* to user-space.  Boot success is defined as successfully running a session as a non-root user for > 2 minutes (or manually rebooted from gdm) - otherwise it's indeterminate, and we'll show the screen.

> 4. there may be a race if 2 users log in concurrently: after 2 minutes, both grub-boot-success.service units will execute concurrently, which may break the grubenv file content if no proper locking is performed (I didn't check the code to confirm/infirm this)

This race should be harmless - they'll both record a success state.

(In reply to Robbie Harwood from comment #1)
> > We have a customer noticing that the /boot/grub2/grubenv file is getting modified continuously, which raises an alert when executing aide monitoring tool (https://aide.github.io/).
> 
> Sounds like the monitoring tool needs to be fixed.

Nope, it's Grub that needs to be fixed: the /boot/grub2/grubenv file should be modified only in rare occasions (through kernel update or administration commands).

> 
> > 1. usually the latest kernel fails to boot because the initramfs is broken, which leads to a kernel panic and system sitting there
> 
> No, usually the latest kernel works :)  Otherwise it's a bug that needs to
> be handled.  Worth noting that the UX won't be better without this feature:
> the system still won't boot, will still panic and sit there, etc..  In both
> cases the admin will need to reboot the system again, and in both cases
> they'll get the menu.
> 
> > 3. the feature is not enabled by default since its implementation inside grub.cfg relies on a "boot_counter" variable which isn't set
> 
> Well, if it weren't a default, it wouldn't be a problem :)  That variable is
> controlled by grub itself - see 12_menu_auto_hide, 10_reset_boot_success,
> grub-set-bootflag.

I mean setting "boot_success" has no impact by default: with or without it with RHEL /dev/default/grub settings, nothing special happens while in the Grub menu:
having grub_success just sets "menu_hide_ok" which isn't taking action when we have a serial console (usually in use with servers) or we don't have GRUB_HIDDEN_TIMEOUT_* variables.

> 
> > 5. it's unclear when a RHEL system should be considered as "booted fine": many systems are acting as services only (e.g. database server) with no user except root connecting to the system
> 
> We're not interested in whether user space works perfectly.  We're
> interested in whether we can *get* to user-space.  Boot success is defined
> as successfully running a session as a non-root user for > 2 minutes (or
> manually rebooted from gdm) - otherwise it's indeterminate, and we'll show
> the screen.
> 

OK then it's not correct to rely on a user session to do this, this should be done by a "system service" pulled by multi-user.target, this service may be triggered by a timer.
Note that on server systems, having "systemd --user" instance is optional and we recommend our users to disable this service.

> > 4. there may be a race if 2 users log in concurrently: after 2 minutes, both grub-boot-success.service units will execute concurrently, which may break the grubenv file content if no proper locking is performed (I didn't check the code to confirm/infirm this)
> 
> This race should be harmless - they'll both record a success state.

At least the tool should check if flag is not already present and modify the file only if that's not the case.

Also it's not acceptable on servers to have grub2-set-bootflag be able to modify the timeout for the menu while logged in as a normal user, only administrators should be able to do this.

Hi Suresh,

It is working as expected.Reason - As Robbie said in comment#1 ,the grubenv file is used to store boot success and other grub information. grub-boot-success.timer will run two minutes after non-root user logins. This timer calls grub-boot-success.service which sets the boot success flag within the environment block.

Also possible that with monitoring tools like AIDE ,it is likely that the modification time of the "/boot/grub2/grubenv" file is being recorded even if none made any changes to it.May be the monitoring tool is designed to track changes to all files on the system, regardless of whether they have been modified by the user or by other processes?

Switching component to aide based on comment#1

IMHO the issue is not with "aide" but Grub2, which, as I already wrote previously, updates CONSTANTLY the file while it should be smarter and update only once ("check boot success flag is set? return, otherwise set it").

Hi Renaud,
yeah, I know. :) Since Robbie disagrees with you, perhaps the way forward is to engage with aide devels.

I'm switching this back to Grub, because the issue is clearly with the implementation of the "Boot Success" functionality.

The tool HAS to be smart, and do the update only when necessary, hence modify grubenv only if there is a change.

aide as such is not culprit and cannot be fixed, it depends on the aide configuration created by the user.
To me, it's completely legit to monitor changed in /boot/grub2/grubenv because this indicates a potential tampering.

I would like a decision to be taken about this bug, it's open for too long time.

Renaud.

Created attachment 1972823 [details]
patch to modify grubenv just once per bootflag

If it is enough to modify grubenv only once per bootflag when an user is logged in, see attached patch.

Just want to comment this:
> 4. there may be a race if 2 users log in concurrently: after 2 minutes, both
> grub-boot-success.service units will execute concurrently, which may break
> the grubenv file content if no proper locking is performed (I didn't check
> the code to confirm/infirm this)

According my understanding of code, it is possible that the grubenv will be changed twice in very short time period, but won't be corrupted. Every instance of service creates its own tempfile and then use rename syscall and it is atomic on Linux.

Created attachment 1972824 [details]
patch to modify grubenv just once per bootflag

wrong patch adding correct one

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.