Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1994255

Summary: After do tuned-adm profile and grubby update kernel options, the /boot/grub2/grubenv file will be empty
Product: Red Hat Enterprise Linux 8 Reporter: mhou <mhou>
Component: grub2Assignee: Javier Martinez Canillas <fmartine>
Status: CLOSED DUPLICATE QA Contact: Release Test Team <release-test-team-automation>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.2CC: fmartine, jskarvad, kzhang, mhou
Target Milestone: betaKeywords: TestBlocker, ZStream
Target Release: ---Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-25 11:36:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description mhou 2021-08-17 07:17:44 UTC
Description of problem:
The first step do the tuned-adm profile $profiles
The second step uses grubby to update kernel options
The third step uses tuned-adm profile $profiles again.

Version-Release number of selected component (if applicable):
kernel version: kernel-rt-4.18.0-193.63.1.rt13.114.el8_2.x86_64
grubby-8.40-38.el8.x86_64 or grubby-8.40-38.el8_2.1.x86_64
tuned-2.13.0-6.el8.noarch

How reproducible:100%


Steps to Reproduce:
#! /bin/bash
set -x
echo "isolated_cores=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46" > /etc/tuned/realtime-virtual-host-variables.conf
echo "isolate_managed_irq=Y" >> /etc/tuned/realtime-virtual-host-variables.conf
tuned-adm profile realtime-virtual-host
echo "after do first tuned and check /boot/grub2"
ls -l /boot/grub2
grubby '--args=default_hugepagesz=1G hugepagesz=1G hugepages=24 isolcpus=24,2,26,4,28,6,30,8,32,10,34,12,36,14,38,16,40,18,42,20,44,22,46 intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable ' --update-kernel=/boot/vmlinuz-4.18.0-193.63.1.rt13.114.el8_2.x86_64
echo "after do grubby check /boot/grub2"
ls -l /boot/grub2
tuned-adm profile realtime-virtual-host
echo "after do second tuned and check /boot/grub2"
ls -l /boot/grub2


Actual results:
# sh --debug test.sh
+ echo isolated_cores=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46
+ echo isolate_managed_irq=Y
+ tuned-adm profile realtime-virtual-host
+ echo 'after do first tuned and check /boot/grub2'
after do first tuned and check /boot/grub2
+ ls -l /boot/grub2
total 28
-rw-r--r--. 1 root root   64 Aug 16 23:59 device.map
drwxr-xr-x. 2 root root   25 Aug 16 23:59 fonts
-rw-r--r--. 1 root root 4613 Aug 17 02:52 grub.cfg
-rw-r--r--. 1 root root 1024 Aug 17 02:52 grubenv
drwxr-xr-x. 2 root root 8192 Aug 16 23:59 i386-pc
+ grubby '--args=default_hugepagesz=1G hugepagesz=1G hugepages=24 isolcpus=24,2,26,4,28,6,30,8,32,10,34,12,36,14,38,16,40,18,42,20,44,22,46 intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable ' --update-kernel=/boot/vmlinuz-4.18.0-193.63.1.rt13.114.el8_2.x86_64
+ echo 'after do grubby check /boot/grub2'
after do grubby check /boot/grub2
+ ls -l /boot/grub2
total 28
-rw-r--r--. 1 root root   64 Aug 16 23:59 device.map
drwxr-xr-x. 2 root root   25 Aug 16 23:59 fonts
-rw-r--r--. 1 root root 4613 Aug 17 02:52 grub.cfg
-rw-r--r--. 1 root root 1024 Aug 17 02:52 grubenv
drwxr-xr-x. 2 root root 8192 Aug 16 23:59 i386-pc
+ tuned-adm profile realtime-virtual-host
+ echo 'after do second tuned and check /boot/grub2'
after do second tuned and check /boot/grub2
+ ls -l /boot/grub2
total 24
-rw-r--r--. 1 root root   64 Aug 16 23:59 device.map
drwxr-xr-x. 2 root root   25 Aug 16 23:59 fonts
-rw-r--r--. 1 root root 4613 Aug 17 02:53 grub.cfg
-rw-r--r--. 1 root root    0 Aug 17 02:53 grubenv
drwxr-xr-x. 2 root root 8192 Aug 16 23:59 i386-pc

If this time reboot system, the system will going to dracut.

Expected results:
any correct kernel option and tuned profile update won't damage /boot/grub2/grubenv file

Additional info:
This issue won't appear on rhel8.4 and rhel8.5 rt-kernel.

Comment 1 mhou 2021-08-17 07:25:37 UTC
I use kernel-rt-4.18.0-329.rt7.110.el8.x86_64(RHEL8.5) to test it again. It works as normal.

Version-Release number of selected component (if applicable):
grubby-8.40-41.el8.x86_64
tuned-2.16.0-1.el8.noarch

# cat test.sh 
#! /bin/bash
echo "isolated_cores=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30" > /etc/tuned/realtime-virtual-host-variables.conf
echo "isolate_managed_irq=Y" >> /etc/tuned/realtime-virtual-host-variables.conf
tuned-adm profile realtime-virtual-host
echo "after do first tuned and check /boot/grub2"
ls -l /boot/grub2
grubby '--args=default_hugepagesz=1G hugepagesz=1G hugepages=24 isolcpus=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30 intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable ' --update-kernel=$(grubby --default-kernel)
echo "after do grubby check /boot/grub2"
ls -l /boot/grub2
tuned-adm profile realtime-virtual-host
echo "after do second tuned and check /boot/grub2"
ls -l /boot/grub2


[root@dell-per740-07 ~]# sh --debug test.sh 
after do first tuned and check /boot/grub2
total 28
-rw-r--r-- 1 root root   64 Aug 11 03:52 device.map
drwxr-xr-x 2 root root   25 Aug 11 03:52 fonts
-rw-r--r-- 1 root root 6499 Aug 17 03:21 grub.cfg
-rw-r--r-- 1 root root 1024 Aug 17 03:21 grubenv
drwxr-xr-x 2 root root 8192 Aug 11 03:52 i386-pc
after do grubby check /boot/grub2
total 28
-rw-r--r-- 1 root root   64 Aug 11 03:52 device.map
drwxr-xr-x 2 root root   25 Aug 11 03:52 fonts
-rw-r--r-- 1 root root 6499 Aug 17 03:21 grub.cfg
-rw-r--r-- 1 root root 1024 Aug 17 03:21 grubenv
drwxr-xr-x 2 root root 8192 Aug 11 03:52 i386-pc
after do second tuned and check /boot/grub2
total 28
-rw-r--r-- 1 root root   64 Aug 11 03:52 device.map
drwxr-xr-x 2 root root   25 Aug 11 03:52 fonts
-rw-r--r-- 1 root root 6499 Aug 17 03:22 grub.cfg
-rw-r--r-- 1 root root 1024 Aug 17 03:22 grubenv
drwxr-xr-x 2 root root 8192 Aug 11 03:52 i386-pc

Comment 2 mhou 2021-08-17 08:08:37 UTC
This issue can occur in another way:
[root@dell-per740-18 ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.63.1.rt13.114.el8_2.x86_64 root=/dev/mapper/rhel_dell--per740--18-root ro ksdevice=bootif crashkernel=auto resume=/dev/mapper/rhel_dell--per740--18-swap rd.lvm.lv=rhel_dell-per740-18/root rd.lvm.lv=rhel_dell-per740-18/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=managed_irq,domain,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 default_hugepagesz=1G hugepagesz=1G hugepages=24 isolcpus=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable
[root@dell-per740-18 ~]# cat /etc/tuned/realtime-virtual-host-variables.conf 
isolated_cores=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46
isolate_managed_irq=Y
[root@dell-per740-18 ~]# tuned-adm profile realtime-virtual-host
[root@dell-per740-18 ~]# grubby '--args=default_hugepagesz=1G hugepagesz=1G hugepages=24 isolcpus=24,2,26,4,28,6,30,8,32,10,34,12,36,14,38,16,40,18,42,20,44,22,46 intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable ' --update-kernel=/boot/vmlinuz-4.18.0-193.63.1.rt13.114.el8_2.x86_64
[root@dell-per740-18 ~]# ll /boot/grub2/
total 28
-rw-r--r--. 1 root root   64 Aug 16 22:38 device.map
drwxr-xr-x. 2 root root   25 Aug 16 22:38 fonts
-rw-r--r--. 1 root root 4612 Aug 17 03:52 grub.cfg
-rw-r--r--. 1 root root 1024 Aug 17 03:52 grubenv
drwxr-xr-x. 2 root root 8192 Aug 16 22:38 i386-pc
[root@dell-per740-18 ~]# reboot

After system comeback, and run tuned-adm profile again.
[root@dell-per740-18 ~]# tuned-adm profile realtime-virtual-host
[root@dell-per740-18 ~]# ll /boot/grub2/
total 24
-rw-r--r--. 1 root root   64 Aug 16 22:38 device.map
drwxr-xr-x. 2 root root   25 Aug 16 22:38 fonts
-rw-r--r--. 1 root root 4612 Aug 17 03:59 grub.cfg
-rw-r--r--. 1 root root    0 Aug 17 03:59 grubenv
drwxr-xr-x. 2 root root 8192 Aug 16 22:38 i386-pc

I can see tune report some error messages:
2021-08-17 03:59:35,559 INFO     tuned.profiles.loader: loading profile: realtime-virtual-host
2021-08-17 03:59:35,574 INFO     tuned.daemon.daemon: starting tuning
2021-08-17 03:59:35,627 INFO     tuned.plugins.base: instance cpu: assigning devices cpu24, cpu45, cpu34, cpu26, cpu2, cpu33, cpu9, cpu13, cpu29, cpu16, cpu0, cpu43, cpu20, cpu15, cpu47, cpu11, cpu18, cpu28, cpu41, cpu27, cpu4, cpu42, cpu7, cpu14, cpu23, cpu31, cpu38, cpu46, cpu21, cpu17, cpu5, cpu35, cpu32, cpu8, cpu37, cpu3, cpu12, cpu1, cpu25, cpu36, cpu30, cpu40, cpu39, cpu22, cpu44, cpu19, cpu6, cpu10
2021-08-17 03:59:35,649 INFO     tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform
2021-08-17 03:59:35,801 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu24'
2021-08-17 03:59:35,865 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu45'
2021-08-17 03:59:35,942 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu34'
2021-08-17 03:59:36,004 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu26'
2021-08-17 03:59:36,070 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu2'
2021-08-17 03:59:36,134 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu33'
2021-08-17 03:59:36,213 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu9'
2021-08-17 03:59:36,272 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu13'
2021-08-17 03:59:36,341 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu29'
2021-08-17 03:59:36,407 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu16'
2021-08-17 03:59:36,483 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu0'
2021-08-17 03:59:36,546 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu43'
2021-08-17 03:59:36,608 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu20'
2021-08-17 03:59:36,682 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu15'
2021-08-17 03:59:36,771 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu47'
2021-08-17 03:59:36,859 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu11'
2021-08-17 03:59:36,932 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu18'
2021-08-17 03:59:37,000 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu28'
2021-08-17 03:59:37,070 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu41'
2021-08-17 03:59:37,134 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu27'
2021-08-17 03:59:37,202 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu4'
2021-08-17 03:59:37,278 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu42'
2021-08-17 03:59:37,343 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu7'
2021-08-17 03:59:37,395 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu14'
2021-08-17 03:59:37,463 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu23'
2021-08-17 03:59:37,536 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu31'
2021-08-17 03:59:37,610 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu38'
2021-08-17 03:59:37,675 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu46'
2021-08-17 03:59:37,752 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu21'
2021-08-17 03:59:37,830 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu17'
2021-08-17 03:59:37,908 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu5'
2021-08-17 03:59:37,977 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu35'
2021-08-17 03:59:38,071 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu32'
2021-08-17 03:59:38,169 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu8'
2021-08-17 03:59:38,236 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu37'
2021-08-17 03:59:38,302 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu3'
2021-08-17 03:59:38,392 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu12'
2021-08-17 03:59:38,461 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu1'
2021-08-17 03:59:38,546 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu25'
2021-08-17 03:59:38,640 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu36'
2021-08-17 03:59:38,729 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu30'
2021-08-17 03:59:38,822 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu40'
2021-08-17 03:59:38,884 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu39'
2021-08-17 03:59:38,961 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu22'
2021-08-17 03:59:39,037 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu44'
2021-08-17 03:59:39,108 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu19'
2021-08-17 03:59:39,187 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu6'
2021-08-17 03:59:39,254 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu10'
2021-08-17 03:59:39,255 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 2
2021-08-17 03:59:39,258 ERROR    tuned.plugins.plugin_sysctl: Failed to read sysctl parameter 'net.core.busy_read', the parameter does not exist
2021-08-17 03:59:39,258 ERROR    tuned.plugins.plugin_sysctl: sysctl option net.core.busy_read will not be set, failed to read the original value.
2021-08-17 03:59:39,258 ERROR    tuned.plugins.plugin_sysctl: Failed to read sysctl parameter 'net.core.busy_poll', the parameter does not exist
2021-08-17 03:59:39,259 ERROR    tuned.plugins.plugin_sysctl: sysctl option net.core.busy_poll will not be set, failed to read the original value.
2021-08-17 03:59:39,259 ERROR    tuned.plugins.plugin_sysctl: Failed to read sysctl parameter 'kernel.numa_balancing', the parameter does not exist
2021-08-17 03:59:39,259 ERROR    tuned.plugins.plugin_sysctl: sysctl option kernel.numa_balancing will not be set, failed to read the original value.
2021-08-17 03:59:39,269 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2021-08-17 03:59:39,271 WARNING  tuned.plugins.plugin_vm: Option 'transparent_hugepages' is not supported on current hardware.
2021-08-17 03:59:39,272 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
2021-08-17 03:59:39,307 ERROR    tuned.utils.commands: Executing grub2-editenv error: grub2-editenv: error: invalid environment block.
2021-08-17 03:59:39,308 WARNING  tuned.plugins.plugin_bootloader: cannot update grubenv: ''
2021-08-17 03:59:39,331 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['start']'
2021-08-17 03:59:39,477 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['start']'
2021-08-17 03:59:41,660 ERROR    tuned.plugins.plugin_script: script '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Unit ksm.service does not exist, proceeding anyway.
Unit ksmtuned.service does not exist, proceeding anyway.
chrt: failed to execute /usr/libexec/qemu-kvm: No such file or directory
Ncat: Connection refused.
Ncat: Connection refused.
running /usr/share/tuned-profiles-nfv-host-bin/tscdeadline_latency.flat failed'
2021-08-17 03:59:41,661 ERROR    tuned.plugins.plugin_script: script '/usr/lib/tuned/realtime-virtual-host/script.sh' returned error code: 1
2021-08-17 03:59:42,453 INFO     tuned.daemon.daemon: static tuning from profile 'realtime-virtual-host' applied

Comment 3 mhou 2021-08-18 02:18:43 UTC
This issue can be triggered in another form. Here is the other way.


[root@dell-per740-77 ~]# grub2-editenv - list | grep kernelopts | sed -e 's/kernelopts=//g'
root=/dev/mapper/rhel_dell--per740--77-root ro crashkernel=auto resume=/dev/mapper/rhel_dell--per740--77-swap rd.lvm.lv=rhel_dell-per740-77/root rd.lvm.lv=rhel_dell-per740-77/swap console=ttyS0,115200n81 
[root@dell-per740-77 ~]# grub2-editenv - set kernelopts="root=/dev/mapper/rhel_dell--per740--77-root ro crashkernel=auto resume=/dev/mapper/rhel_dell--per740--77-swap rd.lvm.lv=rhel_dell-per740-77/root rd.lvm.lv=rhel_dell-per740-77/swap console=ttyS0,115200n81 intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable default_hugepagesz=1G hugepagesz=1G hugepages=24 isolcpus=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46"
[root@dell-per740-77 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg 
[root@dell-per740-77 ~]# cat /etc/tuned/realtime-virtual-host-variables.conf
# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
#
isolated_cores=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46
isolate_managed_irq=Y
[root@dell-per740-77 ~]# tuned-adm profile realtime-virtual-host
[root@dell-per740-77 ~]# ll /boot/grub2/
total 28
-rw-r--r--. 1 root root   64 Aug 17 05:38 device.map
drwxr-xr-x. 2 root root   25 Aug 17 05:38 fonts
-rw-r--r--. 1 root root 4582 Aug 17 22:01 grub.cfg
-rw-r--r--. 1 root root 1024 Aug 17 22:01 grubenv
drwxr-xr-x. 2 root root 8192 Aug 17 05:38 i386-pc
[root@dell-per740-77 ~]# reboot

After the system come back check the grubenv file on /boot/grub2 path and the /proc/cmdline

[root@dell-per740-77 ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.63.1.rt13.114.el8_2.x86_64 root=/dev/mapper/rhel_dell--per740--77-root ro crashkernel=auto resume=/dev/mapper/rhel_dell--per740--77-swap rd.lvm.lv=rhel_dell-per740-77/root rd.lvm.lv=rhel_dell-per740-77/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=managed_irq,domain,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46
[root@dell-per740-77 ~]# ll /boot/grub2/
total 28
-rw-r--r--. 1 root root   64 Aug 17 05:38 device.map
drwxr-xr-x. 2 root root   25 Aug 17 05:38 fonts
-rw-r--r--. 1 root root 4582 Aug 17 22:04 grub.cfg
-rw-r--r--. 1 root root 1024 Aug 17 22:04 grubenv
drwxr-xr-x. 2 root root 8192 Aug 17 05:38 i386-pc

run "tuned-adm profile realtime-virtual-host" again, the grubenv file will empty
[root@dell-per740-77 ~]# ll /boot/grub2/
total 24
-rw-r--r--. 1 root root   64 Aug 17 05:38 device.map
drwxr-xr-x. 2 root root   25 Aug 17 05:38 fonts
-rw-r--r--. 1 root root 4582 Aug 17 22:08 grub.cfg
-rw-r--r--. 1 root root    0 Aug 17 22:08 grubenv
drwxr-xr-x. 2 root root 8192 Aug 17 05:38 i386-pc

Comment 7 Jaroslav Škarvada 2021-08-18 21:40:00 UTC
This is definitely not tuned-profiles-nfv-host-bin bug. Maybe it's tuned bug, but more probably it is grub2 bug.

The tuned-2.13.0 directly calls grub2-editenv and doesn't touch the grubenv otherwise. Thus, it seems it's the grub2-editenv which fails for some reason and erases the grubenv content. Unfortunately I am unable to reproduce the problem, the reproducer doesn't work for me.

Could you please run the following commands instead of the last tuned-adm in your reproducer and provide the tuned.log:
# systemctl stop tuned
# tuned-adm profile realtime-virtual-host
# tuned -D 2>&1 | tee tuned.log

Also am I right that this problem is reproducible only on the RHEL-8.2/RHEL-8.3 and it's not reproducible on the RHEL-8.4/8.5?

Comment 8 mhou 2021-08-19 02:41:04 UTC
Hello Jaroslav

I have been test on rhel8.4.z/rhel8.5 and rhel8.2.z RT kernel. Only found rhel8.2.z can be reproduced this issue. 

I follow you step, here is my test step as below. More tuned log please check attachment.
# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              48
On-line CPU(s) list: 0-47
Thread(s) per core:  2
Core(s) per socket:  12
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz
Stepping:            7
CPU MHz:             1200.062
BogoMIPS:            5400.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            19712K
NUMA node0 CPU(s):   0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46
NUMA node1 CPU(s):   1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47
]# ll /boot/grub2/
total 28
-rw-r--r--. 1 root root   64 Aug 18 21:44 device.map
drwxr-xr-x. 2 root root   25 Aug 18 21:44 fonts
-rw-r--r--. 1 root root 4302 Aug 18 21:44 grub.cfg
-rw-r--r--. 1 root root 1024 Aug 18 22:02 grubenv
drwxr-xr-x. 2 root root 8192 Aug 18 21:44 i386-pc
[root@dell-per740-47 ~]# systemctl stop tuned
[root@dell-per740-47 ~]# systemctl status tuned
● tuned.service - Dynamic System Tuning Daemon
   Loaded: loaded (/usr/lib/systemd/system/tuned.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Wed 2021-08-18 22:34:54 EDT; 5s ago
     Docs: man:tuned(8)
           man:tuned.conf(5)
           man:tuned-adm(8)
  Process: 1825 ExecStart=/usr/sbin/tuned -l -P (code=exited, status=0/SUCCESS)
 Main PID: 1825 (code=exited, status=0/SUCCESS)

Aug 19 02:09:51 dell-per740-47.rhts.eng.pek2.redhat.com systemd[1]: Starting Dynamic System Tuning Daemon...
Aug 19 02:09:54 dell-per740-47.rhts.eng.pek2.redhat.com systemd[1]: Started Dynamic System Tuning Daemon.
Aug 18 22:34:54 dell-per740-47.rhts.eng.pek2.redhat.com systemd[1]: Stopping Dynamic System Tuning Daemon...
Aug 18 22:34:54 dell-per740-47.rhts.eng.pek2.redhat.com tuned[1825]: ** COLLECTED WARNINGS **
Aug 18 22:34:54 dell-per740-47.rhts.eng.pek2.redhat.com tuned[1825]: # SMBIOS implementations newer than version 2.7 are not
Aug 18 22:34:54 dell-per740-47.rhts.eng.pek2.redhat.com tuned[1825]: # fully supported by this version of dmidecode.
Aug 18 22:34:54 dell-per740-47.rhts.eng.pek2.redhat.com tuned[1825]: ** END OF WARNINGS **
Aug 18 22:34:54 dell-per740-47.rhts.eng.pek2.redhat.com systemd[1]: Stopped Dynamic System Tuning Daemon.
[root@dell-per740-47 ~]# tuned-adm profile realtime-virtual-host
Cannot talk to Tuned daemon via DBus. Is Tuned daemon running?
Trying to (re)start tuned...
Tuned (re)started, changes applied.
[root@dell-per740-47 ~]# tuned -D 2>&1 | tee tuned.log

Comment 10 mhou 2021-08-19 02:46:37 UTC
(In reply to mhou from comment #8)
After the first time run tuned-adm, I found /boot/grub2/grubenv file is clear. 
[root@dell-per740-47 ~]# vim tuned.log 
[root@dell-per740-47 ~]# ll /boot/grub2/
total 24
-rw-r--r--. 1 root root   64 Aug 18 21:44 device.map
drwxr-xr-x. 2 root root   25 Aug 18 21:44 fonts
-rw-r--r--. 1 root root 4302 Aug 18 22:35 grub.cfg
-rw-r--r--. 1 root root    0 Aug 18 22:35 grubenv
drwxr-xr-x. 2 root root 8192 Aug 18 21:44 i386-pc

Comment 11 mhou 2021-08-19 08:07:41 UTC
Hello Jaroslav

I check the issue occure environment and find corresponding packages version on rhel8.2.
# rpm -qa | grep grub
grub2-pc-modules-2.02-81.el8.noarch
grub2-tools-extra-2.02-81.el8.x86_64
grub2-pc-2.02-81.el8.x86_64
grub2-tools-minimal-2.02-81.el8.x86_64
grub2-common-2.02-81.el8.noarch
grubby-8.40-38.el8.x86_64
grub2-tools-2.02-81.el8.x86_64
[root@dell-per740-77 perf]# rpm -qa | grep tuned
tuned-2.13.0-6.el8.noarch
tuned-profiles-nfv-host-2.13.0-6.el8.noarch
tuned-profiles-nfv-guest-2.13.0-6.el8.noarch
tuned-profiles-nfv-2.13.0-6.el8.noarch
tuned-profiles-nfv-host-bin-0-0.1.20180302git1edfa966.el8.x86_64
tuned-profiles-realtime-2.13.0-6.el8.noarch


Here is work as well corresponding packages version on rhel8.3.
# rpm -qa | grep tuned
tuned-2.14.0-3.el8.noarch
tuned-profiles-nfv-2.14.0-3.el8.noarch
tuned-profiles-realtime-2.14.0-3.el8.noarch
tuned-profiles-nfv-host-bin-0-0.1.20180302git1edfa966.el8.x86_64
tuned-profiles-nfv-guest-2.14.0-3.el8.noarch
tuned-profiles-nfv-host-2.14.0-3.el8.noarch
[root@dell-per740-60 ~]# rpm -qa | grep grub
grub2-common-2.02-90.el8.noarch
grub2-pc-2.02-90.el8.x86_64
grub2-tools-extra-2.02-90.el8.x86_64
grub2-pc-modules-2.02-90.el8.noarch
grub2-tools-minimal-2.02-90.el8.x86_64
grubby-8.40-41.el8.x86_64
grub2-tools-2.02-90.el8.x86_64

Comment 12 Jaroslav Škarvada 2021-08-19 22:15:52 UTC
Hmm, it seems there is nothing suspicious in the log. Could you borrow me the machine?

Comment 13 mhou 2021-08-20 03:24:46 UTC
Hello Jaroslav

dell-per740-47.rhts.eng.pek2.redhat.com alread lend to you. System already install kernel-rt-4.18.0-193.63.1.rt13.114.el8_2.x86_64.

B.R
Minxi Hou

Comment 14 mhou 2021-08-20 09:59:24 UTC
Hello Jaroslav

I change this bug to grub2 compoment. Let bootloader team have a look this issue. This server still lend you until no more value information be found.

B.R
Minxi

Comment 15 Javier Martinez Canillas 2021-08-23 08:03:57 UTC
(In reply to mhou from comment #14)
> Hello Jaroslav
> 
> I change this bug to grub2 compoment. Let bootloader team have a look this
> issue. This server still lend you until no more value information be found.
> 
> B.R
> Minxi

Do I understand correctly that this issue only happens with 8.2.z and no with
other RHEL 8 minor versions ?

Comment 16 mhou 2021-08-23 09:20:35 UTC
yes, this issue only happens on 8.2.z and no with other RHEL8. I test kernel-rt-4.18.0-193.64.1.rt13.115.el8_2 and kernel-rt-4.18.0-193.60.2.rt13.112.el8_2, all of kernel occur this issue.

Comment 17 Javier Martinez Canillas 2021-08-23 11:09:29 UTC
This is with a legacy BIOS system correct? Could you please try to update your GRUB core image ?

For example with the following command: grub2-install /dev/X (replace X for the block device that
contains your GRUB installation).

I believe this is a duplicate of bug #1761496.

Comment 18 mhou 2021-08-23 11:36:54 UTC
I check bug #1761496, and this issue fixes in grub2-2.02-83.el8 on rhel8.3. Actually when I use grub2-2.02-90 can't find this issue. So In the future, rhel8.2.z will contain a fixed grub version?

Comment 19 Javier Martinez Canillas 2021-08-23 12:17:53 UTC
(In reply to mhou from comment #18)
> I check bug #1761496, and this issue fixes in grub2-2.02-83.el8 on rhel8.3.
> Actually when I use grub2-2.02-90 can't find this issue. So In the future,
> rhel8.2.z will contain a fixed grub version?

It does contain the fix but the problem is that the GRUB core image that is
installed in the GRUB embedding area (the gap between the MBR and the start
of the first partition) is not updated automatically on package upgrade.

So to fix the grub2-install command has to be executed (as mentioned in the
Comment 17).

If you can confirm that the error can't be reproduced after grub2-install,
then I think this BZ should be closed as duplicated of bug #1761496.