Bug 2176261

Summary: grub2-mkconfig problem with cmdline update
Product: Red Hat OpenStack Reporter: Maciej Relewicz <mrelewicz>
Component: openstack-tripleo-heat-templatesAssignee: Vijayalakshmi Candappa <vcandapp>
Status: NEW --- QA Contact: Joe H. Rahme <jhakimra>
Severity: medium Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: camorris, cfontain, hakhande, jmitterm, jpretori, jslagle, mburns, rurena, vcandapp
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Maciej Relewicz 2023-03-07 20:05:32 UTC 
Description of problem:

We are upgrading from rhosp13 to rhosp16.2.4. We create support case for it: 03452776 which is pointing to: https://access.redhat.com/solutions/6989790.

Looks like there is a problem with grub2-mkconfig, we should run grubby at the end. Is there any solution for this case prepared which can be shared with us? We will backport it to our labs. Or you can share any workaround for us which can be implemented in automation which we have for upgrade.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 4 Maciej Relewicz 2023-03-17 11:57:50 UTC 
Proposed solution to put:

UpgradeInitCommand: |
    {% if 'ComputeDpdk' in group_names %}
    sudo grubby --update-kernel=ALL --args="intel_iommu=on iommu=pt"
    {% endif %}

doesnt work, cmdline is not updated, grub configfiles consists proper entry.

# cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos2)/boot/vmlinuz-4.18.0-305.72.1.el8_4.x86_64 root=UUID=fe5bf5b7-6d5c-4cef-9b50-49b297f774da ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=100 hugepagesz=2M hugepages=8192 kernelopts=root=UUID=fe5bf5b7-6d5c-4cef-9b50-49b297f774da ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=100 hugepagesz=2M hugepages=8192 skew_tick=1 nohz=on nohz_full=1,2 rcu_nocbs=1,2 tuned.non_isolcpus=fffffff9 intel_pstate=disable nosoftlockup skew_tick=1 nohz=on nohz_full=1,2 rcu_nocbs=1,2 tuned.non_isolcpus=fffffff9 intel_pstate=disable nosoftlockup

# grep -r intel_iommu /boot/
/boot/grub2/grub.cfg:  set kernelopts="root=UUID=fe5bf5b7-6d5c-4cef-9b50-49b297f774da ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet  intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G hugepages=32 hugepagesz=2M hugepages=1024  "
/boot/grub2/grubenv:kernelopts=root=UUID=fe5bf5b7-6d5c-4cef-9b50-49b297f774da ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet  intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G hugepages=32 hugepagesz=2M hugepages=1024
/boot/efi/EFI/BOOT/grub.cfg:  set kernelopts="root=UUID=fe5bf5b7-6d5c-4cef-9b50-49b297f774da ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet  intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G hugepages=32 hugepagesz=2M hugepages=1024  "
/boot/efi/EFI/BOOT/grubenv:kernelopts=root=UUID=fe5bf5b7-6d5c-4cef-9b50-49b297f774da ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet  intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G hugepages=32 hugepagesz=2M hugepages=1024
/boot/efi/EFI/redhat/grub.cfg:  set kernelopts="root=UUID=fe5bf5b7-6d5c-4cef-9b50-49b297f774da ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet  intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G hugepages=32 hugepagesz=2M hugepages=1024  "
/boot/efi/EFI/redhat/grubenv:kernelopts=root=UUID=fe5bf5b7-6d5c-4cef-9b50-49b297f774da ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet  intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G hugepages=32 hugepagesz=2M hugepages=1024

But i dont see this additlional line in logfiles:

# grep -r grubby /var/log/leapp/leapp-upgrade.log
2023-03-17 00:52:11.43  DEBUG    PID: 1036523 leapp.workflow.FactsCollection.source_boot_loader_scanner: External command has started: ['grubby', '--info', 'ALL']
2023-03-17 00:52:11.73  DEBUG    PID: 1036523 leapp.workflow.FactsCollection.source_boot_loader_scanner: External command has finished: ['grubby', '--info', 'ALL']
2023-03-17 00:52:12.379 DEBUG    PID: 1036979 leapp.workflow.FactsCollection.rpm_scanner: grubby|8.28|26.el7|0|Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>|x86_64|RSA/SHA256, Mon 25 Mar 2019 04:40:23 PM UTC, Key ID 199e2f91fd431d51
2023-03-17 00:53:43.598 DEBUG    PID: 1043088 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: ---> Package grubby.x86_64 8.40-41.el8 will be installed
2023-03-17 00:53:43.966 DEBUG    PID: 1043088 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator:  grubby                        x86_648.40-41.el8                     rhel-8-for-x86_64-baseos-eus-rpms    49 k
2023-03-17 00:53:47.924 DEBUG    PID: 1043088 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: (91/198): grubby-8.40-41.el8.x86_64.rpm         475 kB/s |  49 kB     00:00
2023-03-17 00:54:15.415 DEBUG    PID: 1043088 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator:   Installing       : grubby-8.40-41.el8.x86_64                          181/198
2023-03-17 00:54:20.639 DEBUG    PID: 1043088 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator:   Verifying        : grubby-8.40-41.el8.x86_64                           92/198
2023-03-17 00:54:20.801 DEBUG    PID: 1043088 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator:   grubby-8.40-41.el8.x86_64
2023-03-17 00:54:55.528 DEBUG    PID: 1043657 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: ---> Package grubby.x86_64 8.28-26.el7 will be upgraded
2023-03-17 00:54:55.529 DEBUG    PID: 1043657 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: ---> Package grubby.x86_64 8.40-41.el8 will be an upgrade
2023-03-17 00:54:56.802 DEBUG    PID: 1043657 leapp.workflow.TargetTransactionCheck.dnf_transaction_check:  grubby                                           x86_64  8.40-41.el8                                    rhel-8-for-x86_64-baseos-eus-rpms             49 k
2023-03-17 00:55:30.730 DEBUG    PID: 1043970 leapp.workflow.Download.dnf_package_download: ---> Package grubby.x86_64 8.28-26.el7 will be upgraded
2023-03-17 00:55:30.731 DEBUG    PID: 1043970 leapp.workflow.Download.dnf_package_download: ---> Package grubby.x86_64 8.40-41.el8 will be an upgrade
2023-03-17 00:55:32.5   DEBUG    PID: 1043970 leapp.workflow.Download.dnf_package_download:  grubby                                           x86_64  8.40-41.el8                                    rhel-8-for-x86_64-baseos-eus-rpms             49 k
2023-03-17 00:55:33.304 DEBUG    PID: 1043970 leapp.workflow.Download.dnf_package_download: [SKIPPED] grubby-8.40-41.el8.x86_64.rpm: Already downloaded
2023-03-17 00:59:10.372 DEBUG    PID: 37723 leapp.workflow.InterimPreparation.dnf_dry_run: ---> Package grubby.x86_64 8.28-26.el7 will be upgraded
2023-03-17 00:59:10.373 DEBUG    PID: 37723 leapp.workflow.InterimPreparation.dnf_dry_run: ---> Package grubby.x86_64 8.40-41.el8 will be an upgrade
2023-03-17 00:59:11.649 DEBUG    PID: 37723 leapp.workflow.InterimPreparation.dnf_dry_run:  grubby                                           x86_64  8.40-41.el8                                    rhel-8-for-x86_64-baseos-eus-rpms             49 k
2023-03-17 00:59:29.701 DEBUG    PID: 37898 leapp.workflow.InterimPreparation.add_upgrade_boot_entry: External command has started: ['/usr/sbin/grubby', '--remove-kernel', '/boot/vmlinuz-upgrade.x86_64']
2023-03-17 00:59:29.718 DEBUG    PID: 37898 leapp.workflow.InterimPreparation.add_upgrade_boot_entry: External command has finished: ['/usr/sbin/grubby', '--remove-kernel', '/boot/vmlinuz-upgrade.x86_64']
2023-03-17 00:59:29.719 DEBUG    PID: 37898 leapp.workflow.InterimPreparation.add_upgrade_boot_entry: External command has started: ['/usr/sbin/grubby', '--add-kernel', '/boot/vmlinuz-upgrade.x86_64', '--initrd', '/boot/initramfs-upgrade.x86_64.img', '--title', 'RHEL-Upgrade-Initramfs', '--copy-default', '--make-default', '--args', 'debug enforcing=0 rd.plymouth=0 plymouth.enable=0']
2023-03-17 00:59:29.763 DEBUG    PID: 37898 leapp.workflow.InterimPreparation.add_upgrade_boot_entry: External command has finished: ['/usr/sbin/grubby', '--add-kernel', '/boot/vmlinuz-upgrade.x86_64', '--initrd', '/boot/initramfs-upgrade.x86_64.img', '--title', 'RHEL-Upgrade-Initramfs', '--copy-default', '--make-default', '--args', 'debug enforcing=0 rd.plymouth=0 plymouth.enable=0']
2023-03-17 01:06:00.750 DEBUG    PID: 459 leapp.workflow.InitRamStart.remove_upgrade_boot_entry: External command has started: ['/usr/sbin/grubby', '--remove-kernel=/boot/vmlinuz-upgrade.x86_64']
2023-03-17 01:06:00.907 DEBUG    PID: 459 leapp.workflow.InitRamStart.remove_upgrade_boot_entry: External command has finished: ['/usr/sbin/grubby', '--remove-kernel=/boot/vmlinuz-upgrade.x86_64']
2023-03-17 01:06:59.98  DEBUG    PID: 991 leapp.workflow.RPMUpgrade.dnf_upgrade_transaction: ---> Package grubby.x86_64 8.28-26.el7 will be upgraded
2023-03-17 01:06:59.100 DEBUG    PID: 991 leapp.workflow.RPMUpgrade.dnf_upgrade_transaction: ---> Package grubby.x86_64 8.40-41.el8 will be an upgrade
2023-03-17 01:07:01.367 DEBUG    PID: 991 leapp.workflow.RPMUpgrade.dnf_upgrade_transaction:  grubby                                           x86_64  8.40-41.el8                                    rhel-8-for-x86_64-baseos-eus-rpms             49 k
2023-03-17 01:08:23.368 DEBUG    PID: 991 leapp.workflow.RPMUpgrade.dnf_upgrade_transaction:   Upgrading        : grubby-8.40-41.el8.x86_64                         328/1808
2023-03-17 01:11:10.800 DEBUG    PID: 991 leapp.workflow.RPMUpgrade.dnf_upgrade_transaction:   Cleanup          : grubby-8.28-26.el7.x86_64                        1161/1808
2023-03-17 01:14:11.751 DEBUG    PID: 991 leapp.workflow.RPMUpgrade.dnf_upgrade_transaction:   Verifying        : grubby-8.40-41.el8.x86_64                        1143/1808
2023-03-17 01:14:11.754 DEBUG    PID: 991 leapp.workflow.RPMUpgrade.dnf_upgrade_transaction:   Verifying        : grubby-8.28-26.el7.x86_64                        1144/1808
2023-03-17 01:14:13.555 DEBUG    PID: 991 leapp.workflow.RPMUpgrade.dnf_upgrade_transaction: Upgraded: grubby-8.40-41.el8.x86_64
2023-03-17 01:14:16.117 DEBUG    PID: 991 leapp.workflow.RPMUpgrade.dnf_upgrade_transaction:   grubby-8.40-41.el8.x86_64
2023-03-17 01:14:24.218 DEBUG    PID: 30912 leapp.workflow.RPMUpgrade.check_leftover_packages: grubby|8.40|41.el8|0|Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>|x86_64|RSA/SHA256, Tue 12 May 2020 07:18:40 AM UTC, Key ID 199e2f91fd431d51
2023-03-17 01:15:54.777 DEBUG    PID: 1708 leapp.workflow.Finalization.kernelcmdlineconfig: External command has started: ['grubby', '--update-kernel=/boot/vmlinuz-4.18.0-305.72.1.el8_4.x86_64', '--remove-args', 'debug enforcing=0']
2023-03-17 01:15:54.887 DEBUG    PID: 1708 leapp.workflow.Finalization.kernelcmdlineconfig: External command has finished: ['grubby', '--update-kernel=/boot/vmlinuz-4.18.0-305.72.1.el8_4.x86_64', '--remove-args', 'debug enforcing=0']
2023-03-17 01:15:54.957 DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has started: ['grubby', '--default-kernel']
2023-03-17 01:15:55.37  DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has finished: ['grubby', '--default-kernel']
2023-03-17 01:15:55.40  DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has started: ['grubby', '--default-index']
2023-03-17 01:15:55.114 DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has finished: ['grubby', '--default-index']
2023-03-17 01:15:55.116 DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has started: ['grubby', '--default-title']
2023-03-17 01:15:55.191 DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has finished: ['grubby', '--default-title']
Mar 17 01:06:06 localhost upgrade[908]: 2023-03-17 01:06:00.750 DEBUG    PID: 459 leapp.workflow.InitRamStart.remove_upgrade_boot_entry: External command has started: ['/usr/sbin/grubby', '--remove-kernel=/boot/vmlinuz-upgrade.x86_64']
Mar 17 01:06:06 localhost upgrade[908]: 2023-03-17 01:06:00.907 DEBUG    PID: 459 leapp.workflow.InitRamStart.remove_upgrade_boot_entry: External command has finished: ['/usr/sbin/grubby', '--remove-kernel=/boot/vmlinuz-upgrade.x86_64']
Mar 17 01:07:08 localhost upgrade[908]: ---> Package grubby.x86_64 8.28-26.el7 will be upgraded
Mar 17 01:07:08 localhost upgrade[908]: ---> Package grubby.x86_64 8.40-41.el8 will be an upgrade
Mar 17 01:07:21 localhost upgrade[908]:  grubby                                           x86_64  8.40-41.el8                                    rhel-8-for-x86_64-baseos-eus-rpms             49 k
Mar 17 01:08:24 localhost upgrade[908]:   Upgrading        : grubby-8.40-41.el8.x86_64                         328/1808
Mar 17 01:11:11 localhost upgrade[908]:   Cleanup          : grubby-8.28-26.el7.x86_64                        1161/1808
Mar 17 01:14:21 localhost upgrade[908]:   Verifying        : grubby-8.40-41.el8.x86_64                        1143/1808
Mar 17 01:14:21 localhost upgrade[908]:   Verifying        : grubby-8.28-26.el7.x86_64                        1144/1808
Mar 17 01:14:30 localhost upgrade[908]: Upgraded: grubby-8.40-41.el8.x86_64
Mar 17 01:14:40 localhost upgrade[908]:   grubby-8.40-41.el8.x86_64
Mar 17 01:15:03 localhost upgrade[908]: grubby|8.40|41.el8|0|Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>|x86_64|RSA/SHA256, Tue 12 May 2020 07:18:40 AM UTC, Key ID 199e2f91fd431d51
Mar 17 01:15:54 localhost upgrade[908]: 2023-03-17 01:15:54.777 DEBUG    PID: 1708 leapp.workflow.Finalization.kernelcmdlineconfig: External command has started: ['grubby', '--update-kernel=/boot/vmlinuz-4.18.0-305.72.1.el8_4.x86_64', '--remove-args', 'debug enforcing=0']
Mar 17 01:15:54 localhost upgrade[908]: 2023-03-17 01:15:54.887 DEBUG    PID: 1708 leapp.workflow.Finalization.kernelcmdlineconfig: External command has finished: ['grubby', '--update-kernel=/boot/vmlinuz-4.18.0-305.72.1.el8_4.x86_64', '--remove-args', 'debug enforcing=0']
Mar 17 01:15:55 localhost upgrade[908]: 2023-03-17 01:15:54.957 DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has started: ['grubby', '--default-kernel']
Mar 17 01:15:55 localhost upgrade[908]: 2023-03-17 01:15:55.37  DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has finished: ['grubby', '--default-kernel']
Mar 17 01:15:55 localhost upgrade[908]: 2023-03-17 01:15:55.40  DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has started: ['grubby', '--default-index']
Mar 17 01:15:55 localhost upgrade[908]: 2023-03-17 01:15:55.114 DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has finished: ['grubby', '--default-index']
Mar 17 01:15:55 localhost upgrade[908]: 2023-03-17 01:15:55.116 DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has started: ['grubby', '--default-title']
Mar 17 01:15:55 localhost upgrade[908]: 2023-03-17 01:15:55.191 DEBUG    PID: 1876 leapp.workflow.Finalization.force_default_boot_to_target_kernel_version: External command has finished: ['grubby', '--default-title']

from tht template was properly move to configuration

# grep -r grubby /var/lib/mistral/26c702d1-4266-408f-b02b-e524022b10ed/ContrailDpdk/
/var/lib/mistral/26c702d1-4266-408f-b02b-e524022b10ed/ContrailDpdk/upgrade_tasks.yaml:      \ 'Compute' in group_names %}\n   sudo grubby --update-kernel=/boot/vmlinuz-$(uname\
/var/lib/mistral/26c702d1-4266-408f-b02b-e524022b10ed/ContrailDpdk/upgrade_tasks.yaml:      \ %}\n\n{% if 'ComputeDpdk' in group_names %}\n  sudo grubby --update-kernel=/boot/vmlinuz-$(uname\
/var/lib/mistral/26c702d1-4266-408f-b02b-e524022b10ed/ContrailDpdk/upgrade_tasks_step0.yaml:      \ 'Compute' in group_names %}\n   sudo grubby --update-kernel=/boot/vmlinuz-$(uname\
/var/lib/mistral/26c702d1-4266-408f-b02b-e524022b10ed/ContrailDpdk/upgrade_tasks_step0.yaml:      \ %}\n\n{% if 'ComputeDpdk' in group_names %}\n  sudo grubby --update-kernel=/boot/vmlinuz-$(uname\


on mistral logs:
# grep -r overcloud-contraildpdk-1 package_update.log.1 | grep grub
2023-03-16 20:50:31,011 p=563310 u=mistral n=ansible | 2023-03-16 20:50:31.011163 | 52540003-b680-057f-b155-000000000226 |    CHANGED | fix grub entries to have name start with GRUB_ | overcloud-contraildpdk-1
2023-03-16 20:50:31,281 p=563310 u=mistral n=ansible | 2023-03-16 20:50:31.281231 | 52540003-b680-057f-b155-000000000227 |    CHANGED | fix grub entries in append statement | overcloud-contraildpdk-1
2023-03-16 20:51:34,506 p=563310 u=mistral n=ansible | 2023-03-16 20:51:34.506067 | 52540003-b680-057f-b155-00000000023e |    SKIPPED | Create grub config file | overcloud-contraildpdk-1
2023-03-16 20:59:33,392 p=563310 u=mistral n=ansible | 2023-03-16 20:59:33.392505 | 52540003-b680-057f-b155-000000000226 |    SUMMARY | overcloud-contraildpdk-1 | fix grub entries to have name start with GRUB_ | 0.31s
2023-03-16 21:00:44,971 p=564649 u=mistral n=ansible | 2023-03-16 21:00:44.970889 | 52540003-b680-d433-475b-000000000252 |    SKIPPED | Stat /boot/grub2/grubenv | overcloud-contraildpdk-1
2023-03-16 21:00:45,085 p=564649 u=mistral n=ansible | 2023-03-16 21:00:45.085508 | 52540003-b680-d433-475b-000000000253 |    SKIPPED | Copy /boot/grub2/grubenv to /boot/efi/EFI/redhat/grubenv | overcloud-contraildpdk-1
2023-03-16 22:03:55,553 p=567518 u=mistral n=ansible | 2023-03-16 22:03:55.552936 | 52540003-b680-19b1-e78c-000000000496 |         OK | Check grub config paths | overcloud-contraildpdk-1 | item=/boot/efi/EFI/BOOT
2023-03-16 22:03:55,757 p=567518 u=mistral n=ansible | 2023-03-16 22:03:55.757136 | 52540003-b680-19b1-e78c-000000000496 |         OK | Check grub config paths | overcloud-contraildpdk-1 | item=/boot/efi/EFI/redhat
2023-03-16 22:03:55,960 p=567518 u=mistral n=ansible | 2023-03-16 22:03:55.960778 | 52540003-b680-19b1-e78c-000000000496 |         OK | Check grub config paths | overcloud-contraildpdk-1 | item=/boot/efi/EFI/centos
2023-03-16 22:03:56,164 p=567518 u=mistral n=ansible | 2023-03-16 22:03:56.163962 | 52540003-b680-19b1-e78c-000000000496 |         OK | Check grub config paths | overcloud-contraildpdk-1 | item=/boot/efi/EFI/fedora
2023-03-16 22:03:57,953 p=567518 u=mistral n=ansible | 2023-03-16 22:03:57.953132 | 52540003-b680-19b1-e78c-000000000497 |    CHANGED | Generate grub config | overcloud-contraildpdk-1
2023-03-16 22:03:59,640 p=567518 u=mistral n=ansible | 2023-03-16 22:03:59.639950 | 52540003-b680-19b1-e78c-000000000498 |    CHANGED | Generate EFI grub config | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': True, 'path': '/boot/efi/EFI/BOOT', 'mode': '0700', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 42, 'inode': 107, 'dev': 2066, 'nlink': 2, 'atime': 1679015888.4851851, 'mtime': 1679015429.0975187, 'ctime': 1679015429.0975187, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 0, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'root', 'gr_name': 'root', 'mimetype': 'inode/directory', 'charset': 'binary', 'version': '936753402', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/BOOT', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/BOOT', 'ansible_loop_var': 'item'}
2023-03-16 22:04:01,211 p=567518 u=mistral n=ansible | 2023-03-16 22:04:01.211113 | 52540003-b680-19b1-e78c-000000000498 |    CHANGED | Generate EFI grub config | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': True, 'path': '/boot/efi/EFI/redhat', 'mode': '0700', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 119, 'inode': 4194511, 'dev': 2066, 'nlink': 3, 'atime': 1679015888.4851851, 'mtime': 1679015490.5665166, 'ctime': 1679015490.5665166, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 0, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'root', 'gr_name': 'root', 'mimetype': 'inode/directory', 'charset': 'binary', 'version': '731852227', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/redhat', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/redhat', 'ansible_loop_var': 'item'}
2023-03-16 22:04:01,214 p=567518 u=mistral n=ansible | 2023-03-16 22:04:01.214551 | 52540003-b680-19b1-e78c-000000000498 |    SKIPPED | Generate EFI grub config | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/centos', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/centos', 'ansible_loop_var': 'item'}
2023-03-16 22:04:01,216 p=567518 u=mistral n=ansible | 2023-03-16 22:04:01.215911 | 52540003-b680-19b1-e78c-000000000498 |    SKIPPED | Generate EFI grub config | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/fedora', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/fedora', 'ansible_loop_var': 'item'}
2023-03-16 22:04:01,646 p=567518 u=mistral n=ansible | 2023-03-16 22:04:01.645773 | 52540003-b680-19b1-e78c-000000000499 |    CHANGED | Copy grubenv to EFI directory | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': True, 'path': '/boot/efi/EFI/BOOT', 'mode': '0700', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 42, 'inode': 107, 'dev': 2066, 'nlink': 2, 'atime': 1679015888.4851851, 'mtime': 1679015429.0975187, 'ctime': 1679015429.0975187, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 0, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'root', 'gr_name': 'root', 'mimetype': 'inode/directory', 'charset': 'binary', 'version': '936753402', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/BOOT', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/BOOT', 'ansible_loop_var': 'item'}
2023-03-16 22:04:01,955 p=567518 u=mistral n=ansible | 2023-03-16 22:04:01.955127 | 52540003-b680-19b1-e78c-000000000499 |    CHANGED | Copy grubenv to EFI directory | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': True, 'path': '/boot/efi/EFI/redhat', 'mode': '0700', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 119, 'inode': 4194511, 'dev': 2066, 'nlink': 3, 'atime': 1679015888.4851851, 'mtime': 1679015490.5665166, 'ctime': 1679015490.5665166, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 0, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'root', 'gr_name': 'root', 'mimetype': 'inode/directory', 'charset': 'binary', 'version': '731852227', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/redhat', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/redhat', 'ansible_loop_var': 'item'}
2023-03-16 22:04:01,961 p=567518 u=mistral n=ansible | 2023-03-16 22:04:01.960993 | 52540003-b680-19b1-e78c-000000000499 |    SKIPPED | Copy grubenv to EFI directory | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/centos', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/centos', 'ansible_loop_var': 'item'}
2023-03-16 22:04:02,013 p=567518 u=mistral n=ansible | 2023-03-16 22:04:02.013391 | 52540003-b680-19b1-e78c-000000000499 |    SKIPPED | Copy grubenv to EFI directory | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/fedora', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/fedora', 'ansible_loop_var': 'item'}
2023-03-16 22:10:58,063 p=567518 u=mistral n=ansible | 2023-03-16 22:10:58.063145 | 52540003-b680-19b1-e78c-000000000498 |    SUMMARY | overcloud-contraildpdk-1 | tripleo-kernel : Generate EFI grub config | 3.21s
2023-03-16 22:10:58,063 p=567518 u=mistral n=ansible | 2023-03-16 22:10:58.063560 | 52540003-b680-19b1-e78c-000000000497 |    SUMMARY | overcloud-contraildpdk-1 | tripleo-kernel : Generate grub config | 1.73s
2023-03-16 22:10:58,063 p=567518 u=mistral n=ansible | 2023-03-16 22:10:58.063659 | 52540003-b680-19b1-e78c-000000000496 |    SUMMARY | overcloud-contraildpdk-1 | tripleo-kernel : Check grub config paths | 0.87s
2023-03-16 22:10:58,064 p=567518 u=mistral n=ansible | 2023-03-16 22:10:58.064069 | 52540003-b680-19b1-e78c-000000000499 |    SUMMARY | overcloud-contraildpdk-1 | tripleo-kernel : Copy grubenv to EFI directory | 0.74s
2023-03-16 22:57:21,569 p=570608 u=mistral n=ansible | 2023-03-16 22:57:21.569098 | 52540003-b680-0417-8ee6-000000000496 |         OK | Check grub config paths | overcloud-contraildpdk-1 | item=/boot/efi/EFI/BOOT
2023-03-16 22:57:21,774 p=570608 u=mistral n=ansible | 2023-03-16 22:57:21.774229 | 52540003-b680-0417-8ee6-000000000496 |         OK | Check grub config paths | overcloud-contraildpdk-1 | item=/boot/efi/EFI/redhat
2023-03-16 22:57:21,978 p=570608 u=mistral n=ansible | 2023-03-16 22:57:21.978593 | 52540003-b680-0417-8ee6-000000000496 |         OK | Check grub config paths | overcloud-contraildpdk-1 | item=/boot/efi/EFI/centos
2023-03-16 22:57:22,183 p=570608 u=mistral n=ansible | 2023-03-16 22:57:22.183351 | 52540003-b680-0417-8ee6-000000000496 |         OK | Check grub config paths | overcloud-contraildpdk-1 | item=/boot/efi/EFI/fedora
2023-03-16 22:57:23,870 p=570608 u=mistral n=ansible | 2023-03-16 22:57:23.870693 | 52540003-b680-0417-8ee6-000000000497 |    CHANGED | Generate grub config | overcloud-contraildpdk-1
2023-03-16 22:57:25,562 p=570608 u=mistral n=ansible | 2023-03-16 22:57:25.561764 | 52540003-b680-0417-8ee6-000000000498 |    CHANGED | Generate EFI grub config | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': True, 'path': '/boot/efi/EFI/BOOT', 'mode': '0700', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 73, 'inode': 107, 'dev': 2066, 'nlink': 2, 'atime': 1679015888.4851851, 'mtime': 1679018641.5657248, 'ctime': 1679018641.5657248, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 0, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'root', 'gr_name': 'root', 'mimetype': 'inode/directory', 'charset': 'binary', 'version': '936753402', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/BOOT', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/BOOT', 'ansible_loop_var': 'item'}
2023-03-16 22:57:27,082 p=570608 u=mistral n=ansible | 2023-03-16 22:57:27.082515 | 52540003-b680-0417-8ee6-000000000498 |    CHANGED | Generate EFI grub config | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': True, 'path': '/boot/efi/EFI/redhat', 'mode': '0700', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 150, 'inode': 4194511, 'dev': 2066, 'nlink': 3, 'atime': 1679015888.4851851, 'mtime': 1679018641.8767266, 'ctime': 1679018641.8767266, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 0, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'root', 'gr_name': 'root', 'mimetype': 'inode/directory', 'charset': 'binary', 'version': '731852227', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/redhat', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/redhat', 'ansible_loop_var': 'item'}
2023-03-16 22:57:27,136 p=570608 u=mistral n=ansible | 2023-03-16 22:57:27.136630 | 52540003-b680-0417-8ee6-000000000498 |    SKIPPED | Generate EFI grub config | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/centos', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/centos', 'ansible_loop_var': 'item'}
2023-03-16 22:57:27,138 p=570608 u=mistral n=ansible | 2023-03-16 22:57:27.138046 | 52540003-b680-0417-8ee6-000000000498 |    SKIPPED | Generate EFI grub config | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/fedora', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/fedora', 'ansible_loop_var': 'item'}
2023-03-16 22:57:27,570 p=570608 u=mistral n=ansible | 2023-03-16 22:57:27.570521 | 52540003-b680-0417-8ee6-000000000499 |         OK | Copy grubenv to EFI directory | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': True, 'path': '/boot/efi/EFI/BOOT', 'mode': '0700', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 73, 'inode': 107, 'dev': 2066, 'nlink': 2, 'atime': 1679015888.4851851, 'mtime': 1679018641.5657248, 'ctime': 1679018641.5657248, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 0, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'root', 'gr_name': 'root', 'mimetype': 'inode/directory', 'charset': 'binary', 'version': '936753402', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/BOOT', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/BOOT', 'ansible_loop_var': 'item'}
2023-03-16 22:57:27,878 p=570608 u=mistral n=ansible | 2023-03-16 22:57:27.878335 | 52540003-b680-0417-8ee6-000000000499 |         OK | Copy grubenv to EFI directory | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': True, 'path': '/boot/efi/EFI/redhat', 'mode':'0700', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 150, 'inode': 4194511, 'dev': 2066, 'nlink': 3, 'atime': 1679015888.4851851, 'mtime': 1679018641.8767266, 'ctime': 1679018641.8767266, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 0, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'root', 'gr_name': 'root', 'mimetype': 'inode/directory', 'charset': 'binary', 'version': '731852227', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/redhat', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/redhat', 'ansible_loop_var': 'item'}
2023-03-16 22:57:27,882 p=570608 u=mistral n=ansible | 2023-03-16 22:57:27.882302 | 52540003-b680-0417-8ee6-000000000499 |    SKIPPED | Copy grubenv to EFI directory | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/centos', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/centos', 'ansible_loop_var': 'item'}
2023-03-16 22:57:27,934 p=570608 u=mistral n=ansible | 2023-03-16 22:57:27.934230 | 52540003-b680-0417-8ee6-000000000499 |    SKIPPED | Copy grubenv to EFI directory | overcloud-contraildpdk-1 | item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/boot/efi/EFI/fedora', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/boot/efi/EFI/fedora', 'ansible_loop_var': 'item'}
2023-03-16 23:03:22,981 p=570608 u=mistral n=ansible | 2023-03-16 23:03:22.981753 | 52540003-b680-0417-8ee6-000000000498 |    SUMMARY | overcloud-contraildpdk-1 | tripleo-kernel : Generate EFI grub config | 3.21s
2023-03-16 23:03:22,981 p=570608 u=mistral n=ansible | 2023-03-16 23:03:22.981849 | 52540003-b680-0417-8ee6-000000000497 |    SUMMARY | overcloud-contraildpdk-1 | tripleo-kernel : Generate grub config | 1.62s
2023-03-16 23:03:22,982 p=570608 u=mistral n=ansible | 2023-03-16 23:03:22.982082 | 52540003-b680-0417-8ee6-000000000496 |    SUMMARY | overcloud-contraildpdk-1 | tripleo-kernel : Check grub config paths | 0.88s
2023-03-16 23:03:22,982 p=570608 u=mistral n=ansible | 2023-03-16 23:03:22.982425 | 52540003-b680-0417-8ee6-000000000499 |    SUMMARY | overcloud-contraildpdk-1 | tripleo-kernel : Copy grubenv to EFI directory | 0.73s


when i look into deploy_task for dpdk node i see defined properly kernel args, in step 0, you are manipulating on grub/apply fixes in step4 maybe during that you are dropping things?:

- include_role:
    name: tripleo-kernel
    tasks_from: kernelargs.yml
  name: Configure kernel args and reboot
  vars:
    tripleo_kernel_args: intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G
      hugepages=32 hugepagesz=2M hugepages=1024
    tripleo_kernel_defer_reboot: false
    tripleo_kernel_reboot_timeout: 900
  when: step|int == 0
Comment 5 Rafael Urena 2023-03-20 14:41:46 UTC 
is there a workaround or anything? Get the customer moving forward until a fix can be provided.

Rafael Ureña
Technical Account Manager