Bug 1959945

Summary: [NBDE] RHVH 4.4.6 host fails to startup, without prompting for passphrase
Product: Red Hat Enterprise Virtualization Manager Reporter: SATHEESARAN <sasundar>
Component: vdsmAssignee: Ales Musil <amusil>
Status: CLOSED ERRATA QA Contact: Michael Burman <mburman>
Severity: high Docs Contact:
Priority: high    
Version: 4.4.6CC: arachman, cshao, dfediuck, godas, klaas, lsurette, lsvaty, lveyde, mavital, mburman, michal.skrivanek, mperina, peyu, rhs-bugs, sbonazzo, shlei, srevivo, weiwang, yaniwang, ycui
Target Milestone: ovirt-4.4.6-1Keywords: Regression, ZStream
Target Release: 4.4.6   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: vdsm-4.40.60.7 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1959908 Environment:
Last Closed: 2021-06-03 10:25:43 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1955571    
Bug Blocks: 1959908    

Description SATHEESARAN 2021-05-12 17:14:17 UTC
+++ This bug was initially created as a clone of Bug #1959908 +++

Description of problem:
-----------------------
After binding to tang server, the RHVH 4.4.6 fails to start automatically without prompting for passphrase

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
RHVH 4.4.6
clevis-dracut-15-1.el8.x86_64
clevis-15-1.el8.x86_64
clevis-luks-15-1.el8.x86_64
clevis-systemd-15-1.el8.x86_64

How reproducible:
-----------------
Always

Steps to Reproduce:
--------------------
1. Bind to tang server
2. Reboot the RHVH node

Actual results:
----------------
RHVH fails to start automatically without passphrase

Expected results:
------------------
RHVH should start automatically without prompting for passphrase

Additional info:

--- Additional comment from SATHEESARAN on 2021-05-12 15:45:09 UTC ---

While rebuilding initramfs on RHVH 4.4.6, clevis and clevis-pin-tang dracut modules are missing

[root@ ~]# dracut -vf --regenerate-all --hostonly-cmdline
dracut: Executing: /usr/bin/dracut --kver=4.18.0-304.el8.x86_64 -vf --hostonly-cmdline
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'ifcfg' will not be installed, because it's in the list to be omitted!
dracut: dracut module 'clevis' will not be installed, because it's in the list to be omitted!
dracut: dracut module 'clevis' will not be installed, because it's in the list to be omitted!
dracut: dracut module 'clevis-pin-sss' depends on 'clevis', which can't be installed
dracut: dracut module 'clevis-pin-tang' depends on 'clevis', which can't be installed
..........


[root@ ~]# lsinitrd | grep -i clevis
[root@ ~]#

The same was working well with RHVH 4.4.4-async

--- Additional comment from SATHEESARAN on 2021-05-12 17:12:27 UTC ---

I have found out the root cause of this issue.
vdsm has dropped the dracut configuration file that omits 'clevis' dracut module
This is the reason initramfs lacked clevis and clevis-pin-tang dracut modules, which prevented
the RHVH host from starting up automatically without prompting for passphrase.

[root@ ~]# rpm -qf /usr/lib/dracut/dracut.conf.d/99-vdsm_protect_ifcfg.conf 
vdsm-4.40.50.10-1.el8ev.x86_64

[root@ ~]# cat /usr/lib/dracut/dracut.conf.d/99-vdsm_protect_ifcfg.conf
omit_dracutmodules+=" ifcfg clevis "

So this makes the clevis dracut module not to be included.

Now we need to understand, why clevis dracut module was omitted.
And vdsm should not omit 'clevis' along with 'ifcfg'

Comment 1 Martin Perina 2021-05-13 06:40:07 UTC
This can't be a regresssion, we haven't done any changes around clevis module since 4.4.0 (more info in https://bugzilla.redhat.com/show_bug.cgi?id=1955571#c7 ). BZ1955571 is targeted to 4.4.7, because doing this change is risky, we need to run complete network automation tests to look for regressions and still can't be sure.

And on the other I don't see a way how this could work before in RHV 4.4.z as he have disabled clevis module iin 4.4.0 as part of BZ1760262

Comment 4 Michal Skrivanek 2021-05-13 14:08:58 UTC
I also suppose an easy workaround is to remove /usr/lib/dracut/dracut.conf.d/99-vdsm_protect_ifcfg.conf or add anotheer drop-in to override it back

Comment 5 SATHEESARAN 2021-05-14 05:06:53 UTC
(In reply to Martin Perina from comment #1)
> This can't be a regresssion, we haven't done any changes around clevis
> module since 4.4.0 (more info in
> https://bugzilla.redhat.com/show_bug.cgi?id=1955571#c7 ). BZ1955571 is
> targeted to 4.4.7, because doing this change is risky, we need to run
> complete network automation tests to look for regressions and still can't be
> sure.
> 
> And on the other I don't see a way how this could work before in RHV 4.4.z
> as he have disabled clevis module iin 4.4.0 as part of BZ1760262

Yes, you are right. But from RHHI-V side, we have included the dracut drop-in configuration file
under /etc/dracut.conf.d/ which was working earlier but this isn't working with RHVH 4.4.6 ( based on RHEL 8.4 )

Contents of this dracut configuration file:
[root@ ~]# cat /etc/dracut.conf.d/clevis.conf 
# BEGIN Entry for enp129s0f0
kernel_cmdline="ip=enp129s0f0:dhcp"
omit_dracutmodules+="ifcfg"
omit_dracutmodules+="network-legacy"
add_dracutmodules+="clevis network-manager"      <------ This adds the clevis dracut module which was working good till RHVH 4.4.4-async2
# END Entry for enp129s0f0


But now with RHVH 4.4.6:

[root@ ~]# dracut -vf -m clevis
dracut: Executing: /usr/bin/dracut -vf -m clevis
dracut: dracut module 'ifcfg' will not be installed, because it's in the list to be omitted!
dracut: dracut module 'clevis' will not be installed, because it's in the list to be omitted!
dracut: dracut module 'clevis' cannot be found or installed.

[root@ ~]# cat /etc/dracut.conf.d/clevis.conf 
# BEGIN Entry for enp129s0f0
kernel_cmdline="ip=enp129s0f0:dhcp"
omit_dracutmodules+="ifcfg"
omit_dracutmodules+="network-legacy"
add_dracutmodules+="clevis network-manager"
# END Entry for enp129s0f0

[root@ ~]# dracut -vf --regenerate-all -m clevis
dracut: Executing: /usr/bin/dracut --kver=4.18.0-305.el8.x86_64 -vf -m clevis
dracut: dracut module 'ifcfg' will not be installed, because it's in the list to be omitted!
dracut: dracut module 'clevis' will not be installed, because it's in the list to be omitted!
dracut: dracut module 'clevis' cannot be found or installed.

Comment 9 Michael Burman 2021-05-23 09:15:07 UTC
Verified on - vdsm-4.40.60.7-1.el8ev.x86_64 with

rhvm-4.4.6.8-0.1.el8ev.noarch
nmstate-1.0.2-6.el8_4.noarch
NetworkManager-1.30.0-7.el8.x86_64

No regression found with this fix.

BZ 1959908 can be tested

Comment 21 errata-xmlrpc 2021-06-03 10:25:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Async RHV RHEL Host (ovirt-host) [ovirt-4.4.6]), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2240