Bug 2161533

Nice investigation. Do I get it right the autofs / dpkg / flatpak / systemd "verify commands" are not required as a default behaviour on OCP?

Keep in mind that --verify option also runs "rpm -V <pgklist>" whenever verify_packages variable is set by a plugin, which is:

$ grep " verify_packages " sos/report/plugins/*py
sos/report/plugins/block.py:    verify_packages = ('util-linux',)
sos/report/plugins/convert2rhel.py:    verify_packages = ('convert2rhel$',)
sos/report/plugins/java.py:    verify_packages = ('java.*',)
sos/report/plugins/kernel.py:    verify_packages = ('kernel$',)
sos/report/plugins/nss.py:    verify_packages = ('nss.*',)
sos/report/plugins/openssl.py:    verify_packages = ('openssl.*',)
sos/report/plugins/pam.py:    verify_packages = ('pam_.*',)
sos/report/plugins/perl.py:    verify_packages = ('perl.*',)
sos/report/plugins/rpm.py:    verify_packages = ('rpm',)
sos/report/plugins/system.py:    verify_packages = ('glibc', 'initscripts', 'zlib')
$

I see three possible implementations:
1) remove --verify from RHOCP preset completely. This will a) stop calling the "rpm -V .." checks (is that intentional?) and b) stop calling "ostree fsck", from default behaviour
2) introduce a new ostree plugin option "fsck", disabled by default (enabled via "-k ostree.fsck=yes" / "--plugin-option ostree.fsck=yes") and call "ostree fsck" when *this* option is enabled - then "rpm -V .." checks would be still executed by default
3) do both 1) and 2), such that a) "rpm -V .." are not further run by default, b) "ostree fsck" is disabled by default, and c) the fsck is enabled by more appropriate(?) plugin option, not the generic one


Please let us know what option is preferable.

Hi Pavel,

I'm the least to decide, but in terms of "ostree fsck", it shouldn't be started as it triggered some false alarms. I don't think it will do any harm, but taken that it will put quite some I/O load on the node, it will impact running workload.
Unless there are well known reasons (which I don't know about), we shouldn't do such verification steps during support-data collection.

IMHO option 3 would be most versatile option.

(In reply to Pavel Moravec from comment #1)
> I see three possible implementations:
> 1) remove --verify from RHOCP preset completely. This will a) stop calling
> the "rpm -V .." checks (is that intentional?) and b) stop calling "ostree
> fsck", from default behaviour
> 2) introduce a new ostree plugin option "fsck", disabled by default (enabled
> via "-k ostree.fsck=yes" / "--plugin-option ostree.fsck=yes") and call
> "ostree fsck" when *this* option is enabled - then "rpm -V .." checks would
> be still executed by default
> 3) do both 1) and 2), such that a) "rpm -V .." are not further run by
> default, b) "ostree fsck" is disabled by default, and c) the fsck is enabled
> by more appropriate(?) plugin option, not the generic one

As I understand a change in default behaviour shall need some bigger consensus, I will wait some time for Derrick for next opinion. With no feedback in a few weeks, I will propose PR for the 3rd option (which sounds the best to me as well).

(In reply to Pavel Moravec from comment #1)
> Nice investigation. Do I get it right the autofs / dpkg / flatpak / systemd
> "verify commands" are not required as a default behaviour on OCP?
> 

Correct, they are not needed/wanted for OCP. The autofs, dpkg (not in RHEL), and flatpak packages are not installed by default on RHEL CoreOS anyways. I most situations, I don't know that 'journalctl --verify' has much value either. 


> Keep in mind that --verify option also runs "rpm -V <pgklist>" whenever
> verify_packages variable is set by a plugin, which is:
> 
> $ grep " verify_packages " sos/report/plugins/*py
> sos/report/plugins/block.py:    verify_packages = ('util-linux',)
> sos/report/plugins/convert2rhel.py:    verify_packages = ('convert2rhel$',)
> sos/report/plugins/java.py:    verify_packages = ('java.*',)
> sos/report/plugins/kernel.py:    verify_packages = ('kernel$',)
> sos/report/plugins/nss.py:    verify_packages = ('nss.*',)
> sos/report/plugins/openssl.py:    verify_packages = ('openssl.*',)
> sos/report/plugins/pam.py:    verify_packages = ('pam_.*',)
> sos/report/plugins/perl.py:    verify_packages = ('perl.*',)
> sos/report/plugins/rpm.py:    verify_packages = ('rpm',)
> sos/report/plugins/system.py:    verify_packages = ('glibc', 'initscripts',
> 'zlib')
> $
> 

I think if we had concerns about the integrity of packages on a node, then we would direct the user to include the 'rpm.rpmva=on' option instead of checking this very small subset of packages with '--verify'. 


> I see three possible implementations:
> 1) remove --verify from RHOCP preset completely. This will a) stop calling
> the "rpm -V .." checks (is that intentional?) and b) stop calling "ostree
> fsck", from default behaviour
> 2) introduce a new ostree plugin option "fsck", disabled by default (enabled
> via "-k ostree.fsck=yes" / "--plugin-option ostree.fsck=yes") and call
> "ostree fsck" when *this* option is enabled - then "rpm -V .." checks would
> be still executed by default
> 3) do both 1) and 2), such that a) "rpm -V .." are not further run by
> default, b) "ostree fsck" is disabled by default, and c) the fsck is enabled
> by more appropriate(?) plugin option, not the generic one
> 
> 
> Please let us know what option is preferable.

Thanks for compiling this list. I like option 3. 

I would like to hear from @walters on how useful and/or hazardous 'ostree fsck' might be for everyday support use. I wonder if this is generally safe, something to avoid except in extreme circumstance, or maybe somewhere in between?

I raised https://github.com/sosreport/sos/pull/3147 in upstream.

Would you be able to verify the fix once we have a downstream build candidate ready (a matter of a week)?

Yes, I can test the RHEL sos build.

This bug has been fixed by errata https://access.redhat.com/errata/RHBA-2023:1571 .

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days