Bug 2021935

Summary: RHEL 8.5 /etc/cron.d/rear creates output & email with STDERR
Product: Red Hat Enterprise Linux 8 Reporter: James Hartsock <hartsjc>
Component: rearAssignee: Pavel Cahyna <pcahyna>
Status: CLOSED ERRATA QA Contact: David Jež <djez>
Severity: medium Docs Contact: Šárka Jana <sjanderk>
Priority: medium    
Version: 8.5CC: djez, mschwabe, oliver, ovasik, pcahyna
Target Milestone: rcKeywords: Regression, Triaged
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: rear-2.6-7.el8 Doc Type: Bug Fix
Doc Text:
.ReaR no longer displays a false error message about missing symlink targets Previously, ReaR displayed incorrect error messages about missing symlink targets for the `build` and `source` symlinks under `/usr/lib/modules/` when creating the rescue image. This situation was harmless, and you could safely ignore the error message. With this update, ReaR does not report a false error message about missing symlink targets in this situation.
Story Points: ---
Clone Of:
: 2119501 (view as bug list) Environment:
Last Closed: 2022-11-08 10:02:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2119501    
Bug Blocks:    

Description James Hartsock 2021-11-10 13:06:21 UTC
Description of problem:
RHEL 8.5 /etc/cron.d/rear creates output & email with STDERR

Version-Release number of selected component (if applicable):
rear-2.6-3.el8.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Install rear
2. wait for cron or run manullay
3.

Actual results:
# cat /etc/cron.d/rear
30 1 * * * root test -f /var/lib/rear/layout/disklayout.conf && /usr/sbin/rear checklayout || /usr/sbin/rear mkrescue

# test -f /var/lib/rear/layout/disklayout.conf && /usr/sbin/rear checklayout || /usr/sbin/rear mkrescue
updatedb
Symlink '/usr/lib/modules/4.18.0-348.el8.x86_64/source' -> '/usr/src/kernels/4.18.0-348.el8.x86_64' refers to a non-existing directory on the recovery system.
It will not be copied by default. You can include '/usr/src/kernels/4.18.0-348.el8.x86_64' via the 'COPY_AS_IS' configuration variable.
Symlink '/usr/lib/modules/4.18.0-348.el8.x86_64/build' -> '/usr/src/kernels/4.18.0-348.el8.x86_64' refers to a non-existing directory on the recovery system.
It will not be copied by default. You can include '/usr/src/kernels/4.18.0-348.el8.x86_64' via the 'COPY_AS_IS' configuration variable.


Expected results:

# test -f /var/lib/rear/layout/disklayout.conf && /usr/sbin/rear checklayout || /usr/sbin/rear mkrescue
#

Additional info:

# ll -d /usr/lib/modules/4.18.0-348.el8.x86_64/{build,source} /usr/src/kernels/4.18.0-348.el8.x86_64
lrwxrwxrwx.  1 root root   38 Oct  4 11:42 /usr/lib/modules/4.18.0-348.el8.x86_64/build -> /usr/src/kernels/4.18.0-348.el8.x86_64
lrwxrwxrwx.  1 root root    5 Oct  4 11:42 /usr/lib/modules/4.18.0-348.el8.x86_64/source -> build
drwxr-xr-x. 23 root root 4096 Nov  9 15:03 /usr/src/kernels/4.18.0-348.el8.x86_64

Comment 1 James Hartsock 2021-11-10 13:09:22 UTC
So probably need kernel-devel RPM installed to reproduce too

# rpm -qf /usr/lib/modules/4.18.0-348.el8.x86_64/{build,source} /usr/src/kernels/4.18.0-348.el8.x86_64
kernel-core-4.18.0-348.el8.x86_64
kernel-core-4.18.0-348.el8.x86_64
kernel-devel-4.18.0-348.el8.x86_64

Comment 2 James Hartsock 2021-11-10 13:14:35 UTC
From log...
2021-11-10 07:02:40.343886684 Including build/default/490_fix_broken_links.sh
/tmp/rear.xYg0IJsajKexszh/rootfs /usr/lib/modules
2021-11-10 07:02:40.414246180 Symlink '/usr/lib/modules/4.18.0-348.el8.x86_64/source' -> '/usr/src/kernels/4.18.0-348.el8.x86_64' refers to a non-existing directory on the recovery system.
2021-11-10 07:02:40.415361204 It will not be copied by default. You can include '/usr/src/kernels/4.18.0-348.el8.x86_64' via the 'COPY_AS_IS' configuration variable.
2021-11-10 07:02:40.417487261 Symlink '/usr/lib/modules/4.18.0-348.el8.x86_64/build' -> '/usr/src/kernels/4.18.0-348.el8.x86_64' refers to a non-existing directory on the recovery system.
2021-11-10 07:02:40.418621347 It will not be copied by default. You can include '/usr/src/kernels/4.18.0-348.el8.x86_64' via the 'COPY_AS_IS' configuration variable.

Comment 3 James Hartsock 2021-11-10 14:06:17 UTC
Perhaps should include  /usr/src/kernels/$KERNEL_VERSION/ in...

# tail -n 6 /usr/share/rear/rescue/GNU/Linux/240_kernel_modules.sh
COPY_AS_IS+=(
    /lib/modules/$KERNEL_VERSION/modules.*
    /etc/modules*
    /etc/modules-load?d
    /etc/modprobe*
)

---

Or maybe better yet exclude those links.  At first, thought the COPY_AS_IS_EXCLUDE+=, but that isn't what includes it from my view.
Appears this is from the (a default since ReaR 2.5 per build/GNU/Linux/400_copy_modules.sh):
  MODULES=( 'loaded_modules' )

# echo "MODULES=( 'loaded_modules' )" >> /etc/rear/local.conf 
# rm -rf /var/lib/rear/{layout,recovery}/* /var/log/rear/*
# test -f /var/lib/rear/layout/disklayout.conf && /usr/sbin/rear checklayout || /usr/sbin/rear mkrescue
#


So looks like this is introduced with https://access.redhat.com/errata/RHEA-2021:4344 that updated ReaR to 2.6 (from 2.4 version)
  rhel8.4/AppStream/Packages/rear-2.4-18.el8.x86_64.rpm
  rhel8.5/AppStream/Packages/rear-2.6-3.el8.x86_64.rpm

Comment 4 James Hartsock 2021-11-10 15:17:15 UTC
(In reply to James Hartsock from comment #3)
> Perhaps should include  /usr/src/kernels/$KERNEL_VERSION/ in...
> 
> # tail -n 6 /usr/share/rear/rescue/GNU/Linux/240_kernel_modules.sh
> COPY_AS_IS+=(
>     /lib/modules/$KERNEL_VERSION/modules.*
>     /etc/modules*
>     /etc/modules-load?d
>     /etc/modprobe*
> )
> 
> ---
> 
> Or maybe better yet exclude those links.  At first, thought the
> COPY_AS_IS_EXCLUDE+=, but that isn't what includes it from my view.
> Appears this is from the (a default since ReaR 2.5 per
> build/GNU/Linux/400_copy_modules.sh):
>   MODULES=( 'loaded_modules' )

Correction, default since 2.5 is 'all_modules' ... and I tried 'loaded_modules' as work-around.
# grep -m1 'all_modules' build/GNU/Linux/400_copy_modules.sh
    # Since ReaR version 2.5 we have MODULES=( 'all_modules' ) in default.conf


Sorry about that copy/paste mistake.

Comment 5 Pavel Cahyna 2021-11-10 18:29:03 UTC
Hello, the bug report is not really about the cron job, right? The actual problem is the newly appearing warning. The cron job is just what triggered it in your case.

Comment 7 Pavel Cahyna 2022-03-23 15:53:34 UTC
There is an upstream issue which is closed, but the fix is more a mitigation than a complete solution. We could backport it if not better solution is found.

Comment 8 James Hartsock 2022-03-23 16:15:03 UTC
So (In reply to Pavel Cahyna from comment #5)
> Hello, the bug report is not really about the cron job, right? The actual
> problem is the newly appearing warning. The cron job is just what triggered
> it in your case.

The issue gets noticed raised by customers because they monitor e-mail from cron jobs which create incidents.

So yes, ideally fix the error/warning the command produces is best because simply redirecting cron output or something masks not only this problem but any others that may be legitimate issues.

Comment 13 Pavel Cahyna 2022-08-17 10:17:53 UTC
For the record, the problem can show up in two ways. If package providing the symlink target (kernel-debug-devel or kernel-devel, matching the kernel version) are not installed, one gets messages like

Broken symlink '/usr/lib/modules/4.18.0-416.el8.x86_64/build' in recovery system because 'readlink' cannot determine its link target
Broken symlink '/usr/lib/modules/4.18.0-416.el8.x86_64/source' in recovery system because 'readlink' cannot determine its link target

If the target exist, one gets messages like the one in the original report:

Symlink '/usr/lib/modules/4.18.0-416.el8.x86_64/build' -> '/usr/src/kernels/4.18.0-416.el8.x86_64' refers to a non-existing directory on the recovery system.
It will not be copied by default. You can include '/usr/src/kernels/4.18.0-416.el8.x86_64' via the 'COPY_AS_IS' configuration variable.
Symlink '/usr/lib/modules/4.18.0-416.el8.x86_64/source' -> '/usr/src/kernels/4.18.0-416.el8.x86_64' refers to a non-existing directory on the recovery system.
It will not be copied by default. You can include '/usr/src/kernels/4.18.0-416.el8.x86_64' via the 'COPY_AS_IS' configuration variable.

Comment 25 errata-xmlrpc 2022-11-08 10:02:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rear bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7654