Bug 2216919

Summary: [UBI9] rsyslogd fails to start with "could not transfer the specified internal posix capabilities settings to the kernel, capng_apply=-5"
Product: Red Hat Enterprise Linux 9 Reporter: francois.poirotte
Component: rsyslogAssignee: Attila Lakatos <alakatos>
Status: CLOSED ERRATA QA Contact: Dalibor Pospíšil <dapospis>
Severity: medium Docs Contact:
Priority: high    
Version: 9.2CC: alakatos, ben.hart, billy.oconnell, cowboy, dapospis, davide, dhodovsk, francois.poirotte, ivanov17, jwboyer, lvrabec, rsroka, swa, udaykumar9
Target Milestone: rcKeywords: AutoVerified, Triaged, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: rsyslog-8.2102.0-117.el9 Doc Type: Bug Fix
Doc Text:
Cause: Rsyslog was not able to drop capabilities when executed as a normal user or in a containerized environment. Consequence: Rsyslog exited at startup. Fix: With the fix, if the process does not any capabilities, rsyslog will not try to drop them. Result: Rsyslog is able to start even if no capabilities are present.
Story Points: ---
Clone Of:
: 2225088 (view as bug list) Environment:
Last Closed: 2023-11-07 08:33:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2225088    

Description francois.poirotte 2023-06-23 08:46:58 UTC
Description of problem:

rsyslogd fails to start when run from a ubi9-minimal container started with podman/docker.

Version-Release number of selected component (if applicable):

* registry.access.redhat.com/ubi9/ubi-minimal:9.2
* rsyslog v8.2102.0-113.el9_2

How reproducible:

Start a new UBI 9 container, install the rsyslog package and try to interact with rsyslogd (a simple "rsyslogd -version" is enough to trigger the problem).

Steps to Reproduce:
1. podman run -it --rm registry.access.redhat.com/ubi9/ubi-minimal:9.2
2. microdnf install --nodocs --setopt install_weak_deps=0 -y rsyslog
3. rsyslogd -version

Actual results:

rsyslog internal message (3,-2455): could not transfer  the  specified  internal posix  capabilities settings to the kernel, capng_apply=-5
 [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/2455 ]


Expected results:

rsyslogd  8.2102.0-113.el9_2 (aka 2021.02) compiled with:
	PLATFORM:				x86_64-redhat-linux-gnu
[...]

Additional info:

This issue was initially reported against the rsyslog project (https://github.com/rsyslog/rsyslog/issues/5159).

It is caused by the patch in rsyslog-8.2102.0-rhbz2127404-libcap-ng.patch.
The patch resembles upstream commit https://github.com/rsyslog/rsyslog/commit/ea821b7b9d2bd0e9912ec405fa90c195ca3d2d5b but lacks support for the `libcapng.default` global option which could otherwise be used to ignore the error.

As a workaround, granting all capabilities to the container (podman run --cap-add ALL -it --rm registry.access.redhat.com/ubi9/ubi-minimal:9.2) makes the error go away, but this does not seem like an advisable solution.

Comment 1 Attila Lakatos 2023-06-23 13:56:49 UTC
Hi,

I've already started to work on this problem. I will attach the upstream PR. Thanks for your understanding.

Comment 5 udaykumar9 2023-07-04 18:58:44 UTC
Hi @alakatos 

I see the issue got fixed on Rsyslog side and this is the PR: https://github.com/rsyslog/rsyslog/pull/5166
It seems fix will be released as part of this milestone: https://github.com/rsyslog/rsyslog/milestone/62

Thanks,

Comment 8 Ben 2023-07-06 19:00:47 UTC
Wanted to chime in, encountered this problem in RHEL9 except the host OS itself.. not related to any container items. AAP 2.3 Hybrid controller, after a lengthy investigation looks like `dnf updarte` was ran which upgrades rsyslog from 8.2102.0-105.el9.x86_64 to 8.2102.0-113.el9.x86_64

Which then caused supervisord to fail starting `awx-rsyslogd` with: libcap-ng used by "/usr/sbin/rsyslogd" failed dropping bounding set due to not having CAP_SETPCAP in capng_apply`

Looking at the documentation for Supervisord, I added the following line to /etc/supervisord.d/tower.ini:
stderr_logfile=/var/log/supervisor/awx-rsyslogerr.log

Restarting the supervisord service gave me entries in that file:

`rsyslog internal message (3,-2455): could not transfer  the  specified  internal posix  capabilities settings to the kernel, capng_apply=-5
 [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/2455 ]`

As usual, rsyslogs error page was entirely useless. Luckily I found this Bugzilla which led me down the right path.

Comment 12 Attila Lakatos 2023-07-10 07:22:10 UTC
Hello,

I've already created a build but it seems like automation did not catch it for some reason. I had to trigger it manually.

Comment 22 Sebastien Wains 2023-08-02 19:21:02 UTC
Dealt with the same problem today on AAP 2.3/2.4 on an updated RHEL9.2 (rsyslog-8.2102.0-113.el9_2.x86_64).

An easy way to check the breakage is to try to run rsyslog manually under the awx account.

[root@aap23 rsyslog]# su awx -
bash-5.1$ rsyslogd -f /var/lib/awx/rsyslog/ -i /tmp/rsyslog.pid
conf.d/       rsyslog.conf  
bash-5.1$ rsyslogd -f /var/lib/awx/rsyslog/rsyslog.conf -i /tmp/rsyslog.pid
rsyslog internal message (3,-2455): could not transfer  the  specified  internal posix  capabilities settings to the kernel, capng_apply=-5
 [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/2455 ]

Also found mention about this problem here: https://unix.stackexchange.com/questions/747224/unable-to-run-rsyslogd-as-non-root-user-on-centos-stream-9


> Wanted to chime in, encountered this problem in RHEL9 except the host OS
> itself.. not related to any container items. AAP 2.3 Hybrid controller,
> after a lengthy investigation looks like `dnf updarte` was ran which
> upgrades rsyslog from 8.2102.0-105.el9.x86_64 to 8.2102.0-113.el9.x86_64
> 
> Which then caused supervisord to fail starting `awx-rsyslogd` with:
> libcap-ng used by "/usr/sbin/rsyslogd" failed dropping bounding set due to
> not having CAP_SETPCAP in capng_apply`
> 
> Looking at the documentation for Supervisord, I added the following line to
> /etc/supervisord.d/tower.ini:
> stderr_logfile=/var/log/supervisor/awx-rsyslogerr.log
> 
> Restarting the supervisord service gave me entries in that file:
> 
> `rsyslog internal message (3,-2455): could not transfer  the  specified 
> internal posix  capabilities settings to the kernel, capng_apply=-5
>  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/2455 ]`
> 
> As usual, rsyslogs error page was entirely useless. Luckily I found this
> Bugzilla which led me down the right path.

Comment 23 Attila Lakatos 2023-08-03 07:33:03 UTC
This will be fixed in rsyslog-8.2102.0-117.el9

Comment 24 ivanov17 2023-08-25 03:51:26 UTC
Hmm... I got the same error with the official fedora-minimal:38 container image. 

I tried to run both a rootless and rootfull container, the result is the same.

On Fedora, the package is newer than rsyslog-8.2102:

...
rsyslog                    x86_64 8.2306.0-1.fc38                     updates      2.7 MiB
...

rsyslogd: could not transfer the specified internal posix capabilities settings to the kernel, capng_apply=-5 [v8.2306.0-1.fc38 try https://www.rsyslog.com/e/2455 ]
rsyslogd: run failed with error -2455 (see rsyslog.h or try https://www.rsyslog.com/e/2455 to learn what that number means)

Do I understand correctly that this was fixed in upstream and that this fix should also be applied to Fedora packages?

This is my configuration file:

# cat /etc/rsyslog.conf
global(workDirectory="/var/lib/rsyslog")
global(privdrop.user.id="101")
global(privdrop.group.id="101")
global(internalmsg.severity="warn")
module(load="imuxsock")
module(load="omstdout")
module(load="immark")
input(type="imuxsock" Socket="/dev/log")
template(name="outfmt" type="list") {
    property(name="syslogtag")
    property(name="msg" spifno1stsp="on" )
    property(name="msg" droplastlf="on" )
    constant(value="\n")
}
*.* action(type="omstdout" template="outfmt")

Comment 25 Attila Lakatos 2023-08-25 07:05:15 UTC
(In reply to ivanov17 from comment #24)
> Hmm... I got the same error with the official fedora-minimal:38 container
> image. 
> 
> I tried to run both a rootless and rootfull container, the result is the
> same.
> 
> On Fedora, the package is newer than rsyslog-8.2102:
> 
> ...
> rsyslog                    x86_64 8.2306.0-1.fc38                    
> updates      2.7 MiB
> ...
> 
> rsyslogd: could not transfer the specified internal posix capabilities
> settings to the kernel, capng_apply=-5 [v8.2306.0-1.fc38 try
> https://www.rsyslog.com/e/2455 ]
> rsyslogd: run failed with error -2455 (see rsyslog.h or try
> https://www.rsyslog.com/e/2455 to learn what that number means)
> 
> Do I understand correctly that this was fixed in upstream and that this fix
> should also be applied to Fedora packages?
> 
> This is my configuration file:
> 
> # cat /etc/rsyslog.conf
> global(workDirectory="/var/lib/rsyslog")
> global(privdrop.user.id="101")
> global(privdrop.group.id="101")
> global(internalmsg.severity="warn")
> module(load="imuxsock")
> module(load="omstdout")
> module(load="immark")
> input(type="imuxsock" Socket="/dev/log")
> template(name="outfmt" type="list") {
>     property(name="syslogtag")
>     property(name="msg" spifno1stsp="on" )
>     property(name="msg" droplastlf="on" )
>     constant(value="\n")
> }
> *.* action(type="omstdout" template="outfmt")
-
That makes sense because this BZ is about RHEL and you are using Fedora. I will do a backport for F38<->rawhide.
You need upstream version 8.2308.0 to have this fixed.

Comment 26 udaykumar9 2023-08-29 10:59:10 UTC
quick question i tried using the image `registry.access.redhat.com/ubi9:latest`.

I see the rsyslog version specified as below:

[root@a01bad057b8b /]# dnf --showduplicates list rsyslog
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:00:07 ago on Tue Aug 29 10:55:30 2023.
Available Packages
rsyslog.x86_64                                                           8.2102.0-113.el9_2                                                            ubi-9-appstream-rpms
[root@a01bad057b8b /]

dnf is pointing to older rsyslog version. Is there a way i can point to `rsyslog-8.2102.0-117.el9`? Could you please let me know? 

Thanks,

Comment 27 billy.oconnell@ibm.com 2023-09-02 16:44:43 UTC
I am seeing the same thing that udaykumar9 is seeing.   The new package is not available when building:


0.397 This system is not registered with an entitlement server. You can use subscription-manager to register.
0.397
0.808 Red Hat Universal Base Image 9 (RPMs) - BaseOS  1.4 MB/s | 509 kB     00:00
1.457 Red Hat Universal Base Image 9 (RPMs) - AppStre 3.0 MB/s | 1.7 MB     00:00
2.025 Red Hat Universal Base Image 9 (RPMs) - CodeRea 684 kB/s | 198 kB     00:00
2.135 No match for argument: rsyslog-8.2102.0-117.el9

So, same question:  Is there a way i can point to `rsyslog-8.2102.0-117.el9`? Could you please let me know?

Thanks,
Billy

Comment 28 Attila Lakatos 2023-09-04 07:11:45 UTC
The new version of rsyslog exists in the AppStream RHEL-9 repository.
However, I am not sure which packages does UBI9 pull by default (from 9.2 or 9.3?). Once the package is "shipped", I think UBI9 will contain the appropriate version of rsyslog.

Comment 29 Attila Lakatos 2023-09-04 07:12:34 UTC
If it's based on 9.2, then you will need to wait for https://bugzilla.redhat.com/show_bug.cgi?id=2225088 to be verified.

Comment 31 errata-xmlrpc 2023-11-07 08:33:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rsyslog bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6444