Bug 2216919

Summary: [UBI9] rsyslogd fails to start with "could not transfer the specified internal posix capabilities settings to the kernel, capng_apply=-5"
Product: Red Hat Enterprise Linux 9 Reporter: francois.poirotte
Component: rsyslogAssignee: Attila Lakatos <alakatos>
Status: VERIFIED --- QA Contact: Dalibor Pospíšil <dapospis>
Severity: medium Docs Contact:
Priority: high    
Version: 9.2CC: alakatos, ben.hart, cowboy, dapospis, davide, dhodovsk, jwboyer, lvrabec, rsroka, swa, udaykumar9
Target Milestone: rcKeywords: AutoVerified, Triaged, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: rsyslog-8.2102.0-117.el9 Doc Type: Bug Fix
Doc Text:
Cause: Rsyslog was not able to drop capabilities when executed as a normal user or in a containerized environment. Consequence: Rsyslog exited at startup. Fix: With the fix, if the process does not any capabilities, rsyslog will not try to drop them. Result: Rsyslog is able to start even if no capabilities are present.
Story Points: ---
Clone Of:
: 2225088 (view as bug list) Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2225088    

Description francois.poirotte 2023-06-23 08:46:58 UTC
Description of problem:

rsyslogd fails to start when run from a ubi9-minimal container started with podman/docker.

Version-Release number of selected component (if applicable):

* registry.access.redhat.com/ubi9/ubi-minimal:9.2
* rsyslog v8.2102.0-113.el9_2

How reproducible:

Start a new UBI 9 container, install the rsyslog package and try to interact with rsyslogd (a simple "rsyslogd -version" is enough to trigger the problem).

Steps to Reproduce:
1. podman run -it --rm registry.access.redhat.com/ubi9/ubi-minimal:9.2
2. microdnf install --nodocs --setopt install_weak_deps=0 -y rsyslog
3. rsyslogd -version

Actual results:

rsyslog internal message (3,-2455): could not transfer  the  specified  internal posix  capabilities settings to the kernel, capng_apply=-5
 [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/2455 ]


Expected results:

rsyslogd  8.2102.0-113.el9_2 (aka 2021.02) compiled with:
	PLATFORM:				x86_64-redhat-linux-gnu
[...]

Additional info:

This issue was initially reported against the rsyslog project (https://github.com/rsyslog/rsyslog/issues/5159).

It is caused by the patch in rsyslog-8.2102.0-rhbz2127404-libcap-ng.patch.
The patch resembles upstream commit https://github.com/rsyslog/rsyslog/commit/ea821b7b9d2bd0e9912ec405fa90c195ca3d2d5b but lacks support for the `libcapng.default` global option which could otherwise be used to ignore the error.

As a workaround, granting all capabilities to the container (podman run --cap-add ALL -it --rm registry.access.redhat.com/ubi9/ubi-minimal:9.2) makes the error go away, but this does not seem like an advisable solution.

Comment 1 Attila Lakatos 2023-06-23 13:56:49 UTC
Hi,

I've already started to work on this problem. I will attach the upstream PR. Thanks for your understanding.

Comment 5 udaykumar9 2023-07-04 18:58:44 UTC
Hi @alakatos 

I see the issue got fixed on Rsyslog side and this is the PR: https://github.com/rsyslog/rsyslog/pull/5166
It seems fix will be released as part of this milestone: https://github.com/rsyslog/rsyslog/milestone/62

Thanks,

Comment 8 Ben 2023-07-06 19:00:47 UTC
Wanted to chime in, encountered this problem in RHEL9 except the host OS itself.. not related to any container items. AAP 2.3 Hybrid controller, after a lengthy investigation looks like `dnf updarte` was ran which upgrades rsyslog from 8.2102.0-105.el9.x86_64 to 8.2102.0-113.el9.x86_64

Which then caused supervisord to fail starting `awx-rsyslogd` with: libcap-ng used by "/usr/sbin/rsyslogd" failed dropping bounding set due to not having CAP_SETPCAP in capng_apply`

Looking at the documentation for Supervisord, I added the following line to /etc/supervisord.d/tower.ini:
stderr_logfile=/var/log/supervisor/awx-rsyslogerr.log

Restarting the supervisord service gave me entries in that file:

`rsyslog internal message (3,-2455): could not transfer  the  specified  internal posix  capabilities settings to the kernel, capng_apply=-5
 [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/2455 ]`

As usual, rsyslogs error page was entirely useless. Luckily I found this Bugzilla which led me down the right path.

Comment 12 Attila Lakatos 2023-07-10 07:22:10 UTC
Hello,

I've already created a build but it seems like automation did not catch it for some reason. I had to trigger it manually.

Comment 22 Sebastien Wains 2023-08-02 19:21:02 UTC
Dealt with the same problem today on AAP 2.3/2.4 on an updated RHEL9.2 (rsyslog-8.2102.0-113.el9_2.x86_64).

An easy way to check the breakage is to try to run rsyslog manually under the awx account.

[root@aap23 rsyslog]# su awx -
bash-5.1$ rsyslogd -f /var/lib/awx/rsyslog/ -i /tmp/rsyslog.pid
conf.d/       rsyslog.conf  
bash-5.1$ rsyslogd -f /var/lib/awx/rsyslog/rsyslog.conf -i /tmp/rsyslog.pid
rsyslog internal message (3,-2455): could not transfer  the  specified  internal posix  capabilities settings to the kernel, capng_apply=-5
 [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/2455 ]

Also found mention about this problem here: https://unix.stackexchange.com/questions/747224/unable-to-run-rsyslogd-as-non-root-user-on-centos-stream-9


> Wanted to chime in, encountered this problem in RHEL9 except the host OS
> itself.. not related to any container items. AAP 2.3 Hybrid controller,
> after a lengthy investigation looks like `dnf updarte` was ran which
> upgrades rsyslog from 8.2102.0-105.el9.x86_64 to 8.2102.0-113.el9.x86_64
> 
> Which then caused supervisord to fail starting `awx-rsyslogd` with:
> libcap-ng used by "/usr/sbin/rsyslogd" failed dropping bounding set due to
> not having CAP_SETPCAP in capng_apply`
> 
> Looking at the documentation for Supervisord, I added the following line to
> /etc/supervisord.d/tower.ini:
> stderr_logfile=/var/log/supervisor/awx-rsyslogerr.log
> 
> Restarting the supervisord service gave me entries in that file:
> 
> `rsyslog internal message (3,-2455): could not transfer  the  specified 
> internal posix  capabilities settings to the kernel, capng_apply=-5
>  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/2455 ]`
> 
> As usual, rsyslogs error page was entirely useless. Luckily I found this
> Bugzilla which led me down the right path.

Comment 23 Attila Lakatos 2023-08-03 07:33:03 UTC
This will be fixed in rsyslog-8.2102.0-117.el9