Bug 1878530 - systemd has problems parsing capabilities of new kernel (Failed to parse bus message: Invalid argument)
Summary: systemd has problems parsing capabilities of new kernel (Failed to parse bus...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 31
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-13 20:21 UTC by Michael Riss
Modified: 2020-10-05 18:34 UTC (History)
11 users (show)

Fixed In Version: systemd-245.8-2.fc32 systemd-243.9-1.fc31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-23 17:12:52 UTC
Type: Bug


Attachments (Terms of Use)
new spec file (103.59 KB, text/plain)
2020-09-14 12:31 UTC, Michael Riss
no flags Details
microhttpd build patch (2.49 KB, patch)
2020-09-14 12:31 UTC, Michael Riss
no flags Details | Diff

Description Michael Riss 2020-09-13 20:21:46 UTC
Description of problem:
This seems to be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1853736 but now on Fedora 31.
With the new kernel 5.8.6 a

systemctl show -p CapabilityBoundingSet bash

results in a: Failed to parse bus message: Invalid argument

On the old kernel 5.7.15 the same command results in:

CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read

This malfunction inhibits using ansible with the systemd module, which is a severe problem for the ones affected - therefore the Severity: High classification. 

Version-Release number of selected component (if applicable):
- systemd-243.8-1.fc31
- kernel-5.8.6-101.fc31

How reproducible:

Steps to Reproduce:
1. Install
- systemd-243.8-1.fc31
- kernel-5.8.6-101.fc31

2. Run: systemctl show -p CapabilityBoundingSet bash

3. Optional:
Install and boot into kernel-5.7.15-100.fc31 and repeat for comparison.

Actual results:
"Failed to parse bus message: Invalid argument"

Expected results:
"CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read" 
or similar, depending on the actual capabilities of the current kernel

Additional info:
I tried to simply recompile the systemd srpm, but ran into compile problems on the way that I couldn't solve easily/today. So, the solution may be more complicated than in the other / duplicate case. Sorry.

Comment 1 Michael Riss 2020-09-14 11:04:34 UTC
The compile problem seems to be analog to https://bugs.archlinux.org/task/67208 - microhttpd changed its API.
Systemd seems to have a workaround for it: https://github.com/systemd/systemd/commit/d17eabb1052e7c8c432331a7a782845e36164f01

Comment 2 Michael Riss 2020-09-14 12:28:22 UTC
I now managed to apply the patch to systemd to build against the new version of microhttpd and it also solved the original capability problem. Nice!

I attach patch and changed spec file as I'm rusty again with the developer tools. Sorry. I will see that I push it there also, if I figure out the names of the tools again ... and if I still have a valid account on src.fedoraproject.org.

Comment 3 Michael Riss 2020-09-14 12:31:04 UTC
Created attachment 1714767 [details]
new spec file

Comment 4 Michael Riss 2020-09-14 12:31:41 UTC
Created attachment 1714768 [details]
microhttpd build patch

Comment 5 Michael Riss 2020-09-16 15:00:13 UTC
The call to test for the error is wrong, there is no "bash" service - sorry, I was tired - try some other, e.g.

systemctl show -p CapabilityBoundingSet httpd

Comment 6 Michael Riss 2020-09-16 22:28:41 UTC
I tested my patched version of the systemd with a full automated install of our production system. I created an extra repo with the new systemd packages and added this repo in the kickstart configuration, so it got used right from the start and during the initial installation of the system.
It went well, the problem with the error message of "systemctl show ..." went away, ansible works again and I didn't notice any regressions. 
I think that's about as good as I can test it on my end here. So I would like to hand it over to the experts on your side for further testing.
I opened a pull request https://src.fedoraproject.org/rpms/systemd/pull-request/35 containing my patch.
Please take a look at it and if I can do anything else or better, please tell me.

Comment 7 Michael Riss 2020-09-16 23:29:35 UTC
I just checked the packages that got generated by the pull request, they also work which indicates that the CI has the right kernel installed for this build - good!

Comment 8 Michael Riss 2020-09-16 23:31:36 UTC
Well, at least on the x86_64 architecture. I can't test the others: https://koji.fedoraproject.org/koji/taskinfo?taskID=51617092
Quite cool build system btw.

Comment 9 Michael Riss 2020-09-18 22:23:53 UTC
Question to the maintainers:

How is the situation?
Did you already look at this problem/fix/patch?
Do you see a chance that it gets accepted?
Or is it too late in the life cycle of Fedora 31 to release an update of such a central component?

I have some machines that need this fix and am wondering whether I should set up a private update repo for it.
Of course, if you accept and push this update tomorrow or some time soon there is no need for that.

Would be nice to hear from you, to know where I'm at; also when it's a: "Sorry, didn't have time to look at it yet."
That also gives me already some orientation.

Comment 10 Michael Riss 2020-09-18 22:57:19 UTC
Oops, I just realized something odd.
On https://koji.fedoraproject.org/koji/taskinfo?taskID=51617092 the top line is

Information for task build (f31, systemd-243.8-2.fc30.src.rpm)

It claims it did the build from a srpm for Fedora 30. But I pushed the branch for Fedora 31 and all the resulting rpms are for Fedora 31. There seems something off with the build system.

Comment 11 Fedora Update System 2020-09-20 13:20:31 UTC
FEDORA-2020-0d29e88946 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-0d29e88946

Comment 12 Fedora Update System 2020-09-20 13:22:41 UTC
FEDORA-2020-dc4f0fb907 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-dc4f0fb907

Comment 13 Fedora Update System 2020-09-20 23:55:25 UTC
FEDORA-2020-0d29e88946 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-0d29e88946`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-0d29e88946

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Fedora Update System 2020-09-21 00:39:20 UTC
FEDORA-2020-dc4f0fb907 has been pushed to the Fedora 31 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-dc4f0fb907`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-dc4f0fb907

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 15 Fedora Update System 2020-09-21 08:01:29 UTC
FEDORA-2020-0d29e88946 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-0d29e88946

Comment 16 Fedora Update System 2020-09-21 14:28:31 UTC
FEDORA-2020-0d29e88946 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-0d29e88946`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-0d29e88946

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Zbigniew Jędrzejewski-Szmek 2020-09-21 19:13:16 UTC
Michael, I forgot to thank you for the patches. In the end I didn't use them because
there were other patches to apply too, and it's easier to do this through upstream
stable branches. In the future, please consider opening a PR or issue under
https://github.com/systemd/systemd-stable/ when there's something that needs to
be quickly backported. Also, thank you for testing the update.

Comment 18 Michael Riss 2020-09-21 20:31:58 UTC
Hello Zbigniew,

no problem, I'm just glad the fix finds its way into the main distribution now and I do not have to run an own updates repo indefinitely. ;)

About reporting the problem upstream - in this case I had the feeling it was a Fedora 31 specific situation, because:
A. it just needed to get recompiled with the new kernel includes
B. one of the dependencies *on Fedora 31* got an upgrade with API-breaking changes.

So it was not really a systemd bug, reporting it at the systemd - repo ... they might ask me what they are supposed to do about build problems 
on Fedora 31. Therefore I opened the report here. But sure, should I encounter a bug within systemd itself it's faster to go to the source
and then flush the fix down to the distributions, true.

Comment 19 Zbigniew Jędrzejewski-Szmek 2020-09-21 20:59:12 UTC
This was a bit of a special case. This bug was fixed in three semi-independent ways:
be recompiling systemd against newer kernel headers, by adding a fallback define upstream,
and by changing how capability parsing is done so that the bug won't be repeated the next
time the kernel adds a new capability. So while the first two approaches are OK to do within
Fedora, the last was a big enough change that it merited going through the systemd-stable repo.
So... it's not all bugs that should go through upstream, just some.

Comment 20 Michael Riss 2020-09-21 22:10:50 UTC
Ok, now I see. I have to admit, I did not look into what happened upstream. Code changes to prevent the need to recompile sure go there.

Comment 21 Fedora Update System 2020-09-23 17:12:52 UTC
FEDORA-2020-0d29e88946 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 22 Fedora Update System 2020-10-05 18:34:53 UTC
FEDORA-2020-dc4f0fb907 has been pushed to the Fedora 31 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.