Bug 2035802

Summary: libzstd-1.5.1-1.fc35.s390x completely breaks s390
Product: [Fedora] Fedora Reporter: Heiko Carstens <hca>
Component: zstdAssignee: Pádraig Brady <p>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 35CC: francois.rigault, frank, gabravier, igor.raits, jau, kevin, markus.falb, noloader, pbrobinson, p, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: s390x   
OS: Linux   
Fixed In Version: zstd-1.5.1-2.fc36 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-01-03 17:47:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 245418    

Description Heiko Carstens 2021-12-27 18:03:57 UTC
Description of problem:
The recent update of libzstd to 1.5.1-1 completely breaks s390. Updating the package results in a broken system.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Install Fedora 35 (minimal system)
2. Only update libzstd (dnf update libzstd)
3. reboot

Actual results:
System doesn't come up anymore. Tons of errors on the console.

Expected results:
System comes up :)

Comment 1 Jérôme Audu 2021-12-27 21:44:24 UTC
Same problem on armv7hl ! (aka totaly broken)

This avoid system to boot correctly.... reverting to libzstd-1.5.0-2.fc35 fix the issue...

The most visible issue begin with "systemd-journald.service" which is not able to start.... 
and since many (many) service depend on this, then break everything.... 

Please give us tips to help us to debug this issue!

Comment 2 François Rigault 2021-12-28 14:42:34 UTC

# execstack --query /usr/lib64/libzstd.so.1.5.0
- /usr/lib64/libzstd.so.1.5.0

# execstack --query /usr/lib64/libzstd.so.1.5.1
X /usr/lib64/libzstd.so.1.5.1

restarting systemd-networkd shows:
Dec 28 14:38:52 s390x systemd-networkd[11336]: /usr/lib/systemd/systemd-networkd: error while loading shared libraries: libzstd.so.1: cannot enable executable stack as shared object requires: Operation not permitted

fyi on x86_64:

# execstack --query /usr/lib64/libzstd.so.1.5.1
- /usr/lib64/libzstd.so.1.5.1

Comment 3 Zbigniew Jędrzejewski-Szmek 2021-12-28 14:46:19 UTC
Great sleuthing!
systemd-journald.service indeed has MemoryDenyWriteExecute=yes, so that'd explain the issue.

So... is the executable stack on purpose? If yes, we'll need to adjust systemd-journald.service
to allow that. If no, then libzstd need to be rebuild without it.

Comment 4 Zbigniew Jędrzejewski-Szmek 2021-12-28 16:48:12 UTC
Upstream has commit c67e07f34e89346ed4ee8b7f42972b679583ba96:
commit c67e07f34e89346ed4ee8b7f42972b679583ba96
Author: ko-zu <causeless>
Date:   Sat Nov 13 22:48:33 2021 +0900

    Remove executable flag from GNU_STACK section
    Putting stack marking into every assembly files is required to indicate
    that the stack does not need to be executable.
    Executable flag on stack conflicts with some security measures, Systemd
    MemoryDenyWriteExecute=yes for example.

diff --git lib/decompress/huf_decompress_amd64.S lib/decompress/huf_decompress_amd64.S
index 83e3d75658..97646a2e19 100644
--- lib/decompress/huf_decompress_amd64.S
+++ lib/decompress/huf_decompress_amd64.S
@@ -1,5 +1,12 @@
 #if !defined(HUF_DISABLE_ASM) && defined(__x86_64__)
+/* Stack marking
+ * ref: https://wiki.gentoo.org/wiki/Hardened/GNU_stack_quickstart
+ */
+#if defined(__linux__) && defined(__ELF__)
+.section .note.GNU-stack,"",%progbits

I pushed the following commit to dist-git:

    Disable amd64 assembly on non-intel architectures
    All builds would include an .S file which contains a pre-processor
    directive that makes the file effectively empty on non-amd64
    architectures. Objects built from assembly get an executable stack by
    default. The file has a stanza to make the stack non-executable, but
    on non-amd64 architectures this stanza is skipped together with the
    rest of the file. Nevertheless, it seems that even the empty file
    causes an executable stack to be created. To avoid the issue, do a
    build with the assembler file explicitly skipped.

zstd-1.5.1-2.fc36 is building now. Hopefully it'll fix the issue.

Comment 5 Zbigniew Jędrzejewski-Szmek 2021-12-28 17:08:01 UTC
*** Bug 2035812 has been marked as a duplicate of this bug. ***

Comment 6 Zbigniew Jędrzejewski-Szmek 2021-12-28 17:25:59 UTC
I think it'd be nicer to fix this by adjusting the build system to not include the .S file at all.
But it's a jungle of recursive makefiles that I don't want to touch… I'll leave the bug open
for followup.

Comment 7 Pádraig Brady 2021-12-29 03:01:03 UTC
Sorry for the trouble on this.
I see this issue was flagged as lint:

However the f35 build was pushed to stable before I had a chance to review.

I don't see the lint error on the new build on rawhide (though I'm not sure that lint was run on this revision bump).

In any case I've made an update available to the f35 branch,
where the problematic version has already hit stable:


Will monitor the lint on that update

Comment 8 Jérôme Audu 2021-12-29 09:22:07 UTC
zstd-1.5.1-2.fc35 fix boot issue on armv7hf ! Many thanks to all for this fix!

Comment 9 Frank Crawford 2021-12-29 11:42:15 UTC
Also fixed my issue on an RPI2 system.  Thanks for the quick fix.

Comment 10 Zbigniew Jędrzejewski-Szmek 2021-12-29 11:47:28 UTC
Based on the rpminspect report, an executable stack is present on: arm, i686, s390x.
My patch used '%ifarch %{ix86} x86_64', so it does NOT fix the problem on i686.
It should be changed to '%ifarch x86_64'…

As I wrote above, a proper solution would be to adjust the build system to skip the
asm file altogether when not supported on a given architecture. This would mean that
the compatibility check would have to be moved from CPP ifdefs to the build system logic.
This is a more complex change, but would give a much more robust solution.

Comment 11 Heiko Carstens 2021-12-29 15:23:30 UTC
FWIW, I can also confirm that libzstd-1.5.1-2.fc35 works on s390.
Thank you!

Comment 12 Pádraig Brady 2021-12-29 20:57:54 UTC
Yes good spot on the i686 issue still being present.
I don't know why rpminspect wasn't run for the new revision ?
Anyway I see a few solutions for this issue in general.

1. move the "non exec stack" directive outside of the ifdef in the asm file
2. adjust the build to exclude the asm file when appropriate
3. adjust the rpm build to explicitly only include asm file on x86_64
4. adjust the rpm build to add -Wl,-z,noexecstack to LDFLAGS
5. adjust the rpm build to remove the exec stack flag with `execstack -c`

1 or 2 would be good for upstream to do.
I've reported it upstream at: https://github.com/facebook/zstd/issues/2963

I'll adjust again so that the current solution we're using in fedora (3) is complete.
I.e. i686 build is adjusted to be also have a lib with non executable stack

Comment 13 Jeffrey Walton 2021-12-30 02:01:24 UTC
Now related: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=103863. I should have made the feature request a long time ago. The silent loss of no-exec stacks has bothered me for years.