Bug 1974824

Summary: Latest zlib in ubi8-minimal image is causing failures on s390x
Product: Red Hat Enterprise Linux 8 Reporter: lzhan <lzhan>
Component: zlibAssignee: Lukas Javorsky <ljavorsk>
Status: CLOSED ERRATA QA Contact: RHEL CS Apps Subsystem QE <rhel-cs-apps-subsystem-qe>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.3CC: bugproxy, dafu, databases-maint, dhorak, jomiller, odubaj, panovotn, pkubat, praiskup, tstaudt
Target Milestone: betaKeywords: Triaged
Target Release: 8.5Flags: lzhan: needinfo+
pm-rhel: mirror+
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-19 06:41:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1916117    
Attachments:
Description Flags
File to specify the problematic image none

Description lzhan 2021-06-22 15:45:54 UTC
Created attachment 1793083 [details]
File to specify the problematic image

Description of problem:
Currently on s390x, build of Quarkus applications fails with the following error on the latest s2i java openshift images.

```
[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:1.11.7.Final-redhat-00009:build (default) on project todo-backend: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR] 	[error]: Build step io.quarkus.deployment.pkg.steps.JarResultBuildStep#buildRunnerJar threw an exception: java.lang.RuntimeException: java.util.zip.ZipException: invalid entry compressed size (expected 100 but got 102 bytes)
```
Early investigation seems to point to cve glib2 refresh on ubi8 Java S2I images causing some sort of zip bug on s390x. Doesn't seem to affect x86 and ppc64le at the moment. I will continue to investigate.

The cause of the issue seems to reside in libz.so.1.2.11 shipped from the ubi8-minimal base image. The openj9/openj9-11-rhel8:1.2-46.1622550125  build is based on a ubi8-minimal base image which contains a build of libz.so.1.2.11, which causes the failure. When manually replacing this libz.so with an older version, and injecting it to the openj9 image, the failure would be gone. Therefore, it doesn't seem to be related to openjdk11, not openj8, but seems to be a regression from base image, only on z/s390x.


Version-Release number of selected component (if applicable):
zlib-1.2.11-17.el8.s390x.rpm (possibly)

How reproducible:
Follow the steps below. We are testing Quarkus with the problematic s2i image defined in the yaml file. 

Steps to Reproduce:
1. checkout '1.11' branch of https://github.com/quarkus-qe/quarkus-openshift-test-suite. cd into the directory.
2. replace `external-applications/todo-demo-app/src/test/resources/openjdk-11.yaml` with the attached yaml
3. log in to your OCP cluster on IBM Z with `oc` and create a new project with `oc new-project test`
4. build the test suite libraries:
mvn clean install -DskipTests -DskipITs -B -V -pl app-metadata/deployment,app-metadata/runtime,common -fae
5. run the todo-demo-app tests:
mvn clean verify -B -V clean verify -pl app-metadata/deployment,app-metadata/runtime,common,external-applications/todo-demo-app -Dts.s2i.maven.remote.repository=http://10.0.139.54:8040/artifactory/quarkus-maven-repo-zip

Actual results:
It fails with follow reason:
```
[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:1.11.7.Final-redhat-00009:build (default) on project todo-backend: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR] 	[error]: Build step io.quarkus.deployment.pkg.steps.JarResultBuildStep#buildRunnerJar threw an exception: java.lang.RuntimeException: java.util.zip.ZipException: invalid entry compressed size (expected 100 but got 102 bytes)
```

Expected results:
It should pass.

Additional info:

Comment 1 Petr Kubat 2021-06-22 16:25:38 UTC
Hi Thomas, another issue that seems to have been hit as a result of the IBM Z patches that were part of the recent RHEL updates (BZ#1847438), can you take a look?

Comment 2 IBM Bug Proxy 2021-06-30 21:20:40 UTC
------- Comment From iii.com 2021-06-30 17:11 EDT-------
This looks like a quarkus bug to me.
I have proposed a fix to them: https://github.com/quarkusio/quarkus/pull/18283

Comment 3 Ondrej Dubaj 2021-07-02 08:44:35 UTC
Thank you very much for the fix!

Lei, can you please check if the fix solves your problem?

Thank you.

Comment 4 lzhan 2021-07-05 12:09:28 UTC
Ondrej, sure, this issue will be tracked/verified at https://issues.redhat.com/browse/QUARKUS-1140

Comment 5 Ondrej Dubaj 2021-07-19 05:23:05 UTC
Hello, is the fix already verified ? From the tracker is seems it should be.

Thank you.

Comment 6 lzhan 2021-07-19 11:57:21 UTC
Hi Ondrej, the fix will be verified for the next version release of RHBQ, which is around Sep this year.

Comment 7 Ondrej Dubaj 2021-09-15 05:55:37 UTC
Hi, I see that the resolution on the ticket is already Done, can we close this tracker? Thanks.

Comment 8 lzhan 2021-09-15 12:09:31 UTC
hi Ondrej, this fix was implemented but not yet verified. The verification will happen very soon. cc @

Comment 9 Ondrej Dubaj 2021-09-15 12:30:03 UTC
Thanks for your reply!

Comment 10 dafu 2021-10-18 18:14:40 UTC
Tested with Red Hat build of Quarkus 2.2.3.CR5 and this issue is no longer exists. Thanks for your help! We can resolve this issue.

Comment 11 Ondrej Dubaj 2021-10-19 06:41:53 UTC
Thank you for information! Closing this issue.