Bug 2018327
| Summary: | NPM from nodejs14x module segfaults on arm64 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Narayan Newton <nnewton> | ||||
| Component: | nodejs-14-module | Assignee: | Node.js maintainers <nodejs-maint> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | RHEL CS Apps Subsystem QE <rhel-cs-apps-subsystem-qe> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 8.4 | CC: | hhorak, jwboyer, midawson, rruss, zsvetlik | ||||
| Target Milestone: | rc | Keywords: | Triaged | ||||
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
||||
| Hardware: | aarch64 | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-11-10 19:03:56 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
I can confirm we've reproduced it on another M1 laptop that one colleague has, but we currently don't have this HW available for any further detailed debugging. It seems the npm crashes also on an Intel kernel when running the aarch64 container: "podman run --arch arm64 ..." However, it's hard to tell whether the crash is caused by the same issue, since we get a report of corrupted stack in both cases: $ podman run --arch arm64 -ti --rm -u 0 ubi8/nodejs-14 bash bash-4.4# dnf install gdb ...snipped... Installed: dnf-plugins-core-4.0.18-4.el8.noarch gc-7.6.4-3.el8.aarch64 gcc-gdb-plugin-8.4.1-1.el8.aarch64 gdb-8.2-15.el8.aarch64 gdb-headless-8.2-15.el8.aarch64 guile-5:2.0.14-7.el8.aarch64 libbabeltrace-1.5.4-2.el8.aarch64 libtool-ltdl-2.4.6-25.el8.aarch64 Complete! bash-4.4# npm install npm@latest qemu: uncaught target signal 11 (Segmentation fault) - core dumped4541ad7c775766 Segmentation fault (core dumped) bash-4.4# ls qemu_node_20211102-154700_119.core bash-4.4# gdb /usr/bin/node qemu_node_20211102-154700_119.core GNU gdb (GDB) Red Hat Enterprise Linux 8.2-15.el8 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "aarch64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/node...Reading symbols from .gnu_debugdata for /usr/bin/node...(no debugging symbols found)...done. (no debugging symbols found)...done. [New LWP 119] [New LWP 122] [New LWP 123] [New LWP 124] [New LWP 125] [New LWP 126] [New LWP 127] [New LWP 128] [New LWP 129] [New LWP 130] [New LWP 131] Core was generated by `���U ���U ���U ���U '. #0 0x0020005503cd2dbc in ?? () [Current thread is 1 (LWP 119)] Missing separate debuginfos, use: yum debuginfo-install nodejs-14.17.5-1.module+el8.4.0+12247+e2879e58.aarch64 (gdb) bt #0 0x0020005503cd2dbc in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) Narayan, just to be sure -- what HW and OS specifically are you using? (suspecting M1, but rather asking to be sure). Playing a bit with the nodejs-12 UBI8 container, I see it got broken somewhere between ubi8/nodejs-12:1-90.1626843814 (this is the last image that works) and ubi8/nodejs-12:1-95 (first that fails). I think we would see something similar in case of v14. It means the builds that worked are: nodejs-12.21.0-1.module+el8.3.0+10191+34fb5a07.aarch64 npm-6.14.11-1.12.21.0.1.module+el8.3.0+10191+34fb5a07.aarch64 and the builds that do not work any more are: nodejs-12.22.3-2.module+el8.4.0+11732+c668cc9f.aarch64 npm-6.14.13-1.12.22.3.2.module+el8.4.0+11732+c668cc9f.aarch64 I don't have capacity myself to bisect more at this point, but I at least went through the commits between 12.21.0 and 12.22.3 and except the obviously no-op changes, changes in tests or doc only, these are the upstream changes that do something with the code: https://github.com/nodejs/node/commit/0a35d49f56bdc789ade698595fa4fa09022a28f9 https://github.com/nodejs/node/commit/dfa04d90353e55b63474142859dabce725c493c5 https://github.com/nodejs/node/commit/eec75427811cd80ee0818b16d51ffd2a9a4e113b https://github.com/nodejs/node/commit/8ddea3f16d643cb745e8ce75362d9be233bb42e1 https://github.com/nodejs/node/commit/86f34ee18c36e2fcd3c057abc86704523ec940c8 https://github.com/nodejs/node/commit/f5692093d3aab880afd2fe8c994ebdec0dedcae6 https://github.com/nodejs/node/commit/18726259908377cc3fbc9a046ca69aef3fa37497 https://github.com/nodejs/node/commit/93dd799a864c420c1ce1f9d1bac2ccde7c5ede1d https://github.com/nodejs/node/commit/2da24ac302167ec346965d92dfe5dea3ee5c9424 https://github.com/nodejs/node/commit/7b0ed4ba9229f9f6bd2d877879b6f03f8d5dd4cc https://github.com/nodejs/node/commit/c947f1a0e161cbda057d23a1c90c37e40e71adc6 Nothing seems to be more suspicious than the others though, just from the quick look. Michael, is this something you or anybody else from the Runtimes folks, who are more familiar with the nodejs code, might take a closer look by any chance? Thanks for taking a look at this. You are correct the HW is an M1. Specifically M1 Max, OS X 12.0.1, Docker 4.1.1. A small note, I attempted to reproduce this on a T4G instance and failed to get it to reproduce via Podman or Docker. I am not really sure that is helpful information, but I wasn't expecting it. Honza nobody on our team has access to M1 hardware right now. Any chance you might be able to with access to M1 hardware? From the list of commits above the two that stand out are: V8 cherry pick (platform specific) - https://github.com/nodejs/node/commit/dfa04d90353e55b63474142859dabce725c493c5 npm upgrade - https://github.com/nodejs/node/commit/c947f1a0e161cbda057d23a1c90c37e40e71adc6 The first is a platform specific change, and the second is an upgrade to npm which is what is now crashing. It's not obvious why either one would be a problem, but probably worth seeing if removing either of those resolves the issue. I've checked the latest v14 upstream build inside the container: testuser@hhorak-mac ~ % docker run -ti --rm registry.access.redhat.com/ubi8/nodejs-14 bash bash-4.4$ curl -Lk https://nodejs.org/download/release/v14.18.1/node-v14.18.1-linux-arm64.tar.gz >node-v14.18.1-linux-arm64.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 32.5M 100 32.5M 0 0 17.3M 0 0:00:01 0:00:01 --:--:-- 17.3M bash-4.4$ tar -xf node-v14.18.1-linux-arm64.tar.gz What I've found out: ---------------------- 1) It fails when using npm from upstream, but interpreter from /usr/bin/node: bash-4.4$ node-v14.18.1-linux-arm64/bin/npm install npm@latest Segmentation fault.] - rollbackFailedOptional: verb npm-session d72a126e7a59db9e ---------------------- 2) It works when using interpreter from upstream build, no matter which npm is used: bash-4.4$ node-v14.18.1-linux-arm64/bin/node node-v14.18.1-linux-arm64/bin/npm install npm@latest [..................] - rollbackFailedOptional: verb npm-session c65c8ec8d7757131 npm WARN saveError ENOENT: no such file or directory, open '/opt/app-root/src/package.json' npm notice created a lockfile as package-lock.json. You should commit this file. npm WARN enoent ENOENT: no such file or directory, open '/opt/app-root/src/package.json' npm WARN src No description npm WARN src No repository field. npm WARN src No README data npm WARN src No license field. + npm.2 added 218 packages from 96 contributors and audited 218 packages in 3.748s 10 packages are looking for funding run `npm fund` for details found 9 moderate severity vulnerabilities run `npm audit fix` to fix them, or `npm audit` for details ---------------------- bash-4.4$ node-v14.18.1-linux-arm64/bin/node /usr/bin/npm install npm@latest npm WARN saveError ENOENT: no such file or directory, open '/opt/app-root/src/package.json' npm notice created a lockfile as package-lock.json. You should commit this file. npm WARN enoent ENOENT: no such file or directory, open '/opt/app-root/src/package.json' npm WARN src No description npm WARN src No repository field. npm WARN src No README data npm WARN src No license field. + npm.2 added 218 packages from 96 contributors and audited 218 packages in 2.897s 10 packages are looking for funding run `npm fund` for details found 9 moderate severity vulnerabilities run `npm audit fix` to fix them, or `npm audit` for details ---------------------- So, it means the npm alone might be ok, the problem is somewhere in node (or depended package that upstream node bundles). An initial look at the core file warning: core file may not match specified executable file. Reading symbols from /usr/bin/node...Reading symbols from /usr/lib/debug/usr/bin/node-12.22.5-1.module+el8.4.0+12242+af52a4c7.aarch64.debug...done. done. warning: Ignoring non-absolute filename: <linux-vdso.so.1> Missing separate debuginfo for linux-vdso.so.1 Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/a9/18a12c51b15e31ce583da8ed9caacc1f9c1892 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `npm '. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0020ffffb1b6ddbc in ?? () [Current thread is 1 (Thread 0xffffb1dca010 (LWP 23))] Missing separate debuginfos, use: yum debuginfo-install brotli-1.0.6-3.el8.aarch64 glibc-2.28-151.el8.aarch64 libgcc-8.4.1-1.el8.aarch64 libstdc++-8.4.1-1.el8.aarch64 openssl-libs-1.1.1g-15.el8_3.aarch64 zlib-1.2.11-17.el8.aarch64 (gdb) bt #0 0x0020ffffb1b6ddbc in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) It is interesting that it is asking for the debug info from brotli to be installed. Tyring the suggestion it telms me that it can't find those debuginfo's I tried starting an earlier container where things ran ok, then installed the latest nodejs-12 version with "dnf module install nodejs:12". I see the same behaviour so that would rule out other changes within the container itself being the root cause. Thanks to Michael Dawson and Milad Farazmand, it was found that the issue comes from the kernel bug that is already fixed upstream and is related to chacha20_poly1305_cipher: we believe it is related to https://github.com/torvalds/linux/commit/519a0d7e495a6d3ce62594e485aea2a3a4a2ca0a But this fix is likely not yet included in the linuxkit (https://github.com/linuxkit/linuxkit) that is used by docker VM on Mac. The change in RHEL builds of nodejs that triggered this was when we started to use system profiles for the crypto setting (BZ#1952915). That all means we're not able to fix this properly because linuxkit is not under our control, and reverting the system crypto policy is likely not worth it. What we can do is to remove the problematic cipher chacha20_poly1305_cipher from the list of default ciphers. That can be either done during the configuration already, or maybe better in the container only (because we don't see any issue outside of container running on Mac). It seems to me we can likely set an environment variable NODE_OPTIONS to use the system profile, but exclude the chacha20_poly1305 cipher only. That can be only done on aarch64 to limit the scope of this work-around. Meanwhile, the environment variable can be also set on container start: docker run -e NODE_OPTIONS=--tls-cipher-list='PROFILE=SYSTEM:!CHACHA20' ubi8/nodejs-12 For local tests on linux machine, this seems to reproduce the issue stabilly (behaves the same as on M1 laptop): podman run --arch arm64 -e NODE_OPTIONS=--tls-cipher-list='PROFILE=SYSTEM:!CHACHA20' ubi8/nodejs-12 With that, npm works as expected: $ podman run --arch arm64 -e NODE_OPTIONS=--tls-cipher-list='PROFILE=SYSTEM:!CHACHA20' -ti --rm ubi8/nodejs-12 bash Resolved "ubi8/nodejs-12" as an alias (/home/hhorak/.cache/containers/short-name-aliases.conf) Trying to pull registry.access.redhat.com/ubi8/nodejs-12:latest... Getting image source signatures Copying blob 92e6d5848234 skipped: already exists Copying blob 92960c396edc skipped: already exists Copying blob a43c17e99eba skipped: already exists Copying blob ceacd171a4f9 skipped: already exists Copying blob 2d5d59c100e7 skipped: already exists Copying config aef7d99e4f done Writing manifest to image destination Storing signatures bash-4.4$ npm install npm@latest npm WARN saveError ENOENT: no such file or directory, open '/opt/app-root/src/package.json' npm notice created a lockfile as package-lock.json. You should commit this file. npm WARN enoent ENOENT: no such file or directory, open '/opt/app-root/src/package.json' npm WARN src No description npm WARN src No repository field. npm WARN src No README data npm WARN src No license field. + npm.3 added 218 packages from 96 contributors and audited 218 packages in 16.839s 10 packages are looking for funding run `npm fund` for details found 9 moderate severity vulnerabilities run `npm audit fix` to fix them, or `npm audit` for details (In reply to Honza Horak from comment #23) > Thanks to Michael Dawson and Milad Farazmand, it was found that the issue > comes from the kernel bug that is already fixed upstream and is related to > chacha20_poly1305_cipher: > we believe it is related to > https://github.com/torvalds/linux/commit/ > 519a0d7e495a6d3ce62594e485aea2a3a4a2ca0a Wow, good work! > It seems to me we can likely set an environment variable NODE_OPTIONS to use > the system profile, but exclude the chacha20_poly1305 cipher only. That can > be only done on aarch64 to limit the scope of this work-around. +1 We've further discussed off-line, so sharing some more updates: The same issue that is in the kernel is also present in openssl, which seems to be enough for us to solve the issue (tested): reported as BZ#2021550 Since the fix in OpenSSL might take some time, we still need some work-around till then. The idea with NODE_OPTIONS should work, but due to the possible conflict with the NODE_OPTIONS set to something else on user's side, it seems better to set the default cipher set on aarch64 in RPM directly, so the new proposal is to do something like this: @@ -474,6 +474,16 @@ export LDFLAGS="%{build_ldflags}" # --openssl-use-def-ca-store #%endif +# There is a bug in openssl (RHBZ#2021550) that causes troubles with +# chacha20_poly1305_cipher on aarch64 (only when emulation is used), +# so removing this cipher temporarily, until the openssl issue is fixed. +# Related to RHBZ2018327 +%ifarch aarch64 +cipher_list='PROFILE=SYSTEM:!CHACHA20' +%else +cipher_list='PROFILE=SYSTEM' +%endif + %if %{with shared_brotli} %if %{with bootstrap} @@ -484,7 +494,7 @@ export LDFLAGS="%{build_ldflags}" --without-dtrace \ --with-intl=small-icu \ --openssl-use-def-ca-store \ - --openssl-default-cipher-list=PROFILE=SYSTEM + --openssl-default-cipher-list="$cipher_list" %else ./configure --prefix=%{_prefix} \ --shared-openssl \ @@ -496,7 +506,7 @@ export LDFLAGS="%{build_ldflags}" ... and similarly in the other configure calls. +1 from me on the proposed solution in the last comment. This is a twister -- with openssl-1.1.1k that was released in RHEL-8.5.0 today, the problem does not occur any more. This is likely thanks to this change:
--- openssl-1.1.1g/crypto/poly1305/asm/poly1305-armv8.pl 2021-11-10 19:45:13.752782656 +0100
+++ openssl-1.1.1k/crypto/poly1305/asm/poly1305-armv8.pl 2021-11-10 19:44:36.607639666 +0100
@@ -58,10 +58,14 @@ $code.=<<___;
// forward "declarations" are required for Apple
.extern OPENSSL_armcap_P
+.hidden OPENSSL_armcap_P
+.globl poly1305_init
+.hidden poly1305_init
.globl poly1305_blocks
+.hidden poly1305_blocks
.globl poly1305_emit
+.hidden poly1305_emit
-.globl poly1305_init
.type poly1305_init,%function
.align 5
poly1305_init:
@@ -861,8 +865,8 @@ poly1305_blocks_neon:
st1 {$ACC4}[0],[$ctx]
.Lno_data_neon:
- .inst 0xd50323bf // autiasp
ldr x29,[sp],#80
+ .inst 0xd50323bf // autiasp
ret
.size poly1305_blocks_neon,.-poly1305_blocks_neon
So, I believe we can close this bug now.
Please, make sure you're using the latest container images from the Red Hat Container Catalog and it should be working fine.
Confirming the latest image has fixed this for me. Amazing work tracking this down! Thank you. -N |
Created attachment 1838102 [details] Docker to segfault output Description of problem: NPM segfaults often on ARM64 machines on recent UBI8 images. Version-Release number of selected component (if applicable): npm-1:6.14.14-1.14.17.5.1.module+el8.4.0+12247+e2879e58.aarch64 How reproducible: Always Steps to Reproduce: 1. Run bash in a ubi8 container, enable nodejs 14.x module 2. Install nodejs 3. Run npm install -g npm@latest Actual results: NPM segfaults Expected results: NPM attempts to update itself Additional info: There are other npm commands that segfault, this was just the most reproducible. If you install nodejs from nodesource the segfault does not occur. If you run the same set of commands on x86, the segfault does not occur.