Bug 1998924

I'm making a workaround PR for this in yarnpkg for now: https://src.fedoraproject.org/rpms/yarnpkg/pull-request/1

I am getting a lot of:

npm WARN old lockfile FetchError: request to https://registry.npmjs.org/yn failed, reason: getaddrinfo EAI_AGAIN registry.npmjs.org
npm WARN old lockfile     at ClientRequest.<anonymous> (/usr/lib/node_modules/npm/node_modules/minipass-fetch/lib/index.js:97:14)
npm WARN old lockfile     at ClientRequest.emit (node:events:394:28)
npm WARN old lockfile     at TLSSocket.socketErrorListener (node:_http_client:447:9)
npm WARN old lockfile     at TLSSocket.emit (node:events:406:35)
npm WARN old lockfile     at emitErrorNT (node:internal/streams/destroy:157:8)
npm WARN old lockfile     at emitErrorCloseNT (node:internal/streams/destroy:122:3)
npm WARN old lockfile     at processTicksAndRejections (node:internal/process/task_queues:83:21)
npm WARN old lockfile  Could not fetch metadata for yn.0 FetchError: request to https://registry.npmjs.org/yn failed, reason: getaddrinfo EAI_AGAIN registry.npmjs.org
npm WARN old lockfile     at ClientRequest.<anonymous> (/usr/lib/node_modules/npm/node_modules/minipass-fetch/lib/index.js:97:14)
npm WARN old lockfile     at ClientRequest.emit (node:events:394:28)
npm WARN old lockfile     at TLSSocket.socketErrorListener (node:_http_client:447:9)
npm WARN old lockfile     at TLSSocket.emit (node:events:406:35)
npm WARN old lockfile     at emitErrorNT (node:internal/streams/destroy:157:8)
npm WARN old lockfile     at emitErrorCloseNT (node:internal/streams/destroy:122:3)
npm WARN old lockfile     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
npm WARN old lockfile   code: 'EAI_AGAIN',
npm WARN old lockfile   errno: 'EAI_AGAIN',
npm WARN old lockfile   syscall: 'getaddrinfo',
npm WARN old lockfile   hostname: 'registry.npmjs.org',
npm WARN old lockfile   type: 'system'
npm WARN old lockfile }

Which makes it hard to reproduce, but hopefully the mockbuild will eventually move on.

So, the script has:

cd "$RPM_BUILD_ROOT"

# Large packages such as kernel can have thousands of executable files.
# We take care to not fork/exec thousands of "file"s and "grep"s,
# but run just two of them.
# (Take care to exclude filenames which would mangle "file" output).
find -executable -type f ! -path '*:*' ! -path $'*\n*' \
| file -N --mime-type -f - \
| grep -P ".+(?=: text/)" \
| {
... while ....
}



And I've checked that list on yarnpkg's buildroot. It does not contain /usr/lib/node_modules/yarn/bin/yarn.js at all and so it does not contain many others, like /usr/lib/node_modules/yarn/node_modules/uri-js/dist/es5/uri.all.js.map.


The culprit is in `grep -P ".+(?=: text/)"`: That grep filters out many valid use cases, such as:

./usr/lib/node_modules/yarn/bin/yarn.js: application/javascript
./usr/lib/node_modules/yarn/node_modules/uri-js/dist/es5/uri.all.js.map: application/json


So, the mime type detection here is too clever and all files not detected as "text/" are ignored completely, which exactly does not work for executable things, such as nodejs scripts.


I wonder whether we should "simplify" the text/binary detection by using `file --parameter bytes=8` -- my experiments show that this ideal amount of bytes to either detect application/octet-stream or text/plain. As a side effect, it could also make the script faster.

I am only afraid of the impact. This needs a copr impact check.

Look at what we did with eu-elfclass.  file --mime-type is just not very reliable.

If you want to rewrite shell paths, you should check if a shell path is present at the start of the script, and not rely on some fuzzy notion of MIME type.

> If you want to rewrite shell paths, you should check if a shell path is present...

We do, but we only check "text files" -- we need to distinguish between "executable text file without shebang" (we strip the executable bit) and "executable binary file" (we leave it alone).

I'll check out eu-elfclass and see what we do there. Thanks.

Sorry, typo: eu-elfclassify

It's just a C program that actually parses the file (with a real ELF parser) to determine what it's type is, rather than using the guesses that libmagic makes. (There are still heuristics, but that's to do with ELF, we're not cutting any corners.)

https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/145

I've started a thread on devel (and packaging) list: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/K3QCBUXYR6ZA34I777X6F2RYJKKECJLM/

(In reply to Miro Hrončok from comment #3)
> So, the script has:
> 
> cd "$RPM_BUILD_ROOT"
> 
> # Large packages such as kernel can have thousands of executable files.
> # We take care to not fork/exec thousands of "file"s and "grep"s,
> # but run just two of them.
> # (Take care to exclude filenames which would mangle "file" output).
> find -executable -type f ! -path '*:*' ! -path $'*\n*' \
> | file -N --mime-type -f - \
> | grep -P ".+(?=: text/)" \
> | {
> ... while ....
> }
> 
> 
> 
> And I've checked that list on yarnpkg's buildroot. It does not contain
> /usr/lib/node_modules/yarn/bin/yarn.js at all and so it does not contain
> many others, like
> /usr/lib/node_modules/yarn/node_modules/uri-js/dist/es5/uri.all.js.map.
> 
> 
> The culprit is in `grep -P ".+(?=: text/)"`: That grep filters out many
> valid use cases, such as:
> 
> ./usr/lib/node_modules/yarn/bin/yarn.js: application/javascript
> ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/es5/uri.all.js.map:
> application/json
> 
> 
> So, the mime type detection here is too clever and all files not detected as
> "text/" are ignored completely, which exactly does not work for executable
> things, such as nodejs scripts.

instead use file --mime-type you may use --mime-encoding or none option 

file /usr/lib/node_modules/yarn/bin/yarn.js  -b
Node.js script, ASCII text executable

file /usr/lib/node_modules/yarn/bin/yarn.js --mime-encoding -b
us-ascii

is mime-encoding that is not binary, I think

https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/163