Bug 1581343 - file-5.33 identification of gstreamer1 plugins changed
Summary: file-5.33 identification of gstreamer1 plugins changed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: file
Version: 28
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kamil Dudka
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: PrioritizedBug
Depends On:
Blocks: 1581325
TreeView+ depends on / blocked
 
Reported: 2018-05-22 14:31 UTC by Rex Dieter
Modified: 2018-05-27 19:31 UTC (History)
5 users (show)

Fixed In Version: file-5.33-5.fc29 file-5.33-5.fc28
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-27 19:31:23 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1570246 0 unspecified CLOSED glibc: When built with file 5.33, valgrind stops working due to RPM ELF processing [Fedora] 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1581224 0 unspecified CLOSED rpm: Adjustments in ELF processing for file 5.33 output change 2021-02-22 00:41:40 UTC

Internal Links: 1570246 1581224

Description Rex Dieter 2018-05-22 14:31:38 UTC
Since file-5.33 landed, gstreamer1 plugin provides are no longer being generated, see bug #1581325

I tracked it down to a behavior change in file-5.33, using one example,

$ rpm -q file
file-5.32-3.fc28.x86_64
$ file -L /usr/lib64/gstreamer-1.0/libgstgtk.so
/usr/lib64/gstreamer-1.0/libgstgtk.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=36571638e503d65491eb210008f00ebc052d9ade, stripped

$ rpm -q file
file-5.33-2.fc28.x86_64
$ file -L /usr/lib64/gstreamer-1.0/libgstgtk.so
/usr/lib64/gstreamer-1.0/libgstgtk.so: ELF 64-bit LSB pie executable x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=36571638e503d65491eb210008f00ebc052d9ade, stripped


Historically, gstreamer1.prov includes the snippet:
solist=$(echo $filelist | grep "libgst" | \
        xargs file -L 2>/dev/null | grep "ELF.*shared object" | cut -d: -f1 )

to identify plugins to operate on.  This now returns an empty list due to the behavior change I highlighted

Comment 1 Kamil Dudka 2018-05-22 14:48:19 UTC
This looks similar to bug 1570246 comment #11.  Same question here: Is the output of file-5.33 technically incorrect?

Comment 2 Florian Weimer 2018-05-22 14:53:54 UTC
The identification as PIE executable is a bit dubious, too.  If there's a PT_DYNAMIC segment but no PT_INTERP segment, it's a dynamic object without a program interpreter, and that really isn't an executable by any stretch.

Comment 3 Kamil Dudka 2018-05-22 15:20:03 UTC
Then we should probably revert the change in file-libs.  But at least with ld.so, I was able to execute the binary:

$ /lib64/ld-linux-x86-64.so.2
Usage: ld.so [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
You have invoked `ld.so', the helper program for shared library executables.
[...]

Comment 4 Kamil Dudka 2018-05-22 15:23:13 UTC
The funny thing is that we had bug #1296868 with exactly opposite request ;-)

Comment 5 Florian Weimer 2018-05-22 15:27:14 UTC
(In reply to Kamil Dudka from comment #3)
> Then we should probably revert the change in file-libs.  But at least with
> ld.so, I was able to execute the binary:
> 
> $ /lib64/ld-linux-x86-64.so.2
> Usage: ld.so [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
> You have invoked `ld.so', the helper program for shared library executables.
> [...]

Yes, but ld.so is a special case (PT_DYNAMIC header, but statically linked PIE).  It is difficult to tell this from the usual DSO.

Comment 6 Rex Dieter 2018-05-22 15:28:28 UTC
Thanks for feedback, in the meantime nominating PrioritizedBug

Comment 7 Rex Dieter 2018-05-22 15:41:30 UTC
In addition to (or altnernatively to) restoring 'file' behavior, I'm open to any advice on alternatives on how best to identify loadable plugins vs shared libraries (what the gstreamer check is trying to do)

Comment 8 Kamil Dudka 2018-05-22 16:11:10 UTC
It seems to be caused by this commit:

https://github.com/file/file/commit/FILE5_32-87-g6876ebad

I will revert it and notify upstream about the breakage.

Comment 9 Kamil Dudka 2018-05-22 16:35:23 UTC
I have pushed the following commit to fix this bug:

https://src.fedoraproject.org/cgit/rpms/file.git/commit/?id=87dddb46

Comment 10 Fedora Update System 2018-05-22 16:43:19 UTC
file-5.33-4.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-d1425c85a7

Comment 12 Fedora Update System 2018-05-23 17:37:24 UTC
file-5.33-4.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d1425c85a7

Comment 13 Kamil Dudka 2018-05-24 11:15:25 UTC
This is the response I got from file's upstream:

On Wednesday, May 23, 2018 6:18:04 PM CEST Christos Zoulas wrote:
> On May 23,  2:22pm, kdudka (Kamil Dudka) wrote:
> -- Subject: Re: file: multiple bugs in Fedora triggered by 'pie executable'
> | Yes, latest upstream snapshot prints a comma that file-5.33 did not. 
> | Still the problem remains that shared libraries are classified as pie
> | executables.
> That is a problem :-) So that we are all clear as to what's going on:
>
> 1. Shared objects (pie executables and shared libraries) use the same
> magic number, so they can't be told apart.
> 2. Many suggestions have been made on how to fix the problem, (check
> if they have an intepreter (breaks for static pie and glibc), check
> for entry point location (too fragile and difficult to code), etc.
> The jist of it is that nothing really works. Depending on the name
> to be libfoo.so simply does not work in the general case.
> 3. People kept asking for file(1) to differentiate. Reading in POSIX,
> file should say "executable" if the file is executable.
> http://pubs.opengroup.org/onlinepubs/007904975/utilities/file.html
> (STDOUT section in the table). So I've changed file just to do that.
>
> Are you installing your shared libraries with the execute bit set?
> If you do, then yes, they are pie executables. If you don't believe
> me, try to run them. Yes, most of them core-dump (so do buggy
> programs), but other's like glibc, or ld-linux-x86-64.so.2 work.
> If the kernel believes that they are PIE executables and
> attempts to execute them, then who is file(1) to disagree with the
> kernel :-)
>
> Now the real question is why do linux distributions install *some*
> of their shared libraries executable and others not? (I know the
> answer :-)
>
> Best,
>
> christos

Florian, could you please comment on that?  Is there anything better that upstream can use to distinguish shared libraries from dynamically linked executables?

Comment 14 Florian Weimer 2018-05-24 11:30:13 UTC
I would have expected that you talk to gstreamer upstream why they need to use file to filter things.

Admittedly, it's really hard to tell the various ELF artifacts apart.  Current binutils adds DF_1_PIE to DT_FLAGS_1 for a PIE executable, so that would be a pretty strong indicator that something is actually an executable.  I would probably treat anything of type ET_DYN without the flag as a shared object.  That will give some false negatives, but for Fedora with the frequent mass rebuilds, it should be fairly accurate.

Comment 15 Kamil Dudka 2018-05-24 11:53:33 UTC
Thanks!  I will forward your reply to file's upstream.

The actual problem that prevented the new rule in file-5.33 from doing the right thing on Fedora is that all shared libraries have the executable bit set.  Could you please clarify why the executable bits are needed on all shared libraries on Fedora?

They do not seem to be needed on Ubuntu, for example.

Comment 16 Florian Weimer 2018-05-24 11:57:57 UTC
(In reply to Kamil Dudka from comment #15)
> Thanks!  I will forward your reply to file's upstream.
> 
> The actual problem that prevented the new rule in file-5.33 from doing the
> right thing on Fedora is that all shared libraries have the executable bit
> set.  Could you please clarify why the executable bits are needed on all
> shared libraries on Fedora?

This could be something related to SELinux: Setting the x bit is an auditable event, and you can probably enforce that all machine code comes from files with the x bit using SELinux.

I know that Steve Grubb suggested to use the x bit on even more things.

Comment 17 Rex Dieter 2018-05-24 12:09:02 UTC
RE: comment #14

gstreamer autoprovides script needs to be able to differentiate between (gstreamer) plugins and libraries, that's what 'file' is used for in this context.

Gory details,
https://src.fedoraproject.org/rpms/gstreamer/blob/master/f/gstreamer.prov

Suggestions welcome for anything that's better than the status quo (see linked bug #1581325 )

Comment 18 Kamil Dudka 2018-05-24 12:15:37 UTC
Do all the gstreamer plug-ins use a common API/ABI?

If yes, what prevents you from using readelf to examine the symbols of interest?

Comment 19 Rex Dieter 2018-05-24 12:16:46 UTC
For gstreamer improvements, please comment in the gstreamer bug (bug #1581325), not all gstreamer maintainers are watching this one.

Comment 20 Steve Grubb 2018-05-24 13:21:36 UTC
The executable bit should never figure into what a file might be. I would say that the heuristics is wrong.

Comment 21 Fedora Update System 2018-05-24 13:57:23 UTC
file-5.33-4.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

Comment 22 Kamil Dudka 2018-05-24 14:26:50 UTC
Reopening per bug #1570246 comment #15.

Comment 23 Kamil Dudka 2018-05-24 14:47:41 UTC
follow-up commit:

https://src.fedoraproject.org/cgit/rpms/file.git/commit/?id=2ac456e8

Comment 24 Fedora Update System 2018-05-24 14:49:16 UTC
file-5.33-5.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-a269744e01

Comment 25 Fedora Update System 2018-05-25 18:44:30 UTC
file-5.33-5.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-a269744e01

Comment 26 Fedora Update System 2018-05-27 19:31:23 UTC
file-5.33-5.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.