Bug 1903531
| Summary: | ftype detection is not working | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Renaud Métrich <rmetrich> | ||||||||||||
| Component: | file | Assignee: | Vincent Mihalkovič <vmihalko> | ||||||||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Karel Volný <kvolny> | ||||||||||||
| Severity: | high | Docs Contact: | |||||||||||||
| Priority: | high | ||||||||||||||
| Version: | 8.3 | CC: | dapospis, fsumsal, kdudka, kvolny, lvrabec, rsroka, sgrubb, thoger, tscherf, vmihalko | ||||||||||||
| Target Milestone: | rc | Keywords: | Triaged, ZStream | ||||||||||||
| Target Release: | 8.0 | Flags: | pm-rhel:
mirror+
|
||||||||||||
| Hardware: | All | ||||||||||||||
| OS: | Linux | ||||||||||||||
| Whiteboard: | |||||||||||||||
| Fixed In Version: | file-5.33-17.el8 | Doc Type: | If docs needed, set a value | ||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||
| Clone Of: | |||||||||||||||
| : | 1905906 1905907 1905908 (view as bug list) | Environment: | |||||||||||||
| Last Closed: | 2021-06-29 15:09:54 UTC | Type: | Bug | ||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||
| Documentation: | --- | CRM: | |||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
| Embargoed: | |||||||||||||||
| Bug Depends On: | |||||||||||||||
| Bug Blocks: | 1905906, 1905907, 1905908 | ||||||||||||||
| Attachments: |
|
||||||||||||||
It appears that this issue is libmagic. RHEL 8 has 5.33 and Fedora is 5.39. Detection is fine on Fedora. Created attachment 1740047 [details]
patch 1 for file
Created attachment 1740048 [details]
patch 2 for file
Created attachment 1740049 [details]
patch 3 for file
Created attachment 1740050 [details]
patch 4 for file
Created attachment 1740051 [details]
patch 5 for file
Could you please suggest a set of files where libmagic's result is currently wrong and is going to be fixed by the patchset in question? The first three fix the reported problem. But then there is the wider problem where libmagic doesn't understand our packaging of python. There are a number of files in /usr/bin/ that have platform python. You can grep for them. The fifth patch takes care of identifying other languages that libmagic doesn't identify. But we can script finding things like this:
DIRS="/bin /sbin /usr/bin /usr/sbin /usr/libexec /lib /lib64 /usr/lib /usr/lib64 boot"
tempfile=`mktemp /tmp/plain.XXXXXX`
FOUND=0
for d in $DIRS
do
if [ -h $d ] ; then
continue
fi
find $d -type f -perm /0111 -print >> $tempfile 2>/dev/null
while read f
do
testf=`file --mime-type $f | grep text\/plain`
if [ x"$testf" != x ] ; then
first=`head -n 1 $f`
echo -e "$f\t$first"
fi
done < $tempfile
truncate -s 0 $tempfile
done
rm -f $tempfile
Nothing executable should have text/plain
Hi @sgrubb, I tested all five patches and the recognition (of regular files) works correctly. The patches are safe for RHEL-8 from my point of perspective. Thank you for them! Shall I send patch 4 and patch 5 to upstream? You can send any parts of patch 4 and 5 that are not upstream already to upstream. I'm pretty sure I have sent everything but platform-python because that seems to be a RHEL8 only item. But upstream may not have integrated everything I sent. For example, they don't have ocaml byte code detection in the latest release. unfortunately, the fix is incomplete - file fails to recognise Python in all mentioned cases (specifically when there is one space) (see also the current zstream which has the same fix applied) :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: Setup :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: file-5.33-17.el8.x86_64 :: [ 09:02:35 ] :: [ PASS ] :: Checking for the presence of file rpm :: [ 09:02:35 ] :: [ LOG ] :: Package versions: :: [ 09:02:35 ] :: [ LOG ] :: file-5.33-17.el8.x86_64 :: [ 09:02:35 ] :: [ BEGIN ] :: Creating tmp directory :: actually running 'TmpDir=$(mktemp -d)' :: [ 09:02:35 ] :: [ PASS ] :: Creating tmp directory (Expected 0, got 0) :: [ 09:02:35 ] :: [ BEGIN ] :: Running 'pushd /tmp/tmp.TCMNXvCLyH' /tmp/tmp.TCMNXvCLyH /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working :: [ 09:02:35 ] :: [ PASS ] :: Command 'pushd /tmp/tmp.TCMNXvCLyH' (Expected 0, got 0) :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: Duration: 0s :: Assertions: 3 good, 0 bad :: RESULT: PASS (Setup) :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: Test :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ 09:02:35 ] :: [ BEGIN ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test0.py :: actually running 'file /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test0.py' /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test0.py: Python script, ASCII text executable :: [ 09:02:35 ] :: [ PASS ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test0.py (Expected 0, got 0) :: [ 09:02:35 ] :: [ PASS ] :: File '/var/tmp/rlRun_LOG.csrfo0O5' should contain 'Python script' :: [ 09:02:35 ] :: [ BEGIN ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test1.py :: actually running 'file /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test1.py' /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test1.py: a /usr/bin/python3 script, ASCII text executable :: [ 09:02:35 ] :: [ PASS ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test1.py (Expected 0, got 0) :: [ 09:02:35 ] :: [ FAIL ] :: File '/var/tmp/rlRun_LOG.2yxGMHXO' should contain 'Python script' :: [ 09:02:35 ] :: [ BEGIN ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test2.py :: actually running 'file /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test2.py' /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test2.py: Python script, ASCII text executable :: [ 09:02:35 ] :: [ PASS ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test2.py (Expected 0, got 0) :: [ 09:02:35 ] :: [ PASS ] :: File '/var/tmp/rlRun_LOG.AvCmcpMx' should contain 'Python script' :: [ 09:02:35 ] :: [ BEGIN ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test2t.py :: actually running 'file /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test2t.py' /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test2t.py: Python script, ASCII text executable :: [ 09:02:35 ] :: [ PASS ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test2t.py (Expected 0, got 0) :: [ 09:02:35 ] :: [ PASS ] :: File '/var/tmp/rlRun_LOG.HRFctL2F' should contain 'Python script' :: [ 09:02:35 ] :: [ BEGIN ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test3.py :: actually running 'file /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test3.py' /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test3.py: Python script, ASCII text executable :: [ 09:02:35 ] :: [ PASS ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test3.py (Expected 0, got 0) :: [ 09:02:35 ] :: [ PASS ] :: File '/var/tmp/rlRun_LOG.0IcxCFbp' should contain 'Python script' :: [ 09:02:35 ] :: [ BEGIN ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/testp.py :: actually running 'file /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/testp.py' /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/testp.py: Python script, ASCII text executable :: [ 09:02:35 ] :: [ PASS ] :: Determining type of /mnt/tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/testp.py (Expected 0, got 0) :: [ 09:02:35 ] :: [ PASS ] :: File '/var/tmp/rlRun_LOG.54CK1Sbo' should contain 'Python script' :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: Duration: 0s :: Assertions: 11 good, 1 bad :: RESULT: FAIL (Test) @kvolny do I understand it correctly that the problem is in test1.py, where the file output is "a /usr/bin/python3 script, ASCII text executable" instead of the correct: "Python script, ASCII text executable"? The problem in the /tests/CoreOS/file/Regression/bz1903531-ftype-detection-is-not-working/test1.py (comment #41), can be solved if we increase strength of the new/patched python script magic patterns so that it beats the https://github.com/file/file/blob/master/magic/Magdir/varied.script#5 pattern, but that can be dangerous, because such a change was not in upstream or in fedora. Also the goal, as stated in the comment #32: "Nothing executable should have text/plain" is satisfied. @sgrubb, what do you think about that, is the current errata good enough from your point of view? If there is a detection problem, we should take that to upstream. If a patch is accepted, then we should update Fedora and give that some runtime. Then later, after that is proven good, backport to RHEL 8. I think the current patches match upstream. I think its the 040 that upstream included. Maybe there's a missing patch that translates 040 from octal into ascii? In any event, I edited magic/Magdir/python to remove the 040, run file -C -m magic/Magdir/python. That makes python.mgc in the top level dir. Then to use it, file -m python.mgc /usr/bin/pip3.6 /usr/bin/pip3.6: Python script, ASCII text executable Hi, Since the issue described in this bug should be resolved (VERIFIED), could you please close this bug with resolution 'CURRENTRELEASE' if this bug got fixed ? If the fix for this is not released yet, check if this will ever get fixed. In case of a negative answer then please change it as WONTFIX. If there's anything else to be done on this BZ, if it's still active, not released yet and we actually intend to release it, then please ignore my message. Please note: for those bugs which are not included in errata, please add 'TestOnly' keyword, and those bugs with 'TestOnly' keyword will be closed automatically after GA. TestOnly: Use this when there is no code delivery involved, or for use when code is already upstream and will be incorporated automatically to the next release for testing purposes only. Thank you. now I am confused to which release this belongs ... it has fixed in file-5.33-17.el8, which was the version I've tested, see comment 41, however the target release is 8.4.0, which includes file-5.33-16.el8_3.1 I believe Kamil knows the process best, so could you take a look please? |
Description of problem: Our default rules depend heavily on the ftype being detected, in particular rules 11 and 12: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- 11 allow perm=open all : ftype=%languages trust=1 12 deny_audit perm=any all : ftype=%languages -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- It appears that the detection code is not working properly, depending on how the shebang line is written, as shown in the example below for a python script: test2.py: (2 spaces between shebang and executable) -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- #! /usr/bin/python3 import os print(f'{os.getpid()}: 2 spaces in shebang') -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Result: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- # fapolicyd-cli --ftype test2.sh text/plain -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- This should be "text/x-python" instead. In fact, as soon as more than one space/tab is found between shebang and executable, the detection fails: OK: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- #!/usr/bin/python3 import os print(f'{os.getpid()}: 0 space in shebang') -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- OK: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- #! /usr/bin/python3 import os print(f'{os.getpid()}: 1 space in shebang') -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- FAIL: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- #! /usr/bin/python3 import os print(f'{os.getpid()}: 2 tabs in shebang') -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Version-Release number of selected component (if applicable): fapolicyd-1.0-3.el8_3.3.x86_64 How reproducible: Always, see above.