Bug 137863

This might be a duplicate of bug #131439.

I'm currently investigating with the assistence of Mihai (misa). This
does not appear to be a mailman issue but rather a python compiler and
interpreter issue. Will update shortly.

I agree that part of the problem I reported is in #131439. 
 
But what about /usr/lib/mailman/mail/mailman?  Is that written in 
Python too? 

To answer the question in comment #2. Technically it is written in C
but for all practical effective reasons its python code. Let me
explain. It is a wrapper that enforces security, it validates the
permissions of the user executing it, it validates the command passed
to it, it forces an explict value of PYTHONPATH in the environment so
that only the python code belonging to mailman can be found. It then
execv's the python script given as an argument to it.

It is for all practical purposes python code execution. The ELF
executable validates and then steps aside passing control to the
python interpreter.

python-2.3.4-12 and newer should fix the .pyc files to have the proper
path embedded in them.

using 2.3.4-12 it appears as if the build path is still in the .pyc.
That in and of itself might not be an issue, only as long as executing
the .pyc doesn't cause python to try and touch anything along that path.  

Mihai, what was the change you put into 2.3.4-12? Was it something to
stop the build path from appearing in the .pyc or something to stop
the interpreter from trying to access something on that path?

My fix was to put the _proper_ path in the .pyc files.
It used to have the full buildroot path to the .py file, now it should
have the right one (still absolute path, but without the buildroot).

I'm confused Mihai. If the solution is to modify the python compiler
then doesn't that new python compiler have to be installed on every
build machine that beehive uses? I just rebuilt the mailman package
and sure enough there are still build root paths in the .pyc files, I
assume because the build roots are still using an older python before
your fix.

Do you know exactly what the interpreter is doing when it reads a .pyc
file and encounters the path to the originial .py file? I haven't
found that in the python code yet (I stopped looking after you said
you had a fix), but I did find the code in python which does the
.py/.pyc dependency check and it never looks at any path in the .pyc.
This is done before the .pyc is executed. I'm assuming when execution
of the .pyc begins is when the path to the buildroot source file is
encountered, since it is not involved in the dependency checking
between .py and .pyc (or is it a 2nd check?) I wonder what its doing
with this path. Anyway, at the moment the fix feels like it occurred
in the wrong place unless we're going to update every buildroot.

I'll have to see the spec file for mailman.

This is what the spec file does:

# Compile .pyc and .pyo
%{__python} -c "import compileall;
compileall.compile_dir('$RPM_BUILD_ROOT%{mmdir}', ddir='%{mmdir}')"
%{__python} -O -c "import compileall;
compileall.compile_dir('$RPM_BUILD_ROOT%{mmdir}', ddir='%{mmdir}')"

the comile directory is the absolute path in the build root. Two thoughs:

1) should the spec file cd into the buildroot and then give a relative
path to the compile command?

2) Even if the above succeeded in getting a non build root path into
the .pyc isn't it simply wrong that the interpreter is using a path
embedded in the .pyc?

The problem is that your compileall directives will not execute
because the .pyc and .pyo files already exist. THis is what I did:

rpmbuild -bi mailman.spec
find /var/tmp/mailman-root/usr/lib/mailman -name \*pyc -exec rm -f {} \;
find /var/tmp/mailman-root/usr/lib/mailman -name \*pyo -exec rm -f {} \;

Then ran the above-mentioned commands.
And then the .pyc files no longer had the buildroot.

But what I would do instead of adding the compileall directives in the
spec file is change Makefile.in from:
 $(PYTHON) -c 'from compileall import *;
compile_dir("$(DESTDIR)$(prefix)/Mailman")'

to: 
 $(PYTHON) -c 'from compileall import *;
compile_dir("$(DESTDIR)$(prefix)/Mailman", ddir="$(prefix)/Mailman")'

and then the right thing would happen.

... except that there are other subdirectories that would still have
the wrong .pyc files.

re comment #11, I assume you mean adding ddir would put the wrong path
in some .pyc files, right?

I've had a devil of a time reproducing the problem. I can't generate
any accesses to /usr/src/... either running via cgi/MTA invocation nor
running by hand on the command line. I can't find anything in the
python code per say that would cause it to access a file embedded in
the .pyc. I think the embedded pathnames in the .pyc file are only for
error reporting (e.g. python stack traces). Hah! It suddenly occurred
to me, too bad I'm on my way out the door, but I bet the reason I
haven't been able to reproduce the access is because I'm not
generating any errors that would cause the python interpreter to
generate a stack trace. Tomorrow I'll booby-trap mailman so it
generates a stack-trace, maybe then I'll finally be able to see an
access into /usr/src/...

Russell, when you produced this what where you doing? I do know that
when the SELinux policy is not correct for one often will generate
python stack traces due to open certain files. I wonder if the access
to /usr/src/... is a side effect of a policy induced error, if I don't
have policy errors I'll never see the access because there is no stack
trace. Something to be tested tomorrow.

Created attachment 106352 [details]
strace showing access to /usr/src/build during python exception reporting

I finally succeeded in reproducing the access to a file in
/usr/src/build. It is as I had hypothesized, a consequence of python
generating a stack trace. Here is what I know at the moment:

During normal and correct operation mailman never accesses anything in
/usr/src. Accesses to /usr/src/ have typically been observed only when
the SELinux policy prevents mailman from operating correctly, which
induces a python stack trace due to a python exception. At the time
the python exception is raised the python interpreter is executing a
.pyc file. The .pyc files have the absolute path to the original
source .py file from the buildroot (e.g. /usr/src/build) embedded in
the file. The only time the source file that generated the .pyc is
referenced is during error reporting.

There is an option to the python compiler command that allows the
embedding of a different path for the purpose of error reporting into
the .pyc file. There are some issues with getting this set properly
during the build but I will as my next task pursue this.

However, I want to point out that this SELinux access problem is
probably not unique to mailman. I suspect this problem exists for many
packages that compile .pyc files. Also note that SELinux pretty much
demands that the .pyc files be generated as part of the package build
because the default behavior of the python interpreter to write a .pyc
file will almost always be denied by SELinux security policy. Finally
note that the problem is not limited to python stack traces generated
as a consequence of incorrect SELinux security policy, any python
stack trace where any .pyc file did not override the debug path will
generate the problem, its just that most python codes runs without
exceptions, its the security policy that tends to generate the
exceptions.

We may want to look at all packages that generate .pyc files and
assure they all override the debug pathnames during compilation in the
buildroot.

This should be fixed in mailman-2.1.5-29.fc3 and beyond. I modified
the compile command to add the ddir parameter and forced all python
files to be recompiled and picked up some .py files that has escaped
compilation. This is all done in the root Makefile.in, no longer does
the spec file try to compile. No .pyc file in the rpm any longer has a
reference to /usr/src/build!

Dan, I'm reassigning this to you so you can remove src_t from the
mailman.te policy file which was added to compensate for .pyc stack
trace behavior.

Just as a side note, I wonder if we've got .pyc files in other rpms
which also reference files in the buildroot.

Removed src_t from selinux-policy-strict-1.19.1-4