Bug 236535

Packages for fedora should not be generated using bdist_rpm, instead they should
have their own spec file for which the brp-python-bytecompile is working for all
the python packages in Fedora.  The script is designed to work with Fedora
packaging policy

http://fedoraproject.org/wiki/Packaging/Guidelines

bdist_rpm command can just redefine %__os_install_post if it wants non distro
specific behaviour - the call to the script is made in the distro specific
redhat-rpm-config package not in the default rpm macros. Alternatively it could
%define _unpackaged_files_terminate_build 0 in the generated spec.

Yeah, that's the thing.  bdist_rpm SIDESTEPS the spec issue.  I don't want to
generate a spec file, especially if Python already does it for me.  I want to
wait 15 seconds to get an RPM, not 15 minutes.  I especially don't want to write
a spec file when it's certain that bdist_rpm works.

It's not a bug in distutils.  It's a bug in your RPM environment.  Distutils was
invented to SAVE the developer the effort of building the RPM spec file.  Your
environment is precluding proper distutils operation.  Hence, it's clearly a bug
in your RPM environment.  FIX IT.

Right now, if it weren't for the brp-python-compile script, the entire Cheese
Shop could be built for Fedora, automatically.  Instead, you're requiring that
every single Python package in existence be changed to comply with your
guidelines.  You should be changing the guidelines if it need be, instead of
asking developers to do extra, unnecessary and MONUMENTAL effort that you're
demanding.

Go search for Python modules RPMs in google.  Now narrow your search to just
fedora 6 packages.  Do you see the effect this braindead policy has?  Almost no
one is building Python module packages for fedora.  I've had to build myself the
last 5 I used.

Of course, I'm not a Python packager; I'm a developer -- why should I be
compiling other people's code?  I should be able to just download the
installable package.  But, of course, due to the policy you just explained
("packages for Fedora shouldn't be generated using bdist_rpm"), there are almost
NO INSTALLABLE PACKAGES of Python modules for Fedora.  And why should packagers
build, when bdist_* for other distros works flawlessly, but Fedora demands EXTRA
workarounds?

(Tongue in cheek: Suddenly I begin to understand why there are so many packages
in Ubuntu's repositories.  Could it be that package creation is easier in Ubuntu?)

Do me a favor and fix the script.  Fix the script.  You know it's a walking bug.
 All whoever wrote the script needed to mind was:

- not generating .pyo files (that, if you've read what they are, you'd
understand they really don't optimize anything)
- not byte-compiling files outside site-packages

The correct fix is:
   rm /usr/lib/rpm/brp-python-bytecompile

I am reopening the bug one more time.  I will do it because I see NOTHING on the
guidelines that support your point of view.  I will do it because I see the
clock on the wall of my house and I can tell I've wasted 2 hours working around
an issue in FEDORA.  I will do it because I've been using Red Hat since 5.2, and
I have faith in its developers.  I will do it because I have hopes that
tomorrow, after I "yum upgrade rpm", I will still be able to build Python
modules just like I could after I applied my fix.  Because, tomorrow, I will
have a distro (Fedora) where Python has the same level of functionality and
support available in the more mainstream distros.

But you should know that my loyalty towards Fedora is wearing thinner and
thinner every day.

Again, the automagic python bytecode compile script should be
disabled if it is not useful or helpful to you.

rpm's default configuration disables the script.

FC enabled the script (and writes spec files with *.pyo in %file manifests) so that *.pyo
files are packaged, not generated as side effects. The goal was to attach selinux
file contexts to *.pyo files explicitly.

FC's goal is likely *not* bdist_rpm's goal, and the packaging conventions for your spec files
do not include *.pyo files, so why are you using the python bytocode compile script at all?

*Now* we're getting somewhere.  So the goal was SELinux-related.

======= BUG PART A ================

Statements:

I don't have any spec files.  I'm letting bdist_rpm generate it.

bdist_rpm doesn't generate the .pyo files I complained about. 
brp-python-bytecompile does.

bdist_rpm does not list the .pyo files in its autogenerated spec file.  It
doesn't, because it doesn't generate them.

Neither does the python interpreter generate .pyo files under regular usage.

Additionally, .pyo files are *useless*.  No third-party Python package that I
know of generates .pyo files.  That's because the optimized Python files aren't
really optimized p-code, but comment-stripped .pyc files.  You want real
optimized Python?  Then you should look into psyco.

So whether .pyo files have a security context or not is completely irrelevant. 
They should actually be actively avoided, since they constitute a waste of disk
space.

My suggestion, in this case, would be to kill the .pyo files generation in
Fedora.  They won't be generated anyway.

======== BUG PART B ==============

brp-python-bytecompile compiles .pyc files outside /usr/lib/python2.x/site-packages.

bdist_rpm is unaware of this.  the spec file generated by bdist_rpm doesn't
include the .pyc files.

The possible solutions would be to:

1. not byte-compile the files outside /usr/lib/python2.x/site-packages.  The
disadvantage would be that Python would compile them sooner or later, when run
as root, thus the files would have the wrong security context.
2. somehow make the brp script *add* the newly compiled files to the package
manifest. 

Which route you guys take, it's up to you.  I think 2. is the better solution,
but I honestly have NO idea how to implement it.

Either A or B is implementable. All depends on what policy is being implemented.

(Disclaimer: I am not and will never be part of Fedora, I'm just explaining how to fix)

Fedora chooses to configure to run brp-python-bytecompile. To automagically add
the set of files that were byte compiled, an extra file to accumulate the paths of
*.pyo files could be added and then included in %files using -f.

The other approach, permitting disabling automagic bytcode compilation for spec
files generated by bdist_files is already available (see comment #1(,

The very best solution (imho) would be to add a independently configurable bytecode
compilation macro that bdist_rpm could configure (if bdist_rpm always chooses no *.pyo)
to override distro policies, or to create a mechanism where the spec file generated by
bdist_rpm could create a spec file that followed distro packaging policies.

I'll be happy to suggest exactly how to achieve both of those goals with rpm macro
configuration if you attach a bdist_rpm generated spec file and a tarball that produces
a *.rpm somehow here.

If the spec file produced by bdist_rpm cannot be changed, well, there's no easy solution.

NEEDINFO

Inside the build directory that the entire bdist_Rpm process generates, is an
INSTALLED_FILES file list.  You could theoretically muck with its contents in
order to get any files to be excluded/included there.

I still don't see why brp-python-bytecompile is needed as a matter of policy for
compilation of distutils-managed python modules.

If a user runs a python script with -O2 either via command line or via a #! line
then the pyo will be attempted to be generated. This will attempt to write to
/usr and generate audit logs/denial messages. Fedora's policy of including *.pyo
in python rpms filelist is designed to prevent this.

See also http://www.redhat.com/archives/fedora-devel-list/2007-April/msg00862.html

Whilst this policy may be considered for change in later versions of fedora it's
simply too late for F7. It's possible to change the bdist_rpm python file to not
generate pyo or ignore unpackaged files.

I've come to the sad realization that it's python's side effects (that shouldn't
really be there) what is dragging us down the rabbit's hole.

In light of this new information, the solution would be to make
brp-python-bytecompile:

1. detect any packages being built through "python setup.py bdist_rpm", compile
.py files (as usual), and then INJECT any files added by brp-python-bytecompile
 into INSTALLED_FILES under the RPM build dir.

That way, third party python modules build seamlessly without the need to write
spec files, just as it was with FC4 and before, IIRC.

Someone should really be writing a patch to python that eliminates the
compilation (py into pyc and pyo) side effects.  Side effects suck.

I've looked at brp-python-bytecompile, and it is clearly a bug that it adds new
files to be packaged but doesn't finish the job by adding them to the package. 
I think the right thing to do is have it 1) discover what files were created and
2) add any of them that aren't already packaged to the package.  I'll write the
patch if asked.  I'm only so-so (if that) with RPM, so I hope someone will give
me hints about the RPM part.  What I post will at least WFM, and I'll respond to
comments as I can find the time.  As any patch cannot ship with F7, that should
be sufficient; hopefully it can ship during F7 and with F8.

> I've come to the sad realization that it's python's side effects (that
> shouldn't really be there) what is dragging us down the rabbit's hole.

So you finally understand that the problem is with upstream Python and not with 
Fedora? So can it be closed as WONTFIX or UPSTREAM again?

IMHO, the issue is that bdist-rpm is not following Fedora's packaging 
guidelines. It's up to Fedora to set packaging guidelines for their 
distribution, not the Python or bdist-rpm developers.

Re: comment #9:

The choice of what to include and whether to automagically compile are different.

Poke me if you need help with the RPM part. rpm can add to %files from a manifest. What's
tricky is divvying up the *.pyo files across sub-packages. There are approaches to automating
the inclusion within sub-packages.

(In reply to comment #11)

> The choice of what to include and whether to automagically compile
> are different.

I think the automagic choice has already been made if brp-python-bytecompile is
used, and that's all I've offered to fix. 8-)  Other issues are outside my scope.

It might be better if the *.py? files could be generated during installation and
still be part of the package for removal purposes.  I assume it wasn't feasible
or it would be done that way.

> Poke me if you need help with the RPM part.

Thanks!

I'll wait until someone expresses a desire to have and ship such a patch first,
though.  No sense in just dropping a, umm, down a hole.

> rpm can add to %files from a manifest.

That I (sort of) remembered.

> What's tricky is divvying up the *.pyo files
> across sub-packages. There are approaches to automating the
> inclusion within sub-packages.

Hmm, hadn't even thought of that.  (Why I need hints!)  Well, presumably each
*.pyc and *.pyo file goes in the package with the original *.py file, if I can
figure out how to tell that.  Can I ask RPM for the files currently in each
subpackage?  During this process, while RPM is (I think) already running?  Or is
there a more wild-card approach where an innocent bdist-rpm spec file can be led
to include all the *.py? rather than just the *.py it expects?

In a nuthsell, what brp-python-bytecompile needs to do is something akin to this:

find $RPM_BUILD_ROOT | grep '.py[co]' > tempfile
sed -i 's/$RPM_BUILD_ROOT//g' tempfile
cat tempfile >> $_RPM_BUILD_ROOT/INSTALLED_FILES
uniq < $_RPM_BUILD_ROOT/INSTALLED_FILES > tempfile
mv tempfile $_RPM_BUILD_ROOT/INSTALLED_FILES

(in other words, find all .pyc and pyo files that were compiled in
brp-python-bytecompile, nuke the $RPM_BUILD_ROOT prefix, merge them with
INSTALLED_FILES)

Kevin, I finally understand nothing.

Your packaging guidelines are NOT the center of the universe.  If python modules
build fine in all distros but Fedora, then it's Fedora's job to tune and tweak
its RPM build system to accomodate the Python ecosystem.

I resent this assertion, especially after you've seen that I'm not the only one
feeling the negative effects of the Fedora guidelines.

Python automatically compiles and has always compiled files on runtime.  When
the python team decides to stop doing that, the world will be a better place,
but in the meantime, if you want python, you must cope with what python does. 
That's it.  We all acknlowledge this compilation side-effect is a problem for
system configuration and deployment, but that is unlinkely to change in the next
year.  File a bug on the python bug tracker if you want it to change.

As a user of both Python and Fedora, and as someone who intends to try creating
an F7 live CD that has to include Python rpms and also expects SE Linux to
operate properly, this discussion is very distressing.  

I know how to run bdist_rpm.  I don't know how to build spec files.  I haven't
needed to learn because Python does it for me.  

I have been having to do workarounds for months because of this disconnect
between Fedora packaging and Python distutils.  These include things like
locally loading Python eggs in my home directory and having to do a PYTHONPATH
every time I run something.  I would gladly prepare rpms of a variety of Python
packages if this thing could get resolved.

I also think this disconnect is preposterous, given that much of the
infrastructure of Fedora-related tools -- including some of the latest SE-Linux
tools -- is built in Python.  I have also found that in many cases the Fedora
repositories are behind in Python packages, possibly because of this disconnect,
and there are many useful Python packages not included in the Fedora
repositories (or included in old versions) because of this disconnect.  In
searching for discussions of this I've already seen a WONTFIX non-solution.

I posted an issue on the Python distutils sig list regarding the same issue. 
This thing needs to be solved, and the sooner the better, and for F7.  If the
Fedora packaging guidelines need a special exception for Python, that needs to
be done.

In the mean time, the Fedora packagers and the Python packagers need to put
their heads together and resolve the issue.  AFAIK, the chief maintainer of the
distutils/setuptools responsible for bdist_rpm is Phillip J. Eby <pje at
telecommunity.com>.  I would urge the Fedora packagers to get in touch with him
and get this issue resolved ASAP.

sklein:

I couldn't agree with you more.  That is exactly what I was trying to make the 
Fedora engineers see.

Anyway, I'm detaching from the discussion because I installed KUbuntu.  I may 
be no Eric S. Raymond, but I can ascertain that KUbuntu kicks Fedora's ass.

Wish you guys all the best.

I see a whole lot of discussion, but no conclusions yet.  Is there any intention
to fix this breakage?  It seems clear to me that brp-python-bytecompile needs to
change.  Is there any consensus on what the fix should be, or does the bdist_rpm
versus brp-python-bytecompile debate rage on?

I would really appreciate it if someone could at least post a patch or
workaround.  Having to create .spec files manually is brain dead.

User pnasrat's account has been closed

Reassigning to owner after bugzilla made a mess, sorry about the noise...

Well, a truly trivial fix is to patch Fedora python bdist_rpm to match the
distro policy wrt .pyo files (for SELinux sanity):

--- bdist_rpm.py.xxx    2007-10-25 15:28:48.000000000 +0300
+++ bdist_rpm.py        2007-10-25 15:36:21.000000000 +0300
@@ -493,6 +493,7 @@
             ('build', 'build_script', def_build),
             ('install', 'install_script',
              ("%s install "
+              "-O1 "
               "--root=$RPM_BUILD_ROOT "
               "--record=INSTALLED_FILES") % def_setup_call),
             ('clean', 'clean_script', "rm -rf $RPM_BUILD_ROOT"),

Assigning to python owner for comments and moving version to devel, unlikely to
get fixed for FC6 at this point.

*** Bug 440354 has been marked as a duplicate of this bug. ***

Thanks for connecting me to the earlier bug report.  I've applied the patch
suggested by Paul Matilainen in comment #20.

Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Same problem in F10.

I like the reasoning by Panu in #20 to make bdist_rpm match the distro policy. I propose that his patch should be applied.

To avoid hardcoding the policy we could also make a bdist_rpm option to disable distro compliance and restore upstream behaviour. I can create a patch for that.

Re-assigning to python owner for comments: Could the patch be applied for F11?

Most of this BZ is from 2007 ... and as far as I know comment #7 still apllies, Fedora policy is to ship .pyo files. So I don't see why we would add patches that disable this in rpm?
Do we have anything from FESCo etc. which says this should change?

Or maybe I'm consing what Panu's patch does, does that remove .pyo files or change their content in some way?

No, the opposite, Panu's patch makes sure .pyo files are included in the rpm.

"python setup.py bdist_rpm" created a temporary .spec with 

%build
python setup.py build
%install
python setup.py install --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES
%files -f INSTALLED_FILES

That causes distutil to install .py and .pyc files in the right root and list their names in INSTALLED_FILES. The problem is that brp-python-bytecompile generates .pyo files and they are not in INSTALLED_FILES.

Panu's patch adds -O1 to the install and thus makes distutil create .pyo and list them in INSTALLED_FILES and we are all happy.

Ok, yeh ... that seems great then. I'll add this.

Yay!  I reported this bug and it's now going to be fixed!  And only 20 months after I reported it!

(I mean, I appreciate your review and efforts, but this is something that could have been taken care of in three, maybe four days.  There's something horribly wrong with the process if this bug can linger for almost two years without being shot in the neck.)

Rudd-O: There's something horribly wrong with the tone in your bug report and the attitude you show here.

Mads: Is this a bug tracker or a grudge tracker?

I have pointed out two facts -- one of the sarcastically, yes, but still fact -- the first (the bug) is taken care of, the other (the deficiencies in the process) is up for grabs.  When the bug bit me and I had enough with the bug (and others!), I simply moved to Kubuntu until I could find an acceptable workaround.  And yay for Kubuntu, I finally got around to hands-on learning the nuances of managing Debian systems thanks to it.

I guess what I'm trying to say here is: don't take facts personally, don't turn facts into excuses for subjective ad hominems.

*** Bug 198877 has been marked as a duplicate of this bug. ***

> Yay!  I reported this bug and it's now going to be fixed!  And only 20 months
> after I reported it!

Not fixed yet, AFAICT. Moreover, status is NEEDINFO. What additional info is provided, and who will provide it?

FTR, this bug affects RHEL5 as well.

Is there anything wrong with Panu's patch?

James Antill> Ok, yeh ... that seems great then. I'll add this.

So James will fix this. Good. Can you also port the fix to RHEL5?

s/additional info is provided/additional info is needed/

Right, I will adjust the NEEDINFO to James Antill: 
Has it been fixed for rawhide/F11?

(Charlie, in order to get the support you pay for for RHEL5 I think you should file a separate issue. But FWIW I expect RHEL to be stable more than bugfree, so IMHO a fix for this annoying issue isn't something RH engineers could or should sneak into a minor release.)

> Has it been fixed for rawhide/F11?

 Yes, I fixed it for Fed-11 when I posted comment #28. Not sure it's worth doing an errata for previous versions of Fedora (so I don't mind leaving this open against Fed-10 for now) ... and as you say, for fixes to RHEL BZ isn't the correct support channel.

I confirm that bdist_rpm works fine (for what it does) in rawhide. (Tested by building a Mercurial rpm.)

IMHO it isn't worth backporting to older versions and this issue could be closed-release-pending.

Note the recent upstream activity on http://bugs.python.org/issue1533164 - it might be fixed for 2.7.

Please backport this.

It will save a lot a people a lot of pain.

Cloned for RHEL 5, with necessary patch attached, as bug 530685.

This message is a reminder that Fedora 10 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 10.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '10'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 10's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 10 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Fedora 10 changed to end-of-life (EOL) status on 2009-12-17. Fedora 10 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.