Two big problems with RPM's /usr/lib/rpm/brp-python-bytecompile in Fedora. I know this command exists only due to the good nature and intentions of some random guy roaming the net, but whoever wrote /usr/lib/rpm/brp-python-bytecompile monumentally fucked up. What I'm about to describe affects tons of Python packages, and due to this bug, there's an absolute lack of Python modules in RPM format for Fedora. I'm going to be completely hones here and ask "What the hell where you smoking when you included this post-process script?" Did you test it with only two or three fairly standard Python packages? Because I'm up to three packages now, successfully compiled ONLY after nuking the damn script. So what's the bug? /usr/lib/rpm/brp-python-bytecompile IS THE bug. It breaks about every single Python package I've tried to bdist_rpm today. First, brp-python-bytecompile creates .pyo files. bdist_rpm doesn't create .pyo files -> the INSTALLED_FILES manifest does not list them -> then RPM finds them "installed but unpackaged". Solution? Do not create .pyo files. Second, brp-python-bytecompile byte-compiles everything under $RPM_BUILD_ROOT. This is also wrong, because it byte-compiles python files that go to different directories, such as /usr/bin. Again, bdist_rpm does not byte-compile these files (it only byte-compiles files targeted to the /usr/lib/python2.x/ directory) => the wrongly compiled files do not appear in the INSTALLED_FILES manifest => BAM! instant RPM "installed but unpackaged" failure. Guys, just nuke the damn /usr/lib/rpm/brp-python-bytecompile. You're trying to second-guess Python's distutils here, and you're doing a bad job. It's full with side effects that shouldn't affect the compile process at all, and it took me a good half hour to find out what was wrong. Nuke it, let bdist_rpm do its job, and let us millions of Fedora users enjoy packages we can actually build and install on our machines. I bet bdist_deb doesn't have this problem.
Packages for fedora should not be generated using bdist_rpm, instead they should have their own spec file for which the brp-python-bytecompile is working for all the python packages in Fedora. The script is designed to work with Fedora packaging policy http://fedoraproject.org/wiki/Packaging/Guidelines bdist_rpm command can just redefine %__os_install_post if it wants non distro specific behaviour - the call to the script is made in the distro specific redhat-rpm-config package not in the default rpm macros. Alternatively it could %define _unpackaged_files_terminate_build 0 in the generated spec.
Yeah, that's the thing. bdist_rpm SIDESTEPS the spec issue. I don't want to generate a spec file, especially if Python already does it for me. I want to wait 15 seconds to get an RPM, not 15 minutes. I especially don't want to write a spec file when it's certain that bdist_rpm works. It's not a bug in distutils. It's a bug in your RPM environment. Distutils was invented to SAVE the developer the effort of building the RPM spec file. Your environment is precluding proper distutils operation. Hence, it's clearly a bug in your RPM environment. FIX IT. Right now, if it weren't for the brp-python-compile script, the entire Cheese Shop could be built for Fedora, automatically. Instead, you're requiring that every single Python package in existence be changed to comply with your guidelines. You should be changing the guidelines if it need be, instead of asking developers to do extra, unnecessary and MONUMENTAL effort that you're demanding. Go search for Python modules RPMs in google. Now narrow your search to just fedora 6 packages. Do you see the effect this braindead policy has? Almost no one is building Python module packages for fedora. I've had to build myself the last 5 I used. Of course, I'm not a Python packager; I'm a developer -- why should I be compiling other people's code? I should be able to just download the installable package. But, of course, due to the policy you just explained ("packages for Fedora shouldn't be generated using bdist_rpm"), there are almost NO INSTALLABLE PACKAGES of Python modules for Fedora. And why should packagers build, when bdist_* for other distros works flawlessly, but Fedora demands EXTRA workarounds? (Tongue in cheek: Suddenly I begin to understand why there are so many packages in Ubuntu's repositories. Could it be that package creation is easier in Ubuntu?) Do me a favor and fix the script. Fix the script. You know it's a walking bug. All whoever wrote the script needed to mind was: - not generating .pyo files (that, if you've read what they are, you'd understand they really don't optimize anything) - not byte-compiling files outside site-packages The correct fix is: rm /usr/lib/rpm/brp-python-bytecompile I am reopening the bug one more time. I will do it because I see NOTHING on the guidelines that support your point of view. I will do it because I see the clock on the wall of my house and I can tell I've wasted 2 hours working around an issue in FEDORA. I will do it because I've been using Red Hat since 5.2, and I have faith in its developers. I will do it because I have hopes that tomorrow, after I "yum upgrade rpm", I will still be able to build Python modules just like I could after I applied my fix. Because, tomorrow, I will have a distro (Fedora) where Python has the same level of functionality and support available in the more mainstream distros. But you should know that my loyalty towards Fedora is wearing thinner and thinner every day.
Again, the automagic python bytecode compile script should be disabled if it is not useful or helpful to you. rpm's default configuration disables the script. FC enabled the script (and writes spec files with *.pyo in %file manifests) so that *.pyo files are packaged, not generated as side effects. The goal was to attach selinux file contexts to *.pyo files explicitly. FC's goal is likely *not* bdist_rpm's goal, and the packaging conventions for your spec files do not include *.pyo files, so why are you using the python bytocode compile script at all?
*Now* we're getting somewhere. So the goal was SELinux-related. ======= BUG PART A ================ Statements: I don't have any spec files. I'm letting bdist_rpm generate it. bdist_rpm doesn't generate the .pyo files I complained about. brp-python-bytecompile does. bdist_rpm does not list the .pyo files in its autogenerated spec file. It doesn't, because it doesn't generate them. Neither does the python interpreter generate .pyo files under regular usage. Additionally, .pyo files are *useless*. No third-party Python package that I know of generates .pyo files. That's because the optimized Python files aren't really optimized p-code, but comment-stripped .pyc files. You want real optimized Python? Then you should look into psyco. So whether .pyo files have a security context or not is completely irrelevant. They should actually be actively avoided, since they constitute a waste of disk space. My suggestion, in this case, would be to kill the .pyo files generation in Fedora. They won't be generated anyway. ======== BUG PART B ============== brp-python-bytecompile compiles .pyc files outside /usr/lib/python2.x/site-packages. bdist_rpm is unaware of this. the spec file generated by bdist_rpm doesn't include the .pyc files. The possible solutions would be to: 1. not byte-compile the files outside /usr/lib/python2.x/site-packages. The disadvantage would be that Python would compile them sooner or later, when run as root, thus the files would have the wrong security context. 2. somehow make the brp script *add* the newly compiled files to the package manifest. Which route you guys take, it's up to you. I think 2. is the better solution, but I honestly have NO idea how to implement it.
Either A or B is implementable. All depends on what policy is being implemented. (Disclaimer: I am not and will never be part of Fedora, I'm just explaining how to fix) Fedora chooses to configure to run brp-python-bytecompile. To automagically add the set of files that were byte compiled, an extra file to accumulate the paths of *.pyo files could be added and then included in %files using -f. The other approach, permitting disabling automagic bytcode compilation for spec files generated by bdist_files is already available (see comment #1(, The very best solution (imho) would be to add a independently configurable bytecode compilation macro that bdist_rpm could configure (if bdist_rpm always chooses no *.pyo) to override distro policies, or to create a mechanism where the spec file generated by bdist_rpm could create a spec file that followed distro packaging policies. I'll be happy to suggest exactly how to achieve both of those goals with rpm macro configuration if you attach a bdist_rpm generated spec file and a tarball that produces a *.rpm somehow here. If the spec file produced by bdist_rpm cannot be changed, well, there's no easy solution. NEEDINFO
Inside the build directory that the entire bdist_Rpm process generates, is an INSTALLED_FILES file list. You could theoretically muck with its contents in order to get any files to be excluded/included there. I still don't see why brp-python-bytecompile is needed as a matter of policy for compilation of distutils-managed python modules.
If a user runs a python script with -O2 either via command line or via a #! line then the pyo will be attempted to be generated. This will attempt to write to /usr and generate audit logs/denial messages. Fedora's policy of including *.pyo in python rpms filelist is designed to prevent this. See also http://www.redhat.com/archives/fedora-devel-list/2007-April/msg00862.html Whilst this policy may be considered for change in later versions of fedora it's simply too late for F7. It's possible to change the bdist_rpm python file to not generate pyo or ignore unpackaged files.
I've come to the sad realization that it's python's side effects (that shouldn't really be there) what is dragging us down the rabbit's hole. In light of this new information, the solution would be to make brp-python-bytecompile: 1. detect any packages being built through "python setup.py bdist_rpm", compile .py files (as usual), and then INJECT any files added by brp-python-bytecompile into INSTALLED_FILES under the RPM build dir. That way, third party python modules build seamlessly without the need to write spec files, just as it was with FC4 and before, IIRC. Someone should really be writing a patch to python that eliminates the compilation (py into pyc and pyo) side effects. Side effects suck.
I've looked at brp-python-bytecompile, and it is clearly a bug that it adds new files to be packaged but doesn't finish the job by adding them to the package. I think the right thing to do is have it 1) discover what files were created and 2) add any of them that aren't already packaged to the package. I'll write the patch if asked. I'm only so-so (if that) with RPM, so I hope someone will give me hints about the RPM part. What I post will at least WFM, and I'll respond to comments as I can find the time. As any patch cannot ship with F7, that should be sufficient; hopefully it can ship during F7 and with F8.
> I've come to the sad realization that it's python's side effects (that > shouldn't really be there) what is dragging us down the rabbit's hole. So you finally understand that the problem is with upstream Python and not with Fedora? So can it be closed as WONTFIX or UPSTREAM again? IMHO, the issue is that bdist-rpm is not following Fedora's packaging guidelines. It's up to Fedora to set packaging guidelines for their distribution, not the Python or bdist-rpm developers.
Re: comment #9: The choice of what to include and whether to automagically compile are different. Poke me if you need help with the RPM part. rpm can add to %files from a manifest. What's tricky is divvying up the *.pyo files across sub-packages. There are approaches to automating the inclusion within sub-packages.
(In reply to comment #11) > The choice of what to include and whether to automagically compile > are different. I think the automagic choice has already been made if brp-python-bytecompile is used, and that's all I've offered to fix. 8-) Other issues are outside my scope. It might be better if the *.py? files could be generated during installation and still be part of the package for removal purposes. I assume it wasn't feasible or it would be done that way. > Poke me if you need help with the RPM part. Thanks! I'll wait until someone expresses a desire to have and ship such a patch first, though. No sense in just dropping a, umm, down a hole. > rpm can add to %files from a manifest. That I (sort of) remembered. > What's tricky is divvying up the *.pyo files > across sub-packages. There are approaches to automating the > inclusion within sub-packages. Hmm, hadn't even thought of that. (Why I need hints!) Well, presumably each *.pyc and *.pyo file goes in the package with the original *.py file, if I can figure out how to tell that. Can I ask RPM for the files currently in each subpackage? During this process, while RPM is (I think) already running? Or is there a more wild-card approach where an innocent bdist-rpm spec file can be led to include all the *.py? rather than just the *.py it expects?
In a nuthsell, what brp-python-bytecompile needs to do is something akin to this: find $RPM_BUILD_ROOT | grep '.py[co]' > tempfile sed -i 's/$RPM_BUILD_ROOT//g' tempfile cat tempfile >> $_RPM_BUILD_ROOT/INSTALLED_FILES uniq < $_RPM_BUILD_ROOT/INSTALLED_FILES > tempfile mv tempfile $_RPM_BUILD_ROOT/INSTALLED_FILES (in other words, find all .pyc and pyo files that were compiled in brp-python-bytecompile, nuke the $RPM_BUILD_ROOT prefix, merge them with INSTALLED_FILES)
Kevin, I finally understand nothing. Your packaging guidelines are NOT the center of the universe. If python modules build fine in all distros but Fedora, then it's Fedora's job to tune and tweak its RPM build system to accomodate the Python ecosystem. I resent this assertion, especially after you've seen that I'm not the only one feeling the negative effects of the Fedora guidelines. Python automatically compiles and has always compiled files on runtime. When the python team decides to stop doing that, the world will be a better place, but in the meantime, if you want python, you must cope with what python does. That's it. We all acknlowledge this compilation side-effect is a problem for system configuration and deployment, but that is unlinkely to change in the next year. File a bug on the python bug tracker if you want it to change.
As a user of both Python and Fedora, and as someone who intends to try creating an F7 live CD that has to include Python rpms and also expects SE Linux to operate properly, this discussion is very distressing. I know how to run bdist_rpm. I don't know how to build spec files. I haven't needed to learn because Python does it for me. I have been having to do workarounds for months because of this disconnect between Fedora packaging and Python distutils. These include things like locally loading Python eggs in my home directory and having to do a PYTHONPATH every time I run something. I would gladly prepare rpms of a variety of Python packages if this thing could get resolved. I also think this disconnect is preposterous, given that much of the infrastructure of Fedora-related tools -- including some of the latest SE-Linux tools -- is built in Python. I have also found that in many cases the Fedora repositories are behind in Python packages, possibly because of this disconnect, and there are many useful Python packages not included in the Fedora repositories (or included in old versions) because of this disconnect. In searching for discussions of this I've already seen a WONTFIX non-solution. I posted an issue on the Python distutils sig list regarding the same issue. This thing needs to be solved, and the sooner the better, and for F7. If the Fedora packaging guidelines need a special exception for Python, that needs to be done. In the mean time, the Fedora packagers and the Python packagers need to put their heads together and resolve the issue. AFAIK, the chief maintainer of the distutils/setuptools responsible for bdist_rpm is Phillip J. Eby <pje at telecommunity.com>. I would urge the Fedora packagers to get in touch with him and get this issue resolved ASAP.
sklein: I couldn't agree with you more. That is exactly what I was trying to make the Fedora engineers see. Anyway, I'm detaching from the discussion because I installed KUbuntu. I may be no Eric S. Raymond, but I can ascertain that KUbuntu kicks Fedora's ass. Wish you guys all the best.
I see a whole lot of discussion, but no conclusions yet. Is there any intention to fix this breakage? It seems clear to me that brp-python-bytecompile needs to change. Is there any consensus on what the fix should be, or does the bdist_rpm versus brp-python-bytecompile debate rage on? I would really appreciate it if someone could at least post a patch or workaround. Having to create .spec files manually is brain dead.
User pnasrat's account has been closed
Reassigning to owner after bugzilla made a mess, sorry about the noise...
Well, a truly trivial fix is to patch Fedora python bdist_rpm to match the distro policy wrt .pyo files (for SELinux sanity): --- bdist_rpm.py.xxx 2007-10-25 15:28:48.000000000 +0300 +++ bdist_rpm.py 2007-10-25 15:36:21.000000000 +0300 @@ -493,6 +493,7 @@ ('build', 'build_script', def_build), ('install', 'install_script', ("%s install " + "-O1 " "--root=$RPM_BUILD_ROOT " "--record=INSTALLED_FILES") % def_setup_call), ('clean', 'clean_script', "rm -rf $RPM_BUILD_ROOT"), Assigning to python owner for comments and moving version to devel, unlikely to get fixed for FC6 at this point.
*** Bug 440354 has been marked as a duplicate of this bug. ***
Thanks for connecting me to the earlier bug report. I've applied the patch suggested by Paul Matilainen in comment #20.
Changing version to '9' as part of upcoming Fedora 9 GA. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Same problem in F10. I like the reasoning by Panu in #20 to make bdist_rpm match the distro policy. I propose that his patch should be applied. To avoid hardcoding the policy we could also make a bdist_rpm option to disable distro compliance and restore upstream behaviour. I can create a patch for that. Re-assigning to python owner for comments: Could the patch be applied for F11?
Most of this BZ is from 2007 ... and as far as I know comment #7 still apllies, Fedora policy is to ship .pyo files. So I don't see why we would add patches that disable this in rpm? Do we have anything from FESCo etc. which says this should change?
Or maybe I'm consing what Panu's patch does, does that remove .pyo files or change their content in some way?
No, the opposite, Panu's patch makes sure .pyo files are included in the rpm. "python setup.py bdist_rpm" created a temporary .spec with %build python setup.py build %install python setup.py install --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES %files -f INSTALLED_FILES That causes distutil to install .py and .pyc files in the right root and list their names in INSTALLED_FILES. The problem is that brp-python-bytecompile generates .pyo files and they are not in INSTALLED_FILES. Panu's patch adds -O1 to the install and thus makes distutil create .pyo and list them in INSTALLED_FILES and we are all happy.
Ok, yeh ... that seems great then. I'll add this.
Yay! I reported this bug and it's now going to be fixed! And only 20 months after I reported it! (I mean, I appreciate your review and efforts, but this is something that could have been taken care of in three, maybe four days. There's something horribly wrong with the process if this bug can linger for almost two years without being shot in the neck.)
Rudd-O: There's something horribly wrong with the tone in your bug report and the attitude you show here.
Mads: Is this a bug tracker or a grudge tracker? I have pointed out two facts -- one of the sarcastically, yes, but still fact -- the first (the bug) is taken care of, the other (the deficiencies in the process) is up for grabs. When the bug bit me and I had enough with the bug (and others!), I simply moved to Kubuntu until I could find an acceptable workaround. And yay for Kubuntu, I finally got around to hands-on learning the nuances of managing Debian systems thanks to it. I guess what I'm trying to say here is: don't take facts personally, don't turn facts into excuses for subjective ad hominems.
*** Bug 198877 has been marked as a duplicate of this bug. ***
> Yay! I reported this bug and it's now going to be fixed! And only 20 months > after I reported it! Not fixed yet, AFAICT. Moreover, status is NEEDINFO. What additional info is provided, and who will provide it? FTR, this bug affects RHEL5 as well. Is there anything wrong with Panu's patch? James Antill> Ok, yeh ... that seems great then. I'll add this. So James will fix this. Good. Can you also port the fix to RHEL5?
s/additional info is provided/additional info is needed/
Right, I will adjust the NEEDINFO to James Antill: Has it been fixed for rawhide/F11? (Charlie, in order to get the support you pay for for RHEL5 I think you should file a separate issue. But FWIW I expect RHEL to be stable more than bugfree, so IMHO a fix for this annoying issue isn't something RH engineers could or should sneak into a minor release.)
> Has it been fixed for rawhide/F11? Yes, I fixed it for Fed-11 when I posted comment #28. Not sure it's worth doing an errata for previous versions of Fedora (so I don't mind leaving this open against Fed-10 for now) ... and as you say, for fixes to RHEL BZ isn't the correct support channel.
I confirm that bdist_rpm works fine (for what it does) in rawhide. (Tested by building a Mercurial rpm.) IMHO it isn't worth backporting to older versions and this issue could be closed-release-pending. Note the recent upstream activity on http://bugs.python.org/issue1533164 - it might be fixed for 2.7.
Please backport this. It will save a lot a people a lot of pain.
Cloned for RHEL 5, with necessary patch attached, as bug 530685.
This message is a reminder that Fedora 10 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 10. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '10'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 10's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 10 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Fedora 10 changed to end-of-life (EOL) status on 2009-12-17. Fedora 10 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. Thank you for reporting this bug and we are sorry it could not be fixed.