Bug 953704

Summary: [RFE] provide a RPM default provides filtering macro
Product: [Fedora] Fedora Reporter: T.C. Hollingsworth <tchollingsworth>
Component: python2Assignee: Charalampos Stratakis <cstratak>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: a.badger, bkabrda, carl, cstratak, dmalcolm, ivazqueznet, jonathansteffan, mhroncok, orion, pviktori, rkuska, tomspur, torsava
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-08 15:05:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
New macros to make writing python specs comfortable
none
Updated macro file
none
Python coverage spec diff none

Description T.C. Hollingsworth 2013-04-19 01:07:47 UTC
Other interpreted languages like Perl and Node.js provide an RPM macro named like %{perl_default_filter} that filters the unwanted Provides from the shared object files that make up C extensions to their interpreters.

Python could also benefit from a %{python(2|3)_default_filter} that does the same thing.

Just add something like this to /etc/rpm/macros.python2:

%python2_default_filter %{expand: \
%global __provides_exclude_from ^%{python2_sitearch}/.*\\.so$
}

For more information, see:
https://fedoraproject.org/wiki/Packaging:AutoProvidesAndRequiresFiltering

Comment 1 Thomas Spura 2013-04-19 07:34:32 UTC
@Dave:
I like to work on that as part of the Python package renaming, if possible:
https://fedoraproject.org/wiki/Features/PythonNamingDependingOnImplementation

Comment 2 Bohuslav "Slavek" Kabrda 2013-04-19 14:00:20 UTC
This is a good idea. For some time already, I've wanted to discuss introducing new macros with FPC (shortening the repetitive "%{__python} setup.py install --skip-build --root ..." and the same for "%{__python} setup.py build") - so I may put this into one ticket, does that sound good?

@Thomas
I generally like your proposal, but there is one part I would change - I would rather keep the "current" (== just a sane choice) version of interpreters with non-versioned names. So not pypy1, but just pypy, not python2 and python3, but just python and python3. I have some other ideas, too, let's discuss them by email/on MLs and stay OT here :)

Comment 3 Thomas Spura 2013-04-19 14:49:38 UTC
(In reply to comment #2)
> This is a good idea. For some time already, I've wanted to discuss
> introducing new macros with FPC (shortening the repetitive "%{__python}
> setup.py install --skip-build --root ..." and the same for "%{__python}
> setup.py build") - so I may put this into one ticket, does that sound good?

+1

> @Thomas
> I generally like your proposal, but there is one part I would change - I
> would rather keep the "current" (== just a sane choice) version of
> interpreters with non-versioned names. So not pypy1, but just pypy, not
> python2 and python3, but just python and python3. I have some other ideas,
> too, let's discuss them by email/on MLs and stay OT here :)

With my proposal, it's easy to swtich the default python implementation. I'm not sure, how this could be done otherwise. Maybe with some other "python-definitions" package or so. But ok, let's discuss this elsewhere :)

Comment 4 Bohuslav "Slavek" Kabrda 2013-04-23 10:07:54 UTC
Created attachment 738922 [details]
New macros to make writing python specs comfortable

So, these are the proposed macros. They should probably live in "python-devel-common" package, that would be required by both python-devel and python3-devel. Any comments before I propose this to fpc?

Thanks.

Comment 5 Thomas Spura 2013-04-23 16:09:12 UTC
Those macros help to cope with the current situation of python(=python2) and python3, but not with the general confusing python naming.

In my opinion, these should go into python-devel and the standard python package needs a rename to python2.

Comments for the macros:
- I'd just call it "%python_default_filter" instead of
  "%python_all_default_filter", which should filter all python related internal 
  libraries (also pypy etc.) in the future.
- Is it possible for py_build and py_install to accept parameters?
  e.g. %py_build setup-foo.py and without a parameter, it defaults to setup.py?
  Some projects have for example a setupegg.py and a setup.py.
- Both macros use __python, which should be the default implementation.
  Shouldn't we also add py2_build/py3_build/pypy_build etc?
  That's one example, why I incredebly like the complete separation of __python
  being "some default python implementation, which is just by accident python2 
  for now".

Comment 6 Bohuslav "Slavek" Kabrda 2013-04-24 06:08:29 UTC
(In reply to comment #5)
> Those macros help to cope with the current situation of python(=python2) and
> python3, but not with the general confusing python naming.
> 
> In my opinion, these should go into python-devel and the standard python
> package needs a rename to python2.
> 

I still think that we should have one "python" package, whichever that is :) (and just the second one should be named with version appended to name).

> Comments for the macros:
> - I'd just call it "%python_default_filter" instead of
>   "%python_all_default_filter", which should filter all python related
> internal 
>   libraries (also pypy etc.) in the future.

Good point, I'll rename that.

> - Is it possible for py_build and py_install to accept parameters?
>   e.g. %py_build setup-foo.py and without a parameter, it defaults to
> setup.py?

It's probably my fault that I didn't include any example with this, but:
- %{?py_setup_args} is used to pass parameters to setup.py
- %{?pybuild_args} (which should be py_build_args) is used to pass parameters to "build"
- %{?py_install_args} is used to pass extra parameters to "install"

>   Some projects have for example a setupegg.py and a setup.py.

The default behaviour can be modified by altering %py_setup macro.

> - Both macros use __python, which should be the default implementation.
>   Shouldn't we also add py2_build/py3_build/pypy_build etc?
>   That's one example, why I incredebly like the complete separation of
> __python
>   being "some default python implementation, which is just by accident
> python2 
>   for now".

The idea is that the default python implementation is also the default used. If you want to run with a different implementation, you pass the executable as a parameter. So:

%py_build %__python3 - runs /usr/bin/python3 setup.py build

Although I admit that bringing macros for every python implementation doesn't sound bad. I'll do some fixes and upload new version soon.

Thanks.

Comment 7 Bohuslav "Slavek" Kabrda 2013-04-24 06:43:37 UTC
Created attachment 739290 [details]
Updated macro file

Comment 8 Bohuslav "Slavek" Kabrda 2013-04-24 06:44:40 UTC
Created attachment 739291 [details]
Python coverage spec diff

So, the above is the updated macro file and here is a diff of changes that I made on python-coverage.spec, that demonstrates these changes.

Comment 9 Bohuslav "Slavek" Kabrda 2013-05-02 10:58:42 UTC
Proposed to FPC at https://fedorahosted.org/fpc/ticket/281.

Comment 10 Toshio Ernie Kuratomi 2013-05-02 16:34:58 UTC
I like the %python_default_filter macro.  I do not like the py_build and py_install macros.  The general reason goes back to the zen of python: Explicit is better than implicit.  By using the explicit form people are taught what commands are actually being run and how to modify them if problems arise.

However, I could be alone in that feeling.  So there are some specifics as well:

* The overall recipe for building python modules may be changing soon to use wheels as an intermediate step similar to gems (but starting with tarballs and ending with source installed into locations in site-packages and FHS filesystem locations instead of gems => expanded gems).  If the goal of the macros is to make it so we don't have to change spec files during a transition like that we should see what building via wheel looks like and change the macros accordingly.

* tomspur touches on some of the issues with throwing alternate python2 interpreters into the mix.  I lean towards it being time to figure out the new guidelines for those as well.

* I don't think the pattern of a separate variable for args is all that common.  Would be nice to rework that so we're giving the variables to the macro.  Even if we have to write a helper script to do this it might end up being more maintainable in the end.

Anyhow -- I think we should take this discussion to the mailing list to figure out with the help of ncoghlan and the like.  Maybe we even need to have a few python-sig meetings on irc (after we come up with a list of things we need to change in the mailing list).

Comment 11 Toshio Ernie Kuratomi 2013-05-02 16:37:06 UTC
let me revise that last point a little bit:

I think the default_filter can go to FPC now but I'd like the other parts of it discussed on the mailing list.  (My difference in feeling there is that default_filter is applied to rpm behaviour.  py_build/etc is applied to the python module building process.)

Comment 12 Bohuslav "Slavek" Kabrda 2013-05-03 07:00:57 UTC
(In reply to comment #10)
> I like the %python_default_filter macro.  I do not like the py_build and
> py_install macros.  The general reason goes back to the zen of python:
> Explicit is better than implicit.  By using the explicit form people are
> taught what commands are actually being run and how to modify them if
> problems arise.
> 

IMHO they can just have a look into the macros file, I will document the usage there properly.
Also, zen of Python doesn't seem to be relevant for Python packaging in the same sense that Ruby way of doing things doesn't apply to packaging RubyGems, IMHO.

> However, I could be alone in that feeling.  So there are some specifics as
> well:
> 
> * The overall recipe for building python modules may be changing soon to use
> wheels as an intermediate step similar to gems (but starting with tarballs
> and ending with source installed into locations in site-packages and FHS
> filesystem locations instead of gems => expanded gems).  If the goal of the
> macros is to make it so we don't have to change spec files during a
> transition like that we should see what building via wheel looks like and
> change the macros accordingly.
> 

Yes, this is one of the goals of macros.
I've been looking into how wheels work and I believe that the only thing we will need to add is %py_prep (let's call it this way for now), that would do "wheel unpack". Then we will build the wheel in %build and install it in %install. The later two steps will be done just by modifying the macros, which doesn't need to be done ATM. The first step will probably need to be added to all the specfiles then, now it would probably be just an empty macro.

> * tomspur touches on some of the issues with throwing alternate python2
> interpreters into the mix.  I lean towards it being time to figure out the
> new guidelines for those as well.
> 

True, but that will probably take a lot of time and I think it doesn't have to block these macros.

> * I don't think the pattern of a separate variable for args is all that
> common.  Would be nice to rework that so we're giving the variables to the
> macro.  Even if we have to write a helper script to do this it might end up
> being more maintainable in the end.
> 

So, here's the problem with args - there are basically two "groups" of them. The ones that you pass to "setup.py" and the ones that you pass to "build" or "install". You can't combine these, e.g. if the proper call is
setup.py --foo build --bar,
you _can not_ use
setup.py build --foo --bar
So you'd need to do something like
%py_setup --args-for-setuppy --foo --args-for-action --bar
which is not very convenient (and probably not doable in RPM without using some helper scripts) :) So I think the macros are actually Any better ideas how to solve this?

> Anyhow -- I think we should take this discussion to the mailing list to
> figure out with the help of ncoghlan and the like.  Maybe we even need to
> have a few python-sig meetings on irc (after we come up with a list of
> things we need to change in the mailing list).

I'll start the discussion there (I'll probably postpone it to the next week, if you don't mind).

Thanks for the comments.

Comment 13 Toshio Ernie Kuratomi 2013-05-03 14:28:20 UTC
(In reply to comment #12)
> Also, zen of Python doesn't seem to be relevant for Python packaging in the
> same sense that Ruby way of doing things doesn't apply to packaging
> RubyGems, IMHO.
>
the way I see it expanded vs macroized build script invocation are both in wide-spread use in fedora packaging.  Applying the Zen of python to which form to use for the fedora Python guidelines doesn't override fedora conventions, it just reinforces why this community might choose one form over the other.
 
The rubygem discussions, otoh, wanted to do things like build in %prep and not unzip the source which are not conventions within 
 
> I've been looking into how wheels work and I believe that the only thing we
> will need to add is %py_prep (let's call it this way for now), that would do
> "wheel unpack".

Just wanted to point out for this particular portion that this is *not* what we want to do at all.  Like I said, wheels are just an intermediate format.  We're still going to start with tarballs.

> 
> So, here's the problem with args - there are basically two "groups" of them.
> The ones that you pass to "setup.py" and the ones that you pass to "build"
> or "install". You can't combine these, e.g. if the proper call is
> setup.py --foo build --bar,
> you _can not_ use
> setup.py build --foo --bar
> So you'd need to do something like
> %py_setup --args-for-setuppy --foo --args-for-action --bar
> which is not very convenient (and probably not doable in RPM without using
> some helper scripts) :) So I think the macros are actually Any better ideas
> how to solve this?
>
Well, I already stated a preference for a helper script :-)  But a different way would be to think of it as two macros that compose the command:

%py_setup %build

then you can either have people inject their args in between those or you can have them override the pieces.

> > Anyhow -- I think we should take this discussion to the mailing list to
> > figure out with the help of ncoghlan and the like.  Maybe we even need to
> > have a few python-sig meetings on irc (after we come up with a list of
> > things we need to change in the mailing list).
> 
> I'll start the discussion there (I'll probably postpone it to the next week,
> if you don't mind).
> 
fine with me -- and btw, I realized we had the choice of a couple mailing lists after writing this.  I know that ncoghlan is on python-devel so that might be where we want to start and then move to packaging once we have something less like brainstorming and more like a proposal.

Comment 14 Fedora Admin XMLRPC Client 2013-05-10 04:58:31 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 15 Orion Poplawski 2015-03-19 17:35:57 UTC
Shouldn't:

%py2_build %py_rt_build %__python
%py2_install %py_rt_install %__python

be:

%py2_build %py_rt_build %__python2
%py2_install %py_rt_install %__python2

Comment 16 Fedora Admin XMLRPC Client 2015-05-12 12:02:00 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 17 Fedora Admin XMLRPC Client 2016-01-29 13:04:11 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 18 Fedora Admin XMLRPC Client 2017-01-10 18:48:28 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 19 Petr Viktorin (pviktori) 2017-08-08 15:05:56 UTC
It seems to me that the discussion on this bug doesn't relate much to the original report -- adding a %python2_default_filter macro. A lot of other macros were discussed and added, but not the filter one.

At this point, if the original request is still relevant, I think would be better to open a new bug.

T.C. Hollingsworth, if %pythonX_default_filter macros would still help you, please let me know; we can open a new bug and discuss the specifics there.