Bug 1636020 - Provide %py3_build_expected, and/or %py2_build_expected macro
Summary: Provide %py3_build_expected, and/or %py2_build_expected macro
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: redhat-rpm-config
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Florian Festi
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-10-04 09:25 UTC by Pavel Raiskup
Modified: 2019-06-08 09:28 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-07 14:06:32 UTC
Type: Bug


Attachments (Terms of Use)
Proposal for Fedora 30 and Rawhide (1.04 KB, patch)
2018-10-04 09:25 UTC, Pavel Raiskup
no flags Details | Diff

Description Pavel Raiskup 2018-10-04 09:25:14 UTC
Created attachment 1490416 [details]
Proposal for Fedora 30 and Rawhide

It is pretty inconvenient to maintain python spec files consistently across
all our dist-git branches (el6+, f27+).  See e.g. discussion [1].  It would
be nice to provide macros that give us chance to detect what the particular distro version expects us to do (build python2, python3, or both).

Proposed patch for redhat-rpm-config f30+ attached.  I can equivalently
generate patches against all the branches, but that would be better
through PRs.

This way we could do conditionals like:
    
      %if 0%{?py3_build_expected:1}
      %package -n python3-foo
      ...
      %endif
    
Or the single-line variant:
    
      %{?py2_build_expected:%py2_build}

[1] https://src.fedoraproject.org/rpms/python-progress/pull-request/2

Comment 1 Petr Viktorin 2018-10-04 09:53:22 UTC
What about python2 packages that are still needed in Fedora 30, but then become leaves in Fedora 31 and should remove python2 support then?
python2-progress can be removed now, but other packages might need more time.

What about EL7/EL6; should we add the macros there as well?

Comment 2 Igor Raits 2018-10-04 10:24:51 UTC
This should go through FPC first.

In my opinion, this is really useless and it really depends on packagers and not what fedora dictates.

Comment 3 Pavel Raiskup 2018-10-04 10:34:57 UTC
(In reply to Petr Viktorin from comment #1)
> What about python2 packages that are still needed in Fedora 30, but then
> become leaves in Fedora 31 and should remove python2 support then?

Such packages would use those macros later, not yet there.  Of course
anyone can {re,un}define the defaults.

> What about EL7/EL6; should we add the macros there as well?

Yes, into epel-rpm-macros probably - as long as we are unable to patch
redhat-rpm-macros.

Comment 4 Pavel Raiskup 2018-10-04 10:36:28 UTC
s/not yet there/not now/

Comment 5 Pavel Raiskup 2018-10-04 10:41:21 UTC
(In reply to Igor Gnatenko from comment #2)
> This should go through FPC first.

Feel free, I don't see a problem in that.  It is one/two liner, right?

> In my opinion, this is really useless and it really depends on packagers and
> not what fedora dictates.

This macro doesn't dictate anything to packagers, neither I propose you
to make the macro mandatory for all python packages.  Those macros are
meant to give a convenient info about "what this system prefers".

Comment 6 Petr Viktorin 2018-10-04 10:57:34 UTC
But what does "what this system prefers" mean?

Can you just use %fedora? Python 3 is the default in all supported releases.

Comment 7 Pavel Raiskup 2018-10-04 11:20:19 UTC
(In reply to Petr Viktorin from comment #6)
> But what does "what this system prefers" mean?

I thought that we (by default) expect/prefer this "build pattern":

Fedora 27, 28, 29 -->  build python package against Python 3 and Python 2
Fedora 30, 31     -->  build against Python 3
EPEL 6, 7         -->  build against Python 2

.. except for exceptions.  I don't really mind them here, as this is not
meant to cover everything.

> Can you just use %fedora? Python 3 is the default in all supported releases.

This proposal is absolutely orthogonal to what "default" python means.

Comment 8 Pavel Raiskup 2018-10-04 11:29:26 UTC
Example to demonstrate "how this could be useful":
https://src.fedoraproject.org/rpms/python-progress/pull-request/4#request_diff

That package is otherwise buildable on el6+, fedora 27+ (et least).

Comment 9 Igor Raits 2018-10-04 11:46:11 UTC
(In reply to Pavel Raiskup from comment #5)
> (In reply to Igor Gnatenko from comment #2)
> > This should go through FPC first.
> 
> Feel free, I don't see a problem in that.  It is one/two liner, right?
> 
> > In my opinion, this is really useless and it really depends on packagers and
> > not what fedora dictates.
> 
> This macro doesn't dictate anything to packagers, neither I propose you
> to make the macro mandatory for all python packages.  Those macros are
> meant to give a convenient info about "what this system prefers".

It's not about what system prefers. It's what FPC says packagers to do (obviously by getting agreements with community).

So if you are not going to drive this effort through FPC, I believe that we can close this bug.

Comment 10 Petr Viktorin 2018-10-04 11:46:48 UTC
The rule for Fedora is roughly:
- build all Python packages against Python 3 only, if you can (e.g. if nothing depends on the py2 bits)
- never remove (sub)packages from a released Fedora version
- if nothing depends on the py2 bits and it's just an importable module, remove py2 *now* (that's a case we have automation for and file bugs for)

There's not much special about Fedora 30, we just started the semi-automated mass removal of leaf py2 packages.

Comment 11 Pavel Raiskup 2018-10-04 12:15:33 UTC
(In reply to Petr Viktorin from comment #10)
> There's not much special about Fedora 30, we just started the semi-automated
> mass removal of leaf py2 packages.

If F30 is not "special", what Fedora version will be?  Because at that
point I want to have my python2 subpackages dropped (not sooner) and at
that point it would be convenient to have those macros in hand.

Comment 12 Petr Viktorin 2018-10-04 13:20:45 UTC
(In reply to Pavel Raiskup from comment #11)
> If F30 is not "special", what Fedora version will be? 

For python2-progress, it's F30 because nothing depends on python2-progress there.

I can't answer for all other packages. There are thousands of them, all different, and I don't see into the future.

Comment 13 Pavel Raiskup 2018-10-04 13:47:03 UTC
I'm probably starting to understand where's the problem of this
discussion and misunderstanding.

We are **not OK** to claim (by trivial macro) on F30 that python2 is
(even if only preferably) "dead".  But at the same time we claim that
packages that hurt noone have to be removed (e.g. python2-progress).
_This_ is just plain wrong.

Unless we are brave enough to really deprecate python 2 runtime, tell
me a single reason to drop trivial packages like python2-progress.
It really depends on nothing, so we don't block anyone from removing
the python2 support.

Rephrasing my request to a more generic feature request:
Please, give me a system-wide "button" that you (Python team) will
control.  I (packager) will prepare my packages for that knob in advance,
and you can push the knob as soon as you really consider that Python 2
is dead.  The %py2_expected_support macro was meant to have this purpose.

Comment 14 Petr Viktorin 2018-10-04 13:58:21 UTC
(In reply to Pavel Raiskup from comment #13)
> I'm probably starting to understand where's the problem of this
> discussion and misunderstanding.
> 
> We are **not OK** to claim (by trivial macro) on F30 that python2 is
> (even if only preferably) "dead".  But at the same time we claim that
> packages that hurt noone have to be removed (e.g. python2-progress).
> _This_ is just plain wrong.

One package does not hurt, but there's thousands of them. We have no better way to filter out ones whose removal doesn't hurt, except go and remove them.

> Unless we are brave enough to really deprecate python 2 runtime, tell
> me a single reason to drop trivial packages like python2-progress.

python2 is deprecated:
https://src.fedoraproject.org/rpms/python2/blob/master/f/python2.spec#_108
But that's about adding new packages, not removing old ones.

> It really depends on nothing, so we don't block anyone from removing
> the python2 support.
> 
> Rephrasing my request to a more generic feature request:
> Please, give me a system-wide "button" that you (Python team) will
> control.  I (packager) will prepare my packages for that knob in advance,
> and you can push the knob as soon as you really consider that Python 2
> is dead.  The %py2_expected_support macro was meant to have this purpose.

So you want one for python2. Should we also make knobs for python2-setuptools, python2-six, python2-numpy, and others that many things depend on? Where's the boundary?

Comment 15 Pavel Raiskup 2018-10-04 14:09:56 UTC
(In reply to Petr Viktorin from comment #14)
> One package does not hurt, but there's thousands of them. We have no better
> way to filter out ones whose removal doesn't hurt, except go and remove
> them.

I can not agree here.  If there was some knob, we can check whether package is
ready for it.  Or what exactly is the problem?

> python2 is deprecated:
> https://src.fedoraproject.org/rpms/python2/blob/master/f/python2.spec#_108
> But that's about adding new packages, not removing old ones.

That is very cool, but you forgot to let me know about this fact by
some RPM macro that I could respect in my packages.

> So you want one for python2. Should we also make knobs for
> python2-setuptools, python2-six, python2-numpy, and others that many things
> depend on? Where's the boundary?

No, no specific knob - but one for all python packages.  Why the
python-setuptools can not be prepared for the knob in advance?  If you
don't want to waste your time _now_ on those, the knob can be ignored
(it doesn't make any difference to the current approach).

Comment 16 Miro Hrončok 2018-10-04 14:57:56 UTC
(In reply to Pavel Raiskup from comment #15)
> That is very cool, but you forgot to let me know about this fact by
> some RPM macro that I could respect in my packages.

This is always true. There is no macro. No new python2 packages go into Fedora.

If you really need a way to query it, here it is:

     %if %{?fedora}

Comment 17 Miro Hrončok 2018-10-04 15:08:17 UTC
Note that we are not trying to fight you. We are trying to explain that while this macro might work for one package it certainly doesn't work for other packages. There are no usuals and no exceptions. It's just difeerent for every package.


If we fail to explain that, we are sorry.

Certainly if we could add a "knob" to all packages and than just turn the off switch at some point, it would be lovely. But it just won't happen, there are packages that will most likely require python2 for many years and removing them from the distro would be too painful (nodejs for example).

Packages need to remove their python2 part if those parts are not needed by other things. This is not something a macro can do.

Comment 18 Miro Hrončok 2018-10-04 15:11:53 UTC
Here's a specific example: If you define the macro in a way that works for python-progress, it would break PyGreSQL, it would break python-cassandra-driver.

It's not that python-progress is "normal" and PyGreSQL/python-cassandra-driver is an exception. They just live in a different area of the dependency graph.

Comment 19 Pavel Raiskup 2018-10-04 17:50:37 UTC
> Note that we are not trying to fight you. We are trying to explain that while this macro might work for one package it certainly doesn't work for other packages.

What packages, and why?

Comment 20 Miro Hrončok 2018-10-04 18:01:16 UTC
(In reply to Pavel Raiskup from comment #19)
> > Note that we are not trying to fight you. We are trying to explain that while this macro might work for one package it certainly doesn't work for other packages.
> 
> What packages, and why?

Packages that are needed for other packages (reason is included).

Comment 21 Pavel Raiskup 2018-10-05 07:44:38 UTC
Frankly I still don't get where the problem is.

You seem to ignore the fact that (a) you are not forced
to use the macro and (b) if you opt-in -- you still can override the
system default very easily (even this would be much nicer then the
%if-%else ugly thing).

If you claim that overriding the defaults I propose is difficult
(it is not) here is something which would make it trivial
(~/.rpmmacros for for F30):

   %py3_build_expected 1
   %fedora_py2_kill_in() %{lua:
   if tonumber(rpm.expand('%fedora')) < tonumber(rpm.expand('%1')) then
     rpm.define('py2_build_expected 1')
   else
     rpm.undefine('py2_build_expected')
   end
   }

Many packages can opt-in to use %pyX_build_expected pair, and those who
have to postpone the kill can opt-in too, but just use
"%{?fedora:%fedora_py2_kill_in 31}".  I fail to see how this could
cause any problems when we don't actually force anyone to use that.
It is just convenient thing for packagers who care to avoid this:

%if 0%{?fedora}
  %bcond_without python3
  %if 0%{?fedora} > 29
    %bcond_with python2
  %else
    %bcond_without python2
  %endif
%else
  %if 0%{?rhel} > 7
    %bcond_with    python2
    %bcond_without python3
  %else
    %bcond_without python2
    %bcond_with    python3
  %endif
%endif

Comment 22 Miro Hrončok 2018-10-05 08:36:53 UTC
When you give it some numbers, it starts to make more sense...

So what about: a macro, that sets with/without python2/3 according not to hardwired magic, but arguments?

Something like:

    %setup_with_pythons -f 30 -e 8

The interface could of course have more or less options...

The point here is: Not magical (you still need to read the docs for options to see what they mean, but at least there is no guessing and if a packager sees the numbers and the result they can figure out what's going on). Universal.

The scope however needs to be determined... ?

Do we only support MAX Fedora version (for python2+3 -> python3) and MIN EL version (for python2 -> python3), or do we want to support wider range of options (max/min for fedora/el for 2/3 = 8 options).

(I still won't create this and document this and support this and fix this when broken and add this to the guidelines, but at least I think this is something I could understand. It could probably live in python-srpm-macros, not to bother redhat-rpm-config.)

Comment 23 Pavel Raiskup 2018-10-05 08:50:48 UTC
(In reply to Miro Hrončok from comment #22)
> So what about: a macro, that sets with/without python2/3 according not to
> hardwired magic, but arguments?

I like this idea; namely because it keeps the rpmbuild
--with{,out}=python{2,3} options for those who want to experiment.

Comment 24 Petr Viktorin 2018-10-05 10:25:09 UTC
That's starting to look good, but the -f and -e options still look quite magical to me.

What do you think about this generalization: let's add a macro, let's call it %bcond, where:

    %bcond NAME DEFAULT

conceptually expands to:

    %if DEFAULT
        %bcond_without NAME
    %else
        %bcond_with NAME
    %endif

which you would then call as:

    %bcond python2 (0%{?rhel} <= 7 || 0%{?fedora} < 30)
    %bcond python3 (0%{?rhel} > 7  || 0%{?fedora})

?

(The word "conceptually" hides a lot details, of course.)

Comment 25 Miro Hrončok 2018-10-05 10:47:34 UTC
Seems like this could be called:

    %bcond_if 

Or

    %bcond_when

Comment 26 Igor Raits 2019-06-07 13:59:18 UTC
I agree that we can have some kind of %bcond_if or something like that... But please open ticket on github.com/rpm-software-management/rpm about that. And then we can backport it.

Comment 27 Pavel Raiskup 2019-06-08 09:28:44 UTC
Huh?  Can you please forward the discussion there, then?  Thank you.


Note You need to log in before you can comment on or make changes to this bug.