Bug 785227 - consider switching to pexpect-u
consider switching to pexpect-u
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: pexpect (Show other bugs)
rawhide
All Linux
unspecified Severity medium
: ---
: ---
Assigned To: Robert Scheck
Fedora Extras Quality Assurance
: FutureFeature
Depends On: 878612
Blocks:
  Show dependency treegraph
 
Reported: 2012-01-27 12:35 EST by Andrew McNabb
Modified: 2013-08-21 03:13 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-21 03:13:42 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
patch to pexpect.spec to use the pexpect-u sources (4.61 KB, patch)
2012-01-27 12:35 EST, Andrew McNabb
no flags Details | Diff

  None (edit)
Description Andrew McNabb 2012-01-27 12:35:19 EST
Created attachment 557909 [details]
patch to pexpect.spec to use the pexpect-u sources

The upstream pexpect project does not seem to show many signs of life.
The last release was in January 2008 (4 years ago), and the last commit was in
October 2010 (over a year ago). However, the pexpect-u fork is active and includes support for Unicode and Python 3.

One possibility would be to create a separate pexpect-u package. Unfortunately, the upstream pexpect-u project uses the name "pexpect" for the Python module. If we were to have a separate pexpect-u package, it would conflict with the pexpect package, which is undesirable. As pexpect-u seems to keep backwards compatibility, and since the pexpect project seems to be inactive, switching to use pexpect-u seems like it might be the best option.

I am attaching a patch that makes the proposed change, and I would appreciate any feedback. Note that the upstream pexpect-u tarball is available at:

http://pypi.python.org/packages/source/p/pexpect-u/pexpect-u-2.5.1.tar.gz

Note that the Fedora ipython package uses pexpect, but the upstream IPython package bundles pexpect-u. Python 3 support in pexpect is required before Fedora can include a python3-ipython subpackage.
Comment 1 Thomas Spura 2012-02-05 11:34:13 EST
I vote for a new review request for pexpect-u, which will be named "python-pexpect" and the normal pexpect can then be retired properly.
(python-pexpect could then provide pexpect for a while, till all the deps have changed their requirement)

This would also fix bug #622648.

Robert, what would you prefer?
Comment 2 Andrew McNabb 2012-02-08 21:48:59 EST
Thomas, I think that sounds like a reasonable approach.
Comment 3 Andrew McNabb 2012-07-06 18:42:05 EDT
It's been a while, so I'm just wondering whether there are any thoughts on this from the pexpect maintainers.
Comment 4 Robert Scheck 2012-07-09 05:23:46 EDT
If somebody wants to create a package review for pexpect-u which then retires
pexpect - please just go for it.
Comment 5 Andrew McNabb 2012-07-11 17:03:49 EDT
(In reply to comment #4)
> If somebody wants to create a package review for pexpect-u which then retires
> pexpect - please just go for it.

I actually think it would be wrong to create a separate "pexpect-u" package. As I mentioned earlier, it would conflict with the "pexpect" package. Is there any reason you would be opposed to having the "pexpect" package simply point at the new upstream?
Comment 6 Andrew McNabb 2012-08-03 15:25:06 EDT
(In reply to comment #5)
> (In reply to comment #4)
> > If somebody wants to create a package review for pexpect-u which then retires
> > pexpect - please just go for it.
> 
> I actually think it would be wrong to create a separate "pexpect-u" package.
> As I mentioned earlier, it would conflict with the "pexpect" package. Is
> there any reason you would be opposed to having the "pexpect" package simply
> point at the new upstream?

Not to be tedious, but it would be very helpful to have a discussion about this issue, especially since pexpect is one of the last dependencies holding up ipython on python3.

Just to clarify: my position is that creating a new "pexpect-u" package would be a bad idea because it would conflict with all of the files from the "pexpect" package. My understanding is that the "pexpect" upstream is unmaintained and that the "pexpect-u" upstream is backwards compatible, even to the point of using the name "pexpect" for its Python package.

Is there any reason that it would be objectionable to update the upstream in the "pexpect" package as in the previously attached patch?
Comment 7 Andrew McNabb 2012-11-20 14:00:44 EST
I have created a package review for a new python-pexpect package that would obsolete the current pexpect package as suggested in comment #4. Any help in reviewing this package would be appreciated. Thank you.
Comment 8 Andrew McNabb 2012-11-21 09:38:01 EST
The python-pexpect package is now reviewed and built. Robert, would you like me to add you as an owner?

What's the process for obsoleting the pexpect package? Should we now switch packages from depending on pexpect to depending on python-pexpect in Fedora 18?
Comment 9 Robert Scheck 2012-11-21 10:36:40 EST
Andrew, which branches do you cover? I could offer to retire the package as
to the guidelines on the corresponding branches.
Comment 10 Robert Scheck 2012-11-21 10:37:39 EST
Btw, I assume your python-pexpect has a Provides and Obsoletes for pexpect, yes?
Comment 11 Andrew McNabb 2012-11-21 12:31:30 EST
(In reply to comment #9)
> Andrew, which branches do you cover? I could offer to retire the package as
> to the guidelines on the corresponding branches.

I'm planning on only submitting it to Fedora 18 and later.

(In reply to comment #10)
> Btw, I assume your python-pexpect has a Provides and Obsoletes for pexpect,
> yes?

Yes, it does.
Comment 12 Andrew McNabb 2012-11-26 11:40:30 EST
The python-pexpect package has now been pushed to Fedora 18 stable, and when I do "yum install pexpect", it installs the python-pexpect package instead. I think it should now be safe to retire the pexpect package in Fedora 18 and rawhide.

Robert, would you like to be a co-maintainer on the python-pexpect package?
Comment 13 Garrett Cooper 2012-12-09 20:11:10 EST
It's unfortunate that I missed this discussion earlier.

pexpect-u is not a drop-in replacement for pexpect. Unfortunately the author (Thomas Kluyver) changed the directory structure of the package and broke people who are used to importing pxssh (a popular module in the package) as "import pxssh". It similarly breaks how people import the pexpect module in general because it's no longer relative to site-packages. This makes depending on pexpect "consistency" across OS distributions unnecessarily hard (especially because at $work we support CentOS, a FreeBSD variant, Ubuntu, etc).

Andrew: I'm going to try to inform the author of this breakage if no one else does sooner, but it would be helpful if you could follow up with this as well and please resolve the issue with pexpect-u before it ends up in Fedora 18.
Comment 14 Andrew McNabb 2012-12-09 22:30:27 EST
It looks like pxssh is still there, it's just that you have to do, "from pexpect import pxssh" instead of "import pxssh". Is this the full extent of the issue? If so, then it might be possible to workaround this in the Fedora package.

I don't quite understand what you mean by, "It similarly breaks how people import the pexpect module in general because it's no longer relative to site-packages." Could you add a bit more detail about the problem?

By the way, if you create an upstream bug report about this, would you please add a link here (or perhaps in a new bug report) to make it easier for me to track progress?
Comment 15 Toshio Ernie Kuratomi 2012-12-11 09:32:08 EST
(In reply to comment #13)
> pexpect-u is not a drop-in replacement for pexpect. Unfortunately the author
> (Thomas Kluyver) changed the directory structure of the package and broke
> people who are used to importing pxssh (a popular module in the package) as
> "import pxssh".

Note: I've struggled with just how similar an API needs to be to consider it a replacement for another module as well.  I've reluctantly come to the conclusion that 100% compatibility is an unfair standard.  In the python world, API changes in third-party modules can happen from release to release.  So a replacement that takes over the module name shouldn't be made to stay more compatible than the original codebase.  Otherwise you unfairly prevent innovation and cleanups once a fork has occurred.

API breaking changes are allowed in Fedora between major releases so introducing this in Fedora 18 (not yet released) is possible according to policy.  If there's any Fedora packages that rely on the old pexpect APIs, they should be fixed (likely by the python-pexpect package maintainer since it's so late in the release cycle).  This could also use a release note so that people who use pexpect with their local scripts will be warned.

For the pxssh incompatibility particularly, this common idiom should suffice to make a script portable between implementations:

  try:
      from pexpect import pxssh
  except ImportError:
      import pxssh
Comment 16 Garrett Cooper 2012-12-11 11:55:49 EST
@Andrew:
1. It's a bit more than that; it affects the pexpect module as well, which is to some degree just as widely (if not more widely used).
2. I tried looking for a bugtracker on bitbucket/PyPI, but none exists, so I'm going to email the author directly.

@Toshio: Perhaps, but the adoption of pexpect-u isn't universal across distros. I would be arguing less against change if this was a core python module being modified across versions (urllib vs urllib2, md5 vs shalib, etc), and there was a backwards compatibility mechanism, but this was a module that aimed at being "portable" in python 2.x and py3k by using 2to3 in distutils, and while it succeeded in the overall goal it flat out failed in terms of backwards API compatibility.

This is less of an issue for "new things", except that this is just going to be a fun quirk developers have to work with going forward (more than a handful of infrastructure scripts would need to be modified at my $work because we use pexpect for some degree of automation), and this insanity is part of the reason why forking becomes an option (Gnome 2 vs Gnome 3 vs whatever Ubuntu has with Unity is just one example). Changing APIs would invalidate instructions posted by others on blogs, websites, etc, as well as require changing packages dependent on pexpect (and maintaining Fedora-specific patches] to function with pexpect-u because POLA has been violated unexpectedly.
Comment 17 Andrew McNabb 2012-12-11 12:37:36 EST
(In reply to comment #16)
> @Andrew:
> 1. It's a bit more than that; it affects the pexpect module as well, which
> is to some degree just as widely (if not more widely used).

Sorry, I need more detail. I have to understand the issue before I can address it.

> 2. I tried looking for a bugtracker on bitbucket/PyPI, but none exists, so
> I'm going to email the author directly.

Sounds good. Keep us posted on what you hear back.

> @Toshio: Perhaps, but the adoption of pexpect-u isn't universal across
> distros.

Since pexpect hasn't been updated in five years, it can't be long before pexpect-u becomes universal across distros. The original pexpect project is dead.

> I would be arguing less against change if this was a core python
> module being modified across versions (urllib vs urllib2, md5 vs shalib,
> etc), and there was a backwards compatibility mechanism, but this was a
> module that aimed at being "portable" in python 2.x and py3k by using 2to3
> in distutils, and while it succeeded in the overall goal it flat out failed
> in terms of backwards API compatibility.

The phrase "flat out failed" seems rather argumentative.

> This is less of an issue for "new things", except that this is just going to
> be a fun quirk developers have to work with going forward (more than a
> handful of infrastructure scripts would need to be modified at my $work
> because we use pexpect for some degree of automation), and this insanity is
> part of the reason why forking becomes an option (Gnome 2 vs Gnome 3 vs
> whatever Ubuntu has with Unity is just one example). Changing APIs would
> invalidate instructions posted by others on blogs, websites, etc, as well as
> require changing packages dependent on pexpect (and maintaining
> Fedora-specific patches] to function with pexpect-u because POLA has been
> violated unexpectedly.

As Toshio pointed out, API changes between versions are quite common in Python packages, and this particular update is only occurring between Fedora releases. The import change seems minor when compared to other API updates that gradually occur over time in any project. Not to deny that the change is an inconvenience, but it's important to keep it in perspective.
Comment 18 Fedora End Of Life 2013-07-03 22:53:51 EDT
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 19 Thomas Spura 2013-08-21 03:13:42 EDT
pexpect is now blocked from rawhide and obsoleted by python-pexpect:
https://fedorahosted.org/rel-eng/ticket/5733

Note You need to log in before you can comment on or make changes to this bug.