Bug 448864 - readline (arrow keys) don't work in yum shell
Summary: readline (arrow keys) don't work in yum shell
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: yum
Version: 15
Hardware: x86_64
OS: Linux
low
low
Target Milestone: ---
Assignee: Fedora Packaging Toolset Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2008-05-29 04:42 UTC by Dan Stahlke
Modified: 2014-01-21 23:06 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-08-07 19:38:35 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Non-working experimental patch to use fopencookie to obtain a FILE* from a PyObject* (3.61 KB, patch)
2010-09-13 15:30 UTC, Dave Malcolm
no flags Details | Diff
Sick patch for yum, to work around python readline braindamage (1.22 KB, patch)
2010-09-13 16:24 UTC, James Antill
no flags Details | Diff

Description Dan Stahlke 2008-05-29 04:42:04 UTC
Description of problem:
no readline support in yum - arrow keys output control characters

Version-Release number of selected component (if applicable):
Name       : yum
Arch       : noarch
Version    : 3.2.14
Release    : 10.fc9


How reproducible:
press the arrow keys in "yum shell"

Steps to Reproduce:
1. yum shell
2. press arrow keys
3.
  
Actual results:
~# yum shell
Loaded plugins: refresh-packagekit
Setting up Yum Shell
> ^[[A^[[B^[[C^[[D


Expected results:
Arrow keys should move cursor and recall command history

Additional info:
Bug also in version 3.2.16-2.fc9 from updates-testing.

Comment 1 Seth Vidal 2008-05-29 05:51:04 UTC
Interesting. It used to work.

Comment 2 Seth Vidal 2008-05-29 06:50:49 UTC
okay the line causing the problem is in yummain.py:

about line 48
sys.stdout = codecs.getwriter(locale.getpreferredencoding())(sys.stdout) 

working on a solution that doesn't break everything else.

Comment 3 Seth Vidal 2008-05-29 06:54:44 UTC
specifically it looks like when sys.stdout becomes a codecs.StreamWriter object
it means readline doesn't appear to be getting the commands passed to it

Comment 4 seth vidal 2008-11-07 20:16:05 UTC
reassigning to the python maintainer. :)

Comment 5 Ignacio Vazquez-Abrams 2008-11-07 22:17:40 UTC
readline is one of those magic modules that lives half inside the interpreter, half outside. Import readline after modifying sys.stdout.

Comment 6 Ignacio Vazquez-Abrams 2008-11-07 22:23:30 UTC
Never mind, there was a flaw in my testing. From the code it looks like readline is not at all compatible with non-FILE* sys.stdout.

Comment 7 James Antill 2008-11-10 06:43:52 UTC
yeh, that was my impression from looking at this months ago with my yum hat on :)

This was on 2.5.1 though, but looking at it now I think I decided this was the problem (from the builtin module, Python/bltinmodule.c):

        if (PyFile_AsFile(fin) && PyFile_AsFile(fout)
            && isatty(fileno(PyFile_AsFile(fin)))
            && isatty(fileno(PyFile_AsFile(fout)))) {


...and it looked like an easy fix, but now I see that the readline C module requires two FILE * objects, *sigh* ... and hence PyOS_Readline() downwards needs them.

Comment 8 Bug Zapper 2009-06-10 01:16:37 UTC
This message is a reminder that Fedora 9 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 9.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '9'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 9's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 9 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 9 Jan Hutař 2009-06-10 07:35:04 UTC
This still do not work in F10 (yum-3.2.21-2.fc10.noarch)

Comment 10 Bug Zapper 2009-11-18 12:31:48 UTC
This message is a reminder that Fedora 10 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 10.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '10'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 10's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 10 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 11 James Antill 2009-11-18 14:02:36 UTC
haha, I get to give this to David!:)

Comment 12 Ben Liblit 2009-12-01 19:59:45 UTC
Bug still occurs as originally reported under Fedora 12, with "yum-3.2.25-1.fc12.noarch" installed.

Comment 13 alien_life_form 2010-07-06 07:47:03 UTC
Bug still occurs as originally reported under Fedora *13*, with
"yum-3.2.27-4.fc13.noarch" installed. It is rather annoying... Besides, this bug is an all-platforms, not only x86-64

Comment 14 Trevor Mettam 2010-09-01 04:59:24 UTC
Can we get some movement on this bug please? It's been more that 2 years since it was first reported.

From what I read of previous comments, it would appear the most immediate solution would be to make a yum-specific mod of the readline C library and package it with yum.
Admittedly, this is far from ideal for such a low priority bug, but appears the only way short of pushing what would be a highly controversial change through the readline library for this purpose.

Comment 15 Dave Malcolm 2010-09-10 22:11:47 UTC
Sorry about the lack of activity here.

I still see this bug with yum-3.2.27-4.fc13.noarch and python-2.6.5-9.fc14.x86_64 (local build on my f13 box)

Investigating with that version:
cli.py has:
        yumshell = shell.YumShell(base=self)
and:
        yumshell.cmdloop()

shell.py has:
  class YumShell(cmd.Cmd):
where "cmd" is Python 2.6's "cmd" module.

This has this hook:
     import readline
     self.old_completer = readline.get_completer()
     readline.set_completer(self.complete)
     readline.parse_and_bind(self.completekey+": complete")

"readline" is implemented in C
$ gdb --args /usr/bin/python /usr/bin/yum shell
(gdb) break initreadline
(gdb) frame 9
(gdb) p (PyFrameObject*)$rbp
$3 = 
    Frame 0xdf47a0, for file /usr/lib64/python2.6/cmd.py, line 112, in cmdloop 

So the "import readline" is indeed being called from cmd.py

Using a breakpoint confirms that it's getting input via calls to "builtin_raw_input".

This contains the conditional mentioned in comment #7, which tests to see if both sys.stdin and sys.stdout are backed by FILE*, rather than being purely Python-level streams.

Since sys.stdout is a StreamWriter, this test fails, and "builtin_raw_input" instead uses PyFile_GetLine() on sys.stdin

The readline module's initialization installs:
	PyOS_ReadlineFunctionPointer = call_readline;
which is neved called.

The signature is:
./Include/pythonrun.h:148:PyAPI_DATA(char) *(*PyOS_ReadlineFunctionPointer)(FILE *, FILE *, char *);

and it's defined and used in Parser/myreadline.c, which declares:
PyOS_Readline(FILE *sys_stdin, FILE *sys_stdout, char *prompt)

Note the signature uses FILE* rather than PyObject*.

Ultimately this is used by Modules/readline.c:call_readline to set "rl_instream" and "rl_outstream" globals.

Looking at <readline/readline.h> I see:
/* The address of the function to call to fetch a character from the current
   Readline input stream */
extern rl_getc_func_t *rl_getc_function;

This is called by readline thusly:
	    c = (*rl_getc_function) (rl_instream);
annoyingly, so there doesn't seem to be a robust way of virtualizing the i/o for readline.

Similarly, all usage of rl_outstream seems to be direct invocations such as:
   putc (' ', rl_outstream);
   fprintf (rl_outstream, "M-");
   fflush (rl_outstream);
etc

So it looks like to fix this requires one of these approaches:
(i) to do a major rewrite of GNU readline so that it can accept abstracted file streams
(ii) to somehow wrap up a python StreamWriter as a POSIX FILE*  It might be possible to use fopencookie to do this  (see "man 2 fopencookie")
(iii) to avoid the rebinding of sys.stdout mentioned in comment #2

I'm assuming that (iii) isn't an option.

My gut feeling right now is that (ii) might be the way forward here.  I'm not sure if the resulting FILE* would count as isatty(), but it might at least be closer to working...

Comment 16 James Antill 2010-09-12 13:49:44 UTC
There's another option, although I'm not sure how easy it is:

iv. The StreamWriter must hit a FILE * eventually, so ""traverse" the data structures until you find the underlying FILE * and then use that for readline.

...I guess this is kind of the opposite of ii, but it's how I'd initially thought of doing it. Another option might be:

v. See if editline has a more usable API?

Comment 17 Dave Malcolm 2010-09-13 15:30:18 UTC
Created attachment 446972 [details]
Non-working experimental patch to use fopencookie to obtain a FILE* from a PyObject*

I experimented for a while with using fopencookie to obtain a FILE* from a PyObject  (as I understand it, this is a GNU extension within glibc, not part of POSIX)

Attached is the result.  Note that it's not finished; it's missing lots of error handling; it's a (dis)proof-of-concept.

I don't think this approach is viable, as the FILE* is regarded as not a tty; Python seems to be multiple levels of checking that the FILE* is really a tty.  I'm guessing that this is bulletproofing against a readline restriction.

Comment 18 Dave Malcolm 2010-09-13 15:31:14 UTC
I thought of a new, compromise approach:
As I understand it output during readline is restricted to printing the prompt, and echoing lines that the user has typed (e.g. history lines, auto-completion etc).  This seems to me to be primarily "input-related".

My guess is that the reason for the rebinding of sys.stdout mentioned in comment #2 is to deal with non-ASCII characters within the output from yum: package metadata etc.

So one possible approach might be to fixup YumShell's cmdloop method (currently inherited from cmd.Cmd so that it binds sys.stdout to the original stdout FILE* immediately before the call to raw_input, and binds it back to the streamwrapper immediately after.  One way to do this would be to "monkey-patch" raw_input, but it's more robust to implement a custom "cmdloop" method within YumShell, I think.

That way, readline's outputs that related to input-handing would be restricted to ASCII but would support readline, andt the outputs from the commands could be UTF-8.

This seems similar to James' suggestion in comment #16.

Both ideas really seem like fixes to be applied within yum, rather than being appropriate for python as a whole; reassigning back to yum.

Comment 19 James Antill 2010-09-13 16:24:29 UTC
Created attachment 446983 [details]
Sick patch for yum, to work around python readline braindamage

Well, this does roughly what David suggested (hacks sys.stdout in raw_input, and in readline setup) ... in the cmdloop.

It's _horrible_, but seems to work for yum. All other users of setup_locale() and Cmd are out of luck. Works by overridding raw_input, which means it'll catch "Yes/no" prompt too ... but will hopefully be harmless (utf-8 yes/no prompt with bad encoding will blow it up now, I think).

The other option is basically re-implementing the cmd module, from what I can see (so we can "just" override their raw_input call). That's a lot more code, but a lot less evil.

It'd be nicer if python DTRT instead.

Comment 20 Bug Zapper 2011-06-02 18:30:40 UTC
This message is a reminder that Fedora 13 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 13.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '13'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 13's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 13 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 21 Ben Liblit 2011-06-02 20:22:01 UTC
The problem still occurs using yum-3.2.29-5.fc15 under Fedora 15.  This bug should be updated accordingly and kept open.

Comment 22 Fedora Admin XMLRPC Client 2012-04-27 15:41:06 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 23 Dan Stahlke 2012-06-03 13:29:36 UTC
It is fixed for me in F17 (in fact I think it was even fixed in F16).

Comment 24 Fedora End Of Life 2012-08-07 19:38:39 UTC
This message is a notice that Fedora 15 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 15. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. At this time, all open bugs with a Fedora 'version'
of '15' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we were unable to fix it before Fedora 15 reached end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora, you are encouraged to click on
"Clone This Bug" (top right of this page) and open it against that
version of Fedora.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping


Note You need to log in before you can comment on or make changes to this bug.