Description of problem: no readline support in yum - arrow keys output control characters Version-Release number of selected component (if applicable): Name : yum Arch : noarch Version : 3.2.14 Release : 10.fc9 How reproducible: press the arrow keys in "yum shell" Steps to Reproduce: 1. yum shell 2. press arrow keys 3. Actual results: ~# yum shell Loaded plugins: refresh-packagekit Setting up Yum Shell > ^[[A^[[B^[[C^[[D Expected results: Arrow keys should move cursor and recall command history Additional info: Bug also in version 3.2.16-2.fc9 from updates-testing.
Interesting. It used to work.
okay the line causing the problem is in yummain.py: about line 48 sys.stdout = codecs.getwriter(locale.getpreferredencoding())(sys.stdout) working on a solution that doesn't break everything else.
specifically it looks like when sys.stdout becomes a codecs.StreamWriter object it means readline doesn't appear to be getting the commands passed to it
reassigning to the python maintainer. :)
readline is one of those magic modules that lives half inside the interpreter, half outside. Import readline after modifying sys.stdout.
Never mind, there was a flaw in my testing. From the code it looks like readline is not at all compatible with non-FILE* sys.stdout.
yeh, that was my impression from looking at this months ago with my yum hat on :) This was on 2.5.1 though, but looking at it now I think I decided this was the problem (from the builtin module, Python/bltinmodule.c): if (PyFile_AsFile(fin) && PyFile_AsFile(fout) && isatty(fileno(PyFile_AsFile(fin))) && isatty(fileno(PyFile_AsFile(fout)))) { ...and it looked like an easy fix, but now I see that the readline C module requires two FILE * objects, *sigh* ... and hence PyOS_Readline() downwards needs them.
This message is a reminder that Fedora 9 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 9. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '9'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 9's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 9 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
This still do not work in F10 (yum-3.2.21-2.fc10.noarch)
This message is a reminder that Fedora 10 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 10. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '10'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 10's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 10 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
haha, I get to give this to David!:)
Bug still occurs as originally reported under Fedora 12, with "yum-3.2.25-1.fc12.noarch" installed.
Bug still occurs as originally reported under Fedora *13*, with "yum-3.2.27-4.fc13.noarch" installed. It is rather annoying... Besides, this bug is an all-platforms, not only x86-64
Can we get some movement on this bug please? It's been more that 2 years since it was first reported. From what I read of previous comments, it would appear the most immediate solution would be to make a yum-specific mod of the readline C library and package it with yum. Admittedly, this is far from ideal for such a low priority bug, but appears the only way short of pushing what would be a highly controversial change through the readline library for this purpose.
Sorry about the lack of activity here. I still see this bug with yum-3.2.27-4.fc13.noarch and python-2.6.5-9.fc14.x86_64 (local build on my f13 box) Investigating with that version: cli.py has: yumshell = shell.YumShell(base=self) and: yumshell.cmdloop() shell.py has: class YumShell(cmd.Cmd): where "cmd" is Python 2.6's "cmd" module. This has this hook: import readline self.old_completer = readline.get_completer() readline.set_completer(self.complete) readline.parse_and_bind(self.completekey+": complete") "readline" is implemented in C $ gdb --args /usr/bin/python /usr/bin/yum shell (gdb) break initreadline (gdb) frame 9 (gdb) p (PyFrameObject*)$rbp $3 = Frame 0xdf47a0, for file /usr/lib64/python2.6/cmd.py, line 112, in cmdloop So the "import readline" is indeed being called from cmd.py Using a breakpoint confirms that it's getting input via calls to "builtin_raw_input". This contains the conditional mentioned in comment #7, which tests to see if both sys.stdin and sys.stdout are backed by FILE*, rather than being purely Python-level streams. Since sys.stdout is a StreamWriter, this test fails, and "builtin_raw_input" instead uses PyFile_GetLine() on sys.stdin The readline module's initialization installs: PyOS_ReadlineFunctionPointer = call_readline; which is neved called. The signature is: ./Include/pythonrun.h:148:PyAPI_DATA(char) *(*PyOS_ReadlineFunctionPointer)(FILE *, FILE *, char *); and it's defined and used in Parser/myreadline.c, which declares: PyOS_Readline(FILE *sys_stdin, FILE *sys_stdout, char *prompt) Note the signature uses FILE* rather than PyObject*. Ultimately this is used by Modules/readline.c:call_readline to set "rl_instream" and "rl_outstream" globals. Looking at <readline/readline.h> I see: /* The address of the function to call to fetch a character from the current Readline input stream */ extern rl_getc_func_t *rl_getc_function; This is called by readline thusly: c = (*rl_getc_function) (rl_instream); annoyingly, so there doesn't seem to be a robust way of virtualizing the i/o for readline. Similarly, all usage of rl_outstream seems to be direct invocations such as: putc (' ', rl_outstream); fprintf (rl_outstream, "M-"); fflush (rl_outstream); etc So it looks like to fix this requires one of these approaches: (i) to do a major rewrite of GNU readline so that it can accept abstracted file streams (ii) to somehow wrap up a python StreamWriter as a POSIX FILE* It might be possible to use fopencookie to do this (see "man 2 fopencookie") (iii) to avoid the rebinding of sys.stdout mentioned in comment #2 I'm assuming that (iii) isn't an option. My gut feeling right now is that (ii) might be the way forward here. I'm not sure if the resulting FILE* would count as isatty(), but it might at least be closer to working...
There's another option, although I'm not sure how easy it is: iv. The StreamWriter must hit a FILE * eventually, so ""traverse" the data structures until you find the underlying FILE * and then use that for readline. ...I guess this is kind of the opposite of ii, but it's how I'd initially thought of doing it. Another option might be: v. See if editline has a more usable API?
Created attachment 446972 [details] Non-working experimental patch to use fopencookie to obtain a FILE* from a PyObject* I experimented for a while with using fopencookie to obtain a FILE* from a PyObject (as I understand it, this is a GNU extension within glibc, not part of POSIX) Attached is the result. Note that it's not finished; it's missing lots of error handling; it's a (dis)proof-of-concept. I don't think this approach is viable, as the FILE* is regarded as not a tty; Python seems to be multiple levels of checking that the FILE* is really a tty. I'm guessing that this is bulletproofing against a readline restriction.
I thought of a new, compromise approach: As I understand it output during readline is restricted to printing the prompt, and echoing lines that the user has typed (e.g. history lines, auto-completion etc). This seems to me to be primarily "input-related". My guess is that the reason for the rebinding of sys.stdout mentioned in comment #2 is to deal with non-ASCII characters within the output from yum: package metadata etc. So one possible approach might be to fixup YumShell's cmdloop method (currently inherited from cmd.Cmd so that it binds sys.stdout to the original stdout FILE* immediately before the call to raw_input, and binds it back to the streamwrapper immediately after. One way to do this would be to "monkey-patch" raw_input, but it's more robust to implement a custom "cmdloop" method within YumShell, I think. That way, readline's outputs that related to input-handing would be restricted to ASCII but would support readline, andt the outputs from the commands could be UTF-8. This seems similar to James' suggestion in comment #16. Both ideas really seem like fixes to be applied within yum, rather than being appropriate for python as a whole; reassigning back to yum.
Created attachment 446983 [details] Sick patch for yum, to work around python readline braindamage Well, this does roughly what David suggested (hacks sys.stdout in raw_input, and in readline setup) ... in the cmdloop. It's _horrible_, but seems to work for yum. All other users of setup_locale() and Cmd are out of luck. Works by overridding raw_input, which means it'll catch "Yes/no" prompt too ... but will hopefully be harmless (utf-8 yes/no prompt with bad encoding will blow it up now, I think). The other option is basically re-implementing the cmd module, from what I can see (so we can "just" override their raw_input call). That's a lot more code, but a lot less evil. It'd be nicer if python DTRT instead.
This message is a reminder that Fedora 13 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '13'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 13's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 13 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
The problem still occurs using yum-3.2.29-5.fc15 under Fedora 15. This bug should be updated accordingly and kept open.
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
It is fixed for me in F17 (in fact I think it was even fixed in F16).
This message is a notice that Fedora 15 is now at end of life. Fedora has stopped maintaining and issuing updates for Fedora 15. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At this time, all open bugs with a Fedora 'version' of '15' have been closed as WONTFIX. (Please note: Our normal process is to give advanced warning of this occurring, but we forgot to do that. A thousand apologies.) Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, feel free to reopen this bug and simply change the 'version' to a later Fedora version. Bug Reporter: Thank you for reporting this issue and we are sorry that we were unable to fix it before Fedora 15 reached end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged to click on "Clone This Bug" (top right of this page) and open it against that version of Fedora. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping