Bug 205859 - iso-transl functionality (C-x 8) is broken
iso-transl functionality (C-x 8) is broken
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: emacs (Show other bugs)
5
All Linux
medium Severity medium
: ---
: ---
Assigned To: Chip Coldwell
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-09-08 21:05 EDT by deanm
Modified: 2007-11-30 17:11 EST (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-11-14 16:22:42 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
output of "C-h v iso-transl-char-map" w/in emacs (4.09 KB, text/plain)
2006-11-02 15:49 EST, deanm
no flags Details
don't set keyboard-coding-system when running under X (576 bytes, patch)
2006-11-09 14:58 EST, Chip Coldwell
no flags Details | Diff

  None (edit)
Description deanm 2006-09-08 21:05:29 EDT
Description of problem: 
Entering latin-1 characters using the C-x 8 <character>
machanism (iso-transl)  locks emacs and CPU usage goes to 100%.
C-g works to get out of the lockup.


Version-Release number of selected component (if applicable):
24.4.1  (rpm -q emacs: emacs-21.4-14)


How reproducible:
Just type (for example) C-x 8 ! in a writable file and emacs will start looping.

Steps to Reproduce:
1. see above
2.
3.
  
Actual results:
see above

Expected results:
The latin-1 character ¡ should be inserted in the file


Additional info:
The self-inserting command M-x iso-transl-inverted-exclamation-mark
has the same problem.  So do all the other iso-transl characters.

I have the font on my system since I can insert the character into
an emacs file by other methods.
Comment 1 Chip Coldwell 2006-11-02 13:42:03 EST
I am not able to reproduce this on my system.  What are the contents of your
.emacs file?

Chip
Comment 2 deanm 2006-11-02 14:39:33 EST
Hi Chip.

I removed .emacs and started emacs with -q in my tests.  Problem
persisted.  I also removed the couple of (unrelated) foreign .elc
files just in case.  No change.

I asked on the fedora list for someone to try it and they were unable
to reproduce either.  On the other hand I asked someone here at work
who has (according to him) a vanilla FC5 and the bug occurred.

I will help find the problem if you will direct me in debugging this.
I'm pretty sure it is a bone fide bug.  (I'll be embarrassed if
I'm wrong :-) Please give me next steps and I'll try them and report
the results either here or in private email between us.

Dean
Comment 3 Chip Coldwell 2006-11-02 15:12:52 EST
What does "C-h v iso-transl-char-map" show you?

Chip
Comment 4 deanm 2006-11-02 15:49:41 EST
Created attachment 140184 [details]
output of  "C-h v iso-transl-char-map" w/in emacs
Comment 5 deanm 2006-11-02 15:52:29 EST
My addition comments in addition to the attachment did not get sent.
Here they are:

The output of "C-h v iso-transl-char-map"
is in the attachment.  It was done on a fresh emacs -q
after trying C-x 8 ! to get iso-transl to load.

I then tried "M-x iso-transl-inverted-exclamation-mark"
and got the lockup as well as with "M-x iso-transl-plus-or-minus-sign"
(randomly chosen).  On a third self-insert trial,
"M-x iso-transl-i-umlaut" I got this error in the echo area (and no lockup):
After 0 kbd macro iterations: encoded-kbd-self-insert-ccl: Invalid character:
01777777777, 268435455, 0xfffffff

Looking in *Messages* I see this error on those cases that do lock:
After 0 kbd macro iterations: execute-extended-command: Quit

The output of rpm -qa | fgrep -i emacs is:
=>rpm -qa | fgrep -i emacs
emacs-common-21.4-14
emacs-el-21.4-14
emacs-leim-21.4-14
emacs-21.4-14

Hope this info helps.


Comment 6 Chip Coldwell 2006-11-02 16:00:59 EST
(In reply to comment #5)
> On a third self-insert trial,
> "M-x iso-transl-i-umlaut" I got this error in the echo area (and no lockup):
> After 0 kbd macro iterations: encoded-kbd-self-insert-ccl: Invalid character:
> 01777777777, 268435455, 0xfffffff

That's interesting.  One more experiment -- in a *scratch* buffer, evaluate the
Lisp expression

(lookup-key iso-transl-ctl-x-8-map "!")

(i.e. C-x C-e under that expression).  I'm expecting that you will see
"iso-transl-inverted-exclamation-mark" in the echo area.

Chip
Comment 7 Chip Coldwell 2006-11-02 16:15:54 EST
(In reply to comment #5)

> "M-x iso-transl-i-umlaut" I got this error in the echo area (and no lockup):
> After 0 kbd macro iterations: encoded-kbd-self-insert-ccl: Invalid character:
> 01777777777, 268435455, 0xfffffff

"ccl" is the "Code Conversion Language" (lisp/international/ccl.el).

Here's the code for the function mentioned above:
(defun encoded-kbd-self-insert-ccl ()
  (interactive)
  (let ((str (char-to-string last-command-char))
        (ccl (car (aref (coding-system-spec (keyboard-coding-system)) 4)))
        (vec [nil nil nil nil nil nil nil nil nil])
        result)
    (while (= (length (setq result (ccl-execute-on-string ccl vec str t))) 0)
      (dotimes (i 9) (aset vec i nil))
      (setq str (format "%s%c" str (read-char-exclusive))))
    (setq unread-command-events
          (append result unread-command-events))))

Can you post the result of evaluating this

(car (aref (coding-system-spec (keyboard-coding-system)) 4))

In a *scratch* buffer?  I'm expecting "ccl-decode-mule-utf-8".

Chip
Comment 8 deanm 2006-11-02 16:23:49 EST
(In reply to comment #6)
> (In reply to comment #5)
> > On a third self-insert trial,
> > "M-x iso-transl-i-umlaut" I got this error in the echo area (and no lockup):
> > After 0 kbd macro iterations: encoded-kbd-self-insert-ccl: Invalid character:
> > 01777777777, 268435455, 0xfffffff
> 
> That's interesting.  One more experiment -- in a *scratch* buffer, evaluate the
> Lisp expression
> 
> (lookup-key iso-transl-ctl-x-8-map "!")
> 
> (i.e. C-x C-e under that expression).  I'm expecting that you will see
> "iso-transl-inverted-exclamation-mark" in the echo area.
> 
> Chip
> 

That's what I get.
Comment 9 deanm 2006-11-02 16:27:26 EST
(In reply to comment #7)
> (In reply to comment #5)
> 
> > "M-x iso-transl-i-umlaut" I got this error in the echo area (and no lockup):
> > After 0 kbd macro iterations: encoded-kbd-self-insert-ccl: Invalid character:
> > 01777777777, 268435455, 0xfffffff
> 
> "ccl" is the "Code Conversion Language" (lisp/international/ccl.el).
> 
> Here's the code for the function mentioned above:

<snip>

> Can you post the result of evaluating this
> 
> (car (aref (coding-system-spec (keyboard-coding-system)) 4))
> 
> In a *scratch* buffer?  I'm expecting "ccl-decode-mule-utf-8".
> 
> Chip

I get "ccl-decode-mule-utf-8" as you expected.
Comment 10 Chip Coldwell 2006-11-02 16:43:48 EST
(In reply to comment #9)
> 
> I get "ccl-decode-mule-utf-8" as you expected.

We're getting closer.  Try

(symbol-function 'iso-transl-inverted-exclamation-mark)

I expect [161]

Then try "C-q 2 4 1 RET" in a scratch buffer.  It should give you the inverted
exclamation mark.

Chip
Comment 11 Chip Coldwell 2006-11-02 16:51:01 EST
(In reply to comment #10)
> (In reply to comment #9)
> > 
> > I get "ccl-decode-mule-utf-8" as you expected.
> 
> We're getting closer.  Try
> 
> (symbol-function 'iso-transl-inverted-exclamation-mark)
> 
> I expect [161]
> 
> Then try "C-q 2 4 1 RET" in a scratch buffer.  It should give you the inverted
> exclamation mark.

Or maybe not -- if your buffer is set up for UTF-8 keyboard input.  If that
fails, try

"C-x RET c C-q 2 4 1 RET RET"

Chip
Comment 12 Chip Coldwell 2006-11-02 16:52:24 EST
(In reply to comment #11)
> 
> "C-x RET c C-q 2 4 1 RET RET"

Should have been

"C-x RET c iso-8859-1 RET C-q 2 4 1 RET RET"

Chip
Comment 13 deanm 2006-11-02 16:59:53 EST
(In reply to comment #12)
> (In reply to comment #11)
> > 
> > "C-x RET c C-q 2 4 1 RET RET"
> 
> Should have been
> 
> "C-x RET c iso-8859-1 RET C-q 2 4 1 RET RET"
> 
> Chip
> 


Both cases give back ¡ (inverted !)
Comment 14 Chip Coldwell 2006-11-02 17:10:55 EST
Did

(symbol-function 'iso-transl-inverted-exclamation-mark)

give you [161]?

Chip
Comment 15 deanm 2006-11-02 17:27:44 EST
(In reply to comment #14)
> Did
> 
> (symbol-function 'iso-transl-inverted-exclamation-mark)
> 
> give you [161]?
> 
> Chip
> 

Sorry!  Forget to do that one.  Yes, [161].
Comment 16 Chip Coldwell 2006-11-03 11:36:32 EST
I just reproduced the problem on RHEL-4.

Chip
Comment 17 Chip Coldwell 2006-11-06 13:36:27 EST
I have identified the infinite loop.  It can be reproduced by evaluating the
following form:

(progn 
  (setq last-command-char 161)
  (encoded-kbd-self-insert-ccl))

The issue is an interaction between Lisp and C code.  The definition of
encoded-kbd-self-insert-ccl is in lisp/international/encoded-kb.el:

(defun encoded-kbd-self-insert-ccl ()
  (interactive)
  (let ((str (char-to-string last-command-char))
	(ccl (car (aref (coding-system-spec (keyboard-coding-system)) 4)))
	(vec [nil nil nil nil nil nil nil nil nil])
	result)
    (while (= (length (setq result (ccl-execute-on-string ccl vec str t))) 0)
      (dotimes (i 9) (aset vec i nil))
      (setq str (format "%s%c" str (read-char-exclusive))))
    (setq unread-command-events
	  (append result unread-command-events))))

The last form in the let appends '161' to unread-command-events when evaluating
my test form above.  This Lisp variable is referenced in the C function
read_char (defined in src/keyboard.c:2141) in the following if expression:

  if (CONSP (Vunread_command_events))
    {
      c = XCAR (Vunread_command_events);
      Vunread_command_events = XCDR (Vunread_command_events);

      /* Undo what read_char_x_menu_prompt did when it unread
	 additional keys returned by Fx_popup_menu.  */
      if (CONSP (c)
	  && EQ (XCDR (c), Qdisabled)
	  && (SYMBOLP (XCAR (c)) || INTEGERP (XCAR (c))))
	c = XCAR (c);
      
      /* If the queued event is something that used the mouse,
         set used_mouse_menu accordingly.  */
      if (used_mouse_menu
	  && (EQ (c, Qtool_bar) || EQ (c, Qmenu_bar)))
	*used_mouse_menu = 1;
      
      reread = 1;
      goto reread_for_input_method;
    }

This is causing the re-execution of encoded-kbd-self-insert-ccl, which
re-appends 161 to unread-command-events, ad infinitum.

The upstream version of lisp/encoded-kb.el is substantially different; I'm not
confident that a straightforward backport will fix the bug.

And yes, this code is common to RHEL-4 and FC-5.

Chip
Comment 18 Chip Coldwell 2006-11-06 16:41:28 EST
More details:

command_loop_1 calls read_key_sequence (src/keyboard.c:1451)
  read_key_sequence calls read_char (src/keyborad.c:8209)
    read_char consults Vunread_command_events (src/keyboard.c:2199)
    if it is a cons, then return the car and cdr down.
  read_key_sequence gets the value 161 and determines that from the keymap
  that it must execute encoded-kbd-self-insert-ccl (src/keyboard.c:1661)
    encoded-kbd-self-insert-ccl prepends 161 onto unread-command-events

Thus the infinite loop.
Comment 19 deanm 2006-11-06 18:07:19 EST
Nice work Chip!

Not sure (not being a developer) what is meant by
"the upstream version" is substantially different.
Does this mean the "pure" version before RH patches are applied,
or the current (more recent) version? Or both?

Why do some poeple running FC5 see the bug and others don't?

Dean
Comment 20 Chip Coldwell 2006-11-08 10:12:26 EST
(In reply to comment #19)
> Nice work Chip!
> 
> Not sure (not being a developer) what is meant by
> "the upstream version" is substantially different.
> Does this mean the "pure" version before RH patches are applied,
> or the current (more recent) version? Or both?

We ship emacs 21.[3-4] with our distros, since that is the latest "release"
from the Free Software Foundation.  emacs 22 is a work in progress, a pretest
version just came out recently (if you want to try it, you can download rpms
from http://people.redhat.com/coldwell/bugs/emacs/176171/).  When I say
"upstream", I mean emacs 22.  Often, there are bugfixes in emacs 22 that can be
back-ported to fix problems emacs 21.  In this particular case, it doesn't seem
to be possible because the lisp/international/encoded-kb.el file has undergone
substantial incompatible changes.

> Why do some poeple running FC5 see the bug and others don't?

If I knew that, I'd probably have a fix for the bug.  So that's where I'm going
to focus my efforts now.

Chip
Comment 21 Chip Coldwell 2006-11-08 17:11:57 EST
The fundamental difference between a working FC-5 emacs and the broken RHEL-4
emacs is the key binding.  On my working FC-5 emacs,

(key-binding [161])

evaluates to "self-insert-command".  On my broken RHEL-4 emacs, it evaluates to
"encoded-kbd-self-insert-ccl".  I've already described how this enters the
infinite loop above.  Could you please check the value of this form on your
broken FC-5 emacs and verify that it also returns "encoded-kbd-self-insert-ccl"?

Also, please evaluate the following form (or put in your .emacs file and restart):

(setq overriding-local-map
      '(keymap
       (161 . self-insert-command)))

Then run "C-x 8 !" and verify that you get the inverted exclamation mark.

Chip
Comment 22 Chip Coldwell 2006-11-08 17:28:07 EST
This is interesting.  On the broken RHEL-4 emacs,

(lookup-key (current-global-map) [161])

evaluates to "self-insert-command" and

(key-binding [161])

evaluates to "encoded-kbd-self-insert-ccl".  So the bad binding is not in the
global map.

Chip
Comment 23 deanm 2006-11-08 17:41:24 EST
Here's the results w/o my .emacs being involved:

After "emacs -q" and going to the default *scratch* buffer:

;; This buffer is for notes you don't want to save, and for Lisp evaluation.
;; If you want to create a file, visit that file with C-x C-f,
;; then enter the text in that file's own buffer.

(key-binding [161])^J
encoded-kbd-self-insert-ccl

(setq overriding-local-map
      '(keymap
       (161 . self-insert-command)))^J
(keymap (161 . self-insert-command))

(key-binding [161])^J
;;; No output here!!

;; C-x 8 ! executed here:


;; So _this_ now works. However ...
At this point I tried going into a shell (where I live most of the time)
with "M-x shell"

This hung.

Dean
Comment 24 deanm 2006-11-08 17:46:07 EST
Somehow the inverted ! didn't get into  my last message. It shd. have
followed ";; C-x 8 ! executed here:".  In otherwords, it worked.
But the shell hung.
Comment 25 Chip Coldwell 2006-11-09 10:41:32 EST
(In reply to comment #24)
> Somehow the inverted ! didn't get into  my last message. It shd. have
> followed ";; C-x 8 ! executed here:".  In otherwords, it worked.
> But the shell hung.

The overriding-local-map wasn't meant to be a proposed solution; just a
diagnostic.  Here's another one: if you evaluate

(let ((slot (assq 'encoded-kbd-mode minor-mode-map-alist)))
  (if slot
      (setcdr slot nil)))

does the problem go away?

Chip

Comment 26 Chip Coldwell 2006-11-09 10:44:58 EST
(In reply to comment #25)
> (In reply to comment #24)
> > Somehow the inverted ! didn't get into  my last message. It shd. have
> > followed ";; C-x 8 ! executed here:".  In otherwords, it worked.
> > But the shell hung.
> 
> The overriding-local-map wasn't meant to be a proposed solution; just a
> diagnostic.  Here's another one: if you evaluate
> 
> (let ((slot (assq 'encoded-kbd-mode minor-mode-map-alist)))
>   (if slot
>       (setcdr slot nil)))
> 
> does the problem go away?

In fact, an easier way to achieve the same effect is "C-x RET k RET".

Chip
Comment 27 Chip Coldwell 2006-11-09 10:53:35 EST
It looks like the trouble starts in

/usr/share/emacs/site-lisp/site-start.d/lang-coding-systems-init.el

which includes 

  (cond ((equal locale-coding-system 'utf-8)
        ...
        (set-keyboard-coding-system 'utf-8))

Chip
Comment 28 deanm 2006-11-09 13:33:41 EST
Yes, in the scratch buffer:

(let ((slot (assq 'encoded-kbd-mode minor-mode-map-alist)))
  (if slot
      (setcdr slot nil)))^J
nil
;; C-x 8 ! done here:
¡

Comment 29 deanm 2006-11-09 13:54:00 EST
Somewhat off topic, but back when I first discovered the iso-transl bug,
I needed a workaround.  So I tried using C-\ to no avail.
I first thought that was part of the problem.  Poking around, I discovered
in language/misc-lang.el:

;; Is this appropriate?
;; 	   (exit-function
;; 	    . (lambda ()
;; 		(if (and (fboundp 'w32-add-charset-info)
;; 			 (not (boundp 'w32-unicode-charset-defined)))
;; 		    (setq w32-charset-info-alist
;; 			  (delete (assoc "iso10646-1")
;; 				  w32-charset-info-alist)))))
	   (input-method . "rfc1345")	; maybe not the best choice
	   (documentation . "\
This language environment is a generic one for a subset of the Unicode
character set encoded in UTF-8."))
 nil)

input-method used to be "latin-1-prefix".  So in my .emacs I re-define it
that way so I can do C-\ ! C-\.  Anyway, I agree with whoever put the
comment "maybe not the best choice" in the list code.   Any chance this
can be changed so that "latin-1-prefix" is the toggled input method?
Comment 30 deanm 2006-11-09 13:58:53 EST
Sorry, a bit of misinfo in the last message.  I just checked on an older
Mandrake machine running emacs-21.3.2 and it contains the above code in
misc-lang.el.  So something else has changed since then so that
latin-1-prefix is no longer the default "other" (toggled) input method.

I also meant to write c-\ ~! C-\ above for how I must currently get ¡.
Comment 31 deanm 2006-11-09 14:06:21 EST
Chip, this is to let you know that "C-x RET k RET" fixes the problem.
Comment 32 Chip Coldwell 2006-11-09 14:33:31 EST
I now know the answer to your question: "Why do some poeple running FC5 see the
bug and others don't?"  It depends on how you start emacs.  Launch emacs from
the "Applications" menu in the top left corner of the Desktop by following
"Applications->Programming->Emacs Text Editor" (you can also copy this launcher
into the quicklaunch area at the top of the desktop or onto the desktop itself).
 You will not see the bug.  Next, open an xterm (or gnome terminal) and run
"emacs &" at the command prompt.  You will see the bug.  The issue is the value
of the TERM environment variable, as seen by 

/usr/share/emacs/site-lisp/site-start.d/lang-coding-systems-init.el

which contains

  (cond ((equal locale-coding-system 'utf-8)
         (when (member lang '("ja" "ko" "zh"))
           ;; CJK utf-8 locale needs Mule-UCS
           (require 'un-define))
         (set-default-coding-systems 'utf-8)
         (set-terminal-coding-system 'utf-8)
         (let ((term (getenv "TERM")))
           (when (or (equal term "linux")
                     (equal term "xterm"))
             (set-keyboard-coding-system 'utf-8))))

The bug is that it is not possible to determine if emacs is being launched as an
X application or in a terminal (or terminal emulator) merely by examining the
value of the TERM environment variable.

Chip
Comment 33 Chip Coldwell 2006-11-09 14:36:18 EST
(In reply to comment #29)
> Somewhat off topic,

You're right, this is completely off topic.  Since this is upstream code (albeit
moved from lisp/language/misc-lang.el to lisp/language/utf-8-lang.el) you would
have to take it up with the emacs-devel mailing list:

http://lists.gnu.org/mailman/listinfo/emacs-devel

It strikes me as being more of a preference than a bug, and as you observed it
is easy enough to work around it in your site startup files.

Chip
Comment 34 Chip Coldwell 2006-11-09 14:38:43 EST
(In reply to comment #30)
> Sorry, a bit of misinfo in the last message.  I just checked on an older
> Mandrake machine running emacs-21.3.2 and it contains the above code in
> misc-lang.el.  So something else has changed since then so that
> latin-1-prefix is no longer the default "other" (toggled) input method.

Yes, we switched from Latin-1 (ISO 8859-1) to UTF-8 coding.  Latin-1 does not
include support for Asian languages, etc.

Chip
Comment 35 Chip Coldwell 2006-11-09 14:58:57 EST
Created attachment 140821 [details]
don't set keyboard-coding-system when running under X
Comment 36 Chip Coldwell 2006-11-09 15:20:00 EST
I put some test rpms up at

http://people.redhat.com/coldwell/bugs/emacs/205859/

(source in toplevel, binaries in subdirectories by arch).  Could you please
download and verify that they work?

If so, I'll commit this fix for the next quarterly update.

Chip
Comment 37 deanm 2006-11-09 16:10:03 EST
(In reply to comment #32)
> I now know the answer to your question: 
> "Why do some poeple running FC5 see the
> bug and others don't?"  It depends on how you start emacs.  
> Launch emacs from the "Applications" menu in the top left corner
> of the Desktop by following
> "Applications->Programming->Emacs Text Editor" 
<snip>

I experience the same problem when launching Emacs from the menu.
I also launching from the "Run Command" window (Alt-F2).  Same problem.
Dropping into V-terminal and lauching "emacs -nw" I still see the
problem. (In all cases, .emacs has been moved out of the way).

> You will not see the bug.  Next, open an xterm (or gnome terminal) and run
> "emacs &" at the command prompt.  You will see the bug.  The issue is
> the value of the TERM environment variable, as seen by 
<snip>

In no case does the bug go away.  Note: I'm running KDE if it matters.
Comment 38 deanm 2006-11-09 16:32:30 EST
(In reply to comment #36)
> I put some test rpms up at
> 
> http://people.redhat.com/coldwell/bugs/emacs/205859/

I've installed the i386 test binaries:

~=>rpm -qa | fgrep emacs
emacs-el-21.4-16.4
emacs-21.4-16.4
emacs-nox-21.4-16.4
emacs-common-21.4-16.4
emacs-leim-21.4-16.4
emacs-debuginfo-21.4-16.4

The problem appears to be resolved under X.  Lauching from menu, lauching
from Alt-F2, or lauching from an xterm all work properly. Almost there!

The problem persists in a V-term, either lauching "emacs -nw" or "emacs-nox"
which, I suppose, is the same thing.  Note: echo $TERM returns "linux" in
the V-term.  (no surprise).

Given that your comment #32 predicition did not hold up (before the 
rpm install) there's evidently still some mystery here.
Comment 39 Chip Coldwell 2006-11-10 09:55:37 EST
(In reply to comment #37)
> 
> In no case does the bug go away.  Note: I'm running KDE if it matters.

It probably does.  The issue is the setting of the TERM environment variable. 
In GNOME, it defaults to "dumb" when launching emacs directly from the window
manager.  You can verify that your TERM environment variable is something else
under the KDE with "M-x getenv".

I would argue that the variable should be unset in the window manager, since
there is no terminal (or terminal emulator) until you start one.  So both GNOME
and KDE are buggy in this respect.

Chip
Comment 40 deanm 2006-11-10 14:24:58 EST
(In reply to comment #39)
Some results:
FC5: emacs launched from KDE: M-x getenv TERM ==> linux
FC5: emacs launched from xterm comline: M-x getenv TERM ==> xterm
In both cases, M-x shell --> (from bash comline: echo $TERM ==> dumb

Mandrake 10.0: results same as above.
However on the former C-x 8 ! loops forever w/o your patch
and on the latter it works.

> I would argue that the variable should be unset in the window manager, since
> there is no terminal (or terminal emulator) until you start one.
> So both GNOME and KDE are buggy in this respect.

I'm not qualified to argue about what TERM should be,
although since emacs is, among other things, a text editor, there must
be some notion of a "terminal" with appropriate characteristics  involved.
Maybe "dumb" is correct, though emacs certainly doesn't seem to corrrespond
to "dumb" as defined in /etc/termcap.

At all events, the "encoded-kbd-*" stuff added between ver. 21.3.2 (Mandrake)
and 21.4.1 (FC5) has broken behaviour.
Your patch bypasses (I won't say fixes) the problem under X though not
in a Vterm (where under both Mandrake and FC5, TERM == linux; so KDE is simply
doing nothing to the TERM variable.  Don't know if this is correct or not).

You wrote in #32:
"The bug is that it is not possible to determine if emacs is being launched as
an X application or in a terminal (or terminal emulator) merely by
examining the value of the TERM environment variable."

Unless I'm missing something, this does not seem to be THE bug, which
is an infinite loop in the encoded-kbd-* code.
esp. since characters reachable from C-x 8 are renderable
both under X and in a Vterm. (Can we argue that if the keyboard is
encoded then these chars shd. not be rendered?)  Not sure from your earlier
comments if the bug is fixed in the upstream version
of emacs or not. Your patch makes the bug invisible for 95% of emacs users,
I think, so maybe no more time shd. be spent on it. That's up to you.
But as I see it, the bug is still there.  Is starting emacs with
"encoded-kbd-mode" unconditionally set to nil an option? That also
bypasses the problem.  This together
with a note in the Release Notes for FC6 that C-x 8 is broken when
the mode is enabled, plus a bug report to the Emacs developers might
be the proper way to go.

Dean
Comment 41 Chip Coldwell 2006-11-10 14:51:32 EST
(In reply to comment #40)
> (In reply to comment #39)
> Some results:
> FC5: emacs launched from KDE: M-x getenv TERM ==> linux
> FC5: emacs launched from xterm comline: M-x getenv TERM ==> xterm
> In both cases, M-x shell --> (from bash comline: echo $TERM ==> dumb

"dumb" in this case refers to the terminal emulator that emacs provides to the
shell process.

> Is starting emacs with "encoded-kbd-mode" unconditionally set to nil an option?

Yes, you can put

(set-keyboard-coding-system nil)

in your .emacs.

> This together
> with a note in the Release Notes for FC6 that C-x 8 is broken when
> the mode is enabled, plus a bug report to the Emacs developers might
> be the proper way to go.

I haven't closed this bug because I agree with your position that the solution I
have implemented so far is a workaround, not a fix.  A bug report to emacs-devel
is not likely to accomplish much, as they are all preoccupied with emacs 22
which does not have this problem.  I'm still trying to figure out if it is
possible to make the encoded-kb.el work as intended.

Chip
Comment 42 Chip Coldwell 2006-11-10 14:59:22 EST
(In reply to comment #40)
>
> I'm not qualified to argue about what TERM should be,
> although since emacs is, among other things, a text editor, there must
> be some notion of a "terminal" with appropriate characteristics  involved.

No.  In this context, "terminal" refers to the set of cursor-positioning escape
sequences defined in the terminfo/termcap databases and used by, e.g. the
ncurses library to locate text on screen.  If you connect a real terminal, such
as a VT100, to a real serial port, such as /dev/ttyS0, and run emacs on that
terminal, then it will use the TERM environment to determine which escape
sequences to send to position its text on screen.  Similarly, if you run emacs
in a terminal emulator, such as the xterm program, on a pseudo-terminal, such as
/dev/pts/4, it will use escape sequences that are interpreted by the xterm
program to draw within the xterm program.  However, if you are running emacs as
an X client, then it makes direct calls into the X libraries to draw text on
screen and the TERM environment variable is meaningless.  It certainly tells one
nothing about the character encoding used by the keyboard.

Chip
Comment 43 Chip Coldwell 2006-11-10 15:41:14 EST
Date: Fri, 10 Nov 2006 15:37:25 -0500 (EST)
From: Chip Coldwell <coldwell@redhat.com>
To: emacs-devel@gnu.org
Subject: infinite loop in encoded-kb/iso-transl (emacs 21.4)


I've run into an infinite loop using utf-8 keyboard encoding with
iso-transl on emacs 21.4.  To reproduce the problem, run

C-x RET k utf-8

followed by

C-x 8 !

The inverted exclamation mark this is expected to produce is key 161. If one
evaluates

(key-binding [161])

the result is encoded-kbd-self-insert-ccl.  This function executes a
code conversion language (CCL) program, ccl-decode-mule-utf-8, on the
last-command-char, then sets unread-command-events to the resulting
value.  This value is again 161 (octal 241), as can be verified by
evaluating

(let ((str (char-to-string 161))
      (vec [nil nil nil nil nil nil nil nil nil]))
  (ccl-execute-on-string ccl-decode-mule-utf-8 vec str t))

I think the issue here might be that encoded-kb.el installs a minor
mode keymap for encoded-kbd-mode, and iso-transl.el installs a
key-translation-map.  I don't think we should do both.

Note that if you evaluate

(set-keyboard-coding-system nil)

then

(key-binding [161])

evaluates to self-insert-command and

C-x 8 !

works as expected.

I solicit your comments and advice.

Chip

-- 
Charles M. "Chip" Coldwell
Senior Software Engineer
Red Hat, Inc
978-392-2426



_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel
Comment 44 deanm 2006-11-10 17:12:38 EST
Thanks Chip for all your patience and work on this!
I've reverted to the unpatched emacs of FC5 and put
(set-keyboard-coding-system nil)
in my .emacs file.  I'll be intesteresd to see how the Emacs Folks respond.

I'm always available to try stuff if needed.

Dean
Comment 45 Chip Coldwell 2006-11-13 09:40:44 EST
(In reply to comment #44)
> I'll be intesteresd to see how the Emacs Folks respond.

Predictably (this is the only response I've gotten so far):

Date: Fri, 10 Nov 2006 16:43:33 -0500
From: Chong Yidong <cyd@stupidchicken.com>
To: Chip Coldwell <coldwell@redhat.com>
Cc: emacs-devel@gnu.org
Subject: Re: infinite loop in encoded-kb/iso-transl (emacs 21.4)

Chip Coldwell <coldwell@redhat.com> writes:

> I've run into an infinite loop using utf-8 keyboard encoding with
> iso-transl on emacs 21.4.  To reproduce the problem, run
>
> C-x RET k utf-8
>
> followed by
>
> C-x 8 !

I can't reproduce this using Emacs 22, so this was probably fixed at
some point.
Comment 46 Chip Coldwell 2006-11-14 14:31:13 EST
Relevant text from the emacs manual justifying the patch in comment #35

   * If your keyboard can generate character codes 128 (decimal) and up,
     representing non-ASCII characters, you can type those character
     codes directly.

     On a graphical display, you should not need to do anything special
     to use these keys; they should simply work.  On a text-only
     terminal, you should use the command `M-x
     set-keyboard-coding-system' or the variable
     `keyboard-coding-system' to specify which coding system your
     keyboard uses (*note Terminal Coding::). 
Comment 47 Chip Coldwell 2006-11-14 16:22:42 EST
I've decided to close this bug.  I don't think it will be possible to fix
without a major rewrite of encoded-kb.el (such as can be found in emacs 22), and
the workaround seems sane (we shouldn't assume a utf-8 encoded keyboard when
running under a window system).

Chip

Note You need to log in before you can comment on or make changes to this bug.