Bug 508383 - trouble with utf-8 characters in xemacs
trouble with utf-8 characters in xemacs
Status: CLOSED DEFERRED
Product: Fedora
Classification: Fedora
Component: xemacs (Show other bugs)
11
All Linux
low Severity medium
: ---
: ---
Assigned To: Jerry James
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-06-26 16:12 EDT by jonathan baron
Modified: 2010-04-27 11:45 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-04-27 11:45:31 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
illustration of font problem (320 bytes, text/plain)
2009-06-26 17:08 EDT, jonathan baron
no flags Details

  None (edit)
Description jonathan baron 2009-06-26 16:12:31 EDT
Description of problem:
Instead of beginning quotes and end quotes in deja vu sans mono,
I get small signs like << and >>.  Other symbols like ' are also
distorted.

Version-Release number of selected component (if applicable):
21.5

How reproducible:
always

Steps to Reproduce:
1. set font to -misc-dejavu sans mono-medium-r-*-*-*-220-*-*-*-*-iso10646-1
2. load some text with "smart quotes"
3.
  
Actual results:
smart quotes displayed as small, incorrect characters

Expected results:
should like the way they do in gnome-terminal or emacs,
both of which display correct characters


Additional info:

I have been thinking that the problem is failure to display utf-8 correctly at all.  The troublesome characters tend to be those that take 2 or 3 bytes.  The directions for getting xemacs to display utf-8 say to include the following in the init.el file:

(require 'un-define)
(set-coding-priority-list '(utf-8))
(set-coding-category-system 'utf-8 'utf-8)

But the first command causes an error because the un-define package is not part of either xemacs-packages or xemacs-packages-extra.  It is part of Mule-UCS.  To get Mule-UCS, I went to the xemacs web page and installed the xemacs-mule-sumo and xemacs-mule-sumo-2009-02-17 packages by hand.  These did contain Mule-UCS, but when I tried the three lines above, first it didn't load anything.  Then I added the following line:

(load "/usr/local/lib/xemacs/mule-packages/lisp/mule-ucs/un-define.elc")

and I got an error message that this package was incompatible with my version of xemacs.

At that point, I decided to report this bug, because it might be that there is something preventing the Fedora version of xemacs from loading anything from Mule-UCS.

On the other hand, it may be that the problem has nothing to do with Mule-UCS.  After all, xemacs is displaying SOMETHING (just the wrong thing).  But, on the other hand again, the problem is not in the font itself, which works fine everywhere else, including emacs.  (Don't ask why I don't switch to emacs.  There is a reason.)
Comment 1 Jerry James 2009-06-26 16:34:21 EDT
Where did you find those directions?  They are for XEmacs 21.4, which had only partial and somewhat buggy support for reading Unicode.  Version 21.5 should read Unicode files out of the box.  Can you restore the RPM packages and see if you can reproduce the problem with "xemacs -vanilla"?  If you can, would you please attach a file that demonstrates the problem?
Comment 2 jonathan baron 2009-06-26 17:08:30 EDT
Created attachment 349607 [details]
illustration of font problem
Comment 3 Jerry James 2009-06-26 17:19:05 EDT
Where did you get that font?  I've got every single package available in the F-11 repository with the string "dejavu" in the name installed on my box, and I get this:

$ xlsfonts -fn '-misc-dejavu sans mono-medium-r-*-*-*-220-*-*-*-*-iso10646-1'
xlsfonts: pattern "-misc-dejavu sans mono-medium-r-*-*-*-220-*-*-*-*-iso10646-1" unmatched

In fact, it looks like none of the dejavu fonts are available via the legacy font system anymore.  They appear to be fontconfig only at this point.
Comment 4 jonathan baron 2009-06-26 18:20:35 EDT
(In reply to comment #3)
> Where did you get that font?

baron@bloch /etc/X11/fontpath.d/dejavu # rpm -qf DejaVuSansMono.ttf 
dejavu-sans-mono-fonts-2.29-2.fc11.noarch
Comment 5 Jerry James 2009-06-26 18:38:22 EDT
So are you running ttfmkdir on system font dirs to make TrueType fonts available to the legacy font system?  You appear to have done something above and beyond what Fedora provides to get access to a font that doesn't display correctly.

I have tried several of the legacy fonts that are available in Fedora, and so far all of them have displayed the left and right quotes correctly in XEmacs.

The << and >> characters you mentioned in comment #1 are called guillemets, by the way, and are the quotation marks symbols used in French writing (as well as a number of other languages).  I wonder if you're having some kind of locale problem, rather than a font problem.
Comment 6 jonathan baron 2009-06-26 19:00:51 EDT
(In reply to comment #5)
> So are you running ttfmkdir on system font dirs to make TrueType fonts
> available to the legacy font system?  You appear to have done something above
> and beyond what Fedora provides to get access to a font that doesn't display
> correctly.

I did not do anything special.  I even checked another computer that had a relatively fresh install (actually upgraded from FC9, but that is the best I can do), and it was the same.  All I did was install the dejavu fonts.

> I have tried several of the legacy fonts that are available in Fedora, and so
> far all of them have displayed the left and right quotes correctly in XEmacs.
> 
> The << and >> characters you mentioned in comment #1 are called guillemets, by
> the way, and are the quotation marks symbols used in French writing (as well as

Thanks.  The other problem, though, is that many of these characters are TINY, as well as incorrect.

> a number of other languages).  I wonder if you're having some kind of locale
> problem, rather than a font problem.  

from env:
LANG=en_US.utf8

Do you know some other font that makes a clear distinction between l and 1?  Bitstream vera sans does make this distinction but has the same problem with quotes, etc.  For the work I do, this is crucial.  I'm happy to switch fonts if there is some other one.

p.s. I meant eta, not chi, in my earlier comment.
Comment 7 jonathan baron 2009-06-26 21:34:04 EDT
(In reply to comment #5)

> The << and >> characters you mentioned in comment #1 are called guillemets, by
> the way, and are the quotation marks symbols used in French writing

I looked more carefully.  The problem was that they are so small.  They are actually the correct symbols, but they were too small for me to notice that unless I took my glasses off and practically pressed my nose to the screen.  (I'm nearsighted.)

So it is clear that the problem is NOT what I originally thought.  It is not the failure to do unicode.  It is purely a size problem.

If nobody else is having this problem, maybe it isn't worth pursuing.  I have been living with it for years.  (I have a script that converts these characters to more standard ones.  The trouble is that I can't keep up with Microsoft, which keeps inventing new ones.  Mostly these result from using antiword on doc files.)
Comment 8 jonathan baron 2009-07-01 12:41:41 EDT
I also reported this to the xemacs bug reporting system, and received very helpful replies from Stephen Turnbull.  Apparently one problem is that the Fedora version of xemacs is compiled without Xft.  There may be a reason for that, but, if so, it would be nice then to have a second version WITH Xft.  Eventually I will try compiling from source, but I don't have time at the moment.

The discussion is at 
xemacs-tracker@calypso.tux.org
and the number of the issue is 536.  You need to register to see this, I think.

Apparently there IS no other monospace font that is anywhere as nice as dejavu sans mono or vera sans mono (which has the same problem), so this might be an issue worth pursuing after all.
Comment 9 jonathan baron 2009-07-01 12:43:00 EDT
Sorry, the url is
http://tracker.xemacs.org/XEmacs/its/issue536
Comment 10 jonathan baron 2009-07-02 14:13:47 EDT
I tried compiling from the srpm, with
rpmbuild -bb --with-xft xemacs.spec
and it stopped in the middle with an error that I cannot figure out.  The last few lines on my screen were:

/usr/bin/ld: Dwarf Error: Offset (7110) greater than or equal to .debug_str size (1804).
font-mgr.o: In function `Ffc_config_set_rescan_interval':
/home/baron/rpmbuild/BUILD/xemacs-21.5.28/src/font-mgr.c:926: undefined reference to `FcConfigSetRescanInverval_REPLACE_BY_FcConfigSetRescanInterval'
font-mgr.o: In function `Ffc_config_get_rescan_interval':
/home/baron/rpmbuild/BUILD/xemacs-21.5.28/src/font-mgr.c:912: undefined reference to `FcConfigGetRescanInverval_REPLACE_BY_FcConfigGetRescanInterval'
collect2: ld returned 1 exit status
make[1]: *** [temacs] Error 1
make[1]: Leaving directory `/home/baron/rpmbuild/BUILD/xemacs-21.5.28/src'
make: *** [src] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.PidvIz (%build)

Looking at /var/tmp/... gave me no clue that I could decipher.
Comment 11 Jerry James 2009-07-02 14:35:04 EDT
In reply to comment #10, that appears to be fallout from a bug in earlier versions of libfontconfig.so, where FcConfigGetRescanInterval was misspelled FcConfigGetRescanInverval.  XEmacs has code to detect the misspelling and work around it.  Apparently the workaround misfired here.  I'll have a look and see if I can figure out how to fix the workaround.
Comment 12 jonathan baron 2009-07-03 10:29:59 EDT
(In reply to comment #11)
> In reply to comment #10, that appears to be fallout from a bug in earlier
> versions of libfontconfig.so, where FcConfigGetRescanInterval was misspelled
> FcConfigGetRescanInverval.  XEmacs has code to detect the misspelling and work
> around it.  Apparently the workaround misfired here.  I'll have a look and see
> if I can figure out how to fix the workaround.  

It seems to be in xemacs-21.5-28/src/font-mgr.c.
I tried just changing Inverval to Interval in that file, and then everything seemed to work, step by step, until the last step, which was
rpmbuild -bb --short-circuit --with xft xemacs.spec
when I got (after much encouraging output):

/home/baron/rpmbuild/BUILD/xemacs-21.5.28/src/font-mgr.c:926: undefined reference to `FcConfigSetRescanInverval_REPLACE_BY_FcConfigSetRescanInterval'
font-mgr.o: In function `Ffc_config_get_rescan_interval':
/home/baron/rpmbuild/BUILD/xemacs-21.5.28/src/font-mgr.c:912: undefined reference to `FcConfigGetRescanInverval_REPLACE_BY_FcConfigGetRescanInterval'
collect2: ld returned 1 exit status
make[1]: *** [temacs] Error 1
make[1]: Leaving directory `/home/baron/rpmbuild/BUILD/xemacs-21.5.28/src'
make: *** [src] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.RbHyiF (%build)

I don't know C. I'm giving up and hoping that you/someone else can fix this.
Comment 13 Bug Zapper 2010-04-27 11:16:54 EDT
This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '11'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 11's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 11 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 14 Jerry James 2010-04-27 11:20:53 EDT
Argh, this completely fell off my radar for months.  Jonathan, are you still having this problem?  I'll also note that I plan to create an xemacs-xft package in the very near future, which may help.
Comment 15 jonathan baron 2010-04-27 11:40:03 EDT
(In reply to comment #14)
> Argh, this completely fell off my radar for months.  Jonathan, are you still
> having this problem?  I'll also note that I plan to create an xemacs-xft
> package in the very near future, which may help.    

Sorry.  But I have now switched to emacs because it does anti-aliasing (and for other reasons).  I just tried xemacs, and I could not find an example to test it on.
Comment 16 Jerry James 2010-04-27 11:45:31 EDT
Thanks for checking.  In that case, I'm going to close this bug report.

Note You need to log in before you can comment on or make changes to this bug.