Bug 1039102 - Can't see unicode chars in plotted images
Summary: Can't see unicode chars in plotted images
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: gnuplot
Version: 19
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Frantisek Kluknavsky
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-06 15:55 UTC by Mihkel Vain
Modified: 2014-12-12 04:20 UTC (History)
7 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2014-12-12 04:20:35 UTC


Attachments (Terms of Use)
No unicode chars in title (37.09 KB, image/png)
2013-12-06 15:55 UTC, Mihkel Vain
no flags Details

Description Mihkel Vain 2013-12-06 15:55:56 UTC
Created attachment 833672 [details]
No unicode chars in title

When I do something like this in Fedora 19 or Fedora 20:

gnuplot> set encoding utf8
gnuplot> set term pngcairo
gnuplot> set title "T\U+FFC3\U+FFB6\U+FFC3\U+FFB6" ← I wrote "Töö", but öö was automatically replaced
gnuplot> set output 'graafik.png'
gnuplot> plot sin(x)
gnuplot> show title
	title is "T", offset at ((character units) 0, 0, 0), using font ",16" textcolor rgb "#6a6a6a"

Eg. the title of my plotted image has no unicode characters. See attached image.

When I do the same in DragonFlyBSD, there are no problems (gnuplot version and other software I use are the same. Same goes to locale settings), so I'm thinking this is a bug of Fedoras version of gnuplot or perhaps a bug in Fedora input method? I don't know...

Comment 1 Frantisek Kluknavsky 2014-11-07 20:43:11 UTC
Hi,

"\U+FFC3\U+FFB6" is not utf-8. These seem like Unicode code points in their weird abstract notation. When I write ö in utf-8 as two escaped 3-digit octal chars

gnuplot> set title "\303\266"

then I get a correct image. Also, writing national characters directly does the right thing for me, they are shown as they should both in terminal screen and in plotted graph. Strangely, in hexadecimal notation "\xc3\xb6" it does not work, it creates "xc3xb6" as 6 characters of plain text.

Gnuplot in Fedora uses readline library. Maybe it's maintainers can tell us more about how "\U+FFC3\U+FFB6" appear and why hexadecimal notation does not get translated.

How do your environment variables look like? Especially $LOCALE and $LC_* ?

Comment 2 Mihkel Vain 2014-11-08 07:55:17 UTC
Thanks for the reply. I have not used gnuplot for a while, partly because that bug. But today I tried it again and I can confirm that this bug still exists.
When I do this:

gnuplot> set title "\303\266"

then I can see ö in title.

My locale is et_EE.utf8

$ locale
LANG=et_EE.utf8
LC_CTYPE="et_EE.utf8"
LC_NUMERIC="et_EE.utf8"
LC_TIME="et_EE.utf8"
LC_COLLATE="et_EE.utf8"
LC_MONETARY="et_EE.utf8"
LC_MESSAGES="et_EE.utf8"
LC_PAPER="et_EE.utf8"
LC_NAME="et_EE.utf8"
LC_ADDRESS="et_EE.utf8"
LC_TELEPHONE="et_EE.utf8"
LC_MEASUREMENT="et_EE.utf8"
LC_IDENTIFICATION="et_EE.utf8"
LC_ALL=

Comment 3 Jan Chaloupka 2014-11-10 07:36:53 UTC
Before typing gnuplot> set title "\xc3\xb6":

#0  el_wgets (el=el@entry=0xa0aaa0, nread=nread@entry=0x7fffffffdbf4) at read.c:484
#1  0x0000003718c2407d in el_gets (el=0xa0aaa0, nread=nread@entry=0x7fffffffdbf4) at eln.c:80
#2  0x0000003718c20b20 in readline (p=0x50e30e "gnuplot> ") at readline.c:420
#3  0x000000000041eaa7 in read_line (prompt=<optimized out>, start=<optimized out>) at ../../src/command.c:2669
#4  0x0000000000421e7c in com_line () at ../../src/command.c:314
#5  0x0000000000415a76 in main (argc=0, argv=0x7fffffffdf78) at ../../src/plot.c:684

After typing gnuplot> set title "\xc3\xb6":

#0  __ct_encode_string (s=0xa229d0 L"set title \"\\xc3\\xb6\"\n", conv=0xa0af78) at chartype.c:79
#1  0x0000003718c24091 in el_gets (el=<optimized out>, nread=nread@entry=0x7fffffffdbf4) at eln.c:82
#2  0x0000003718c20b20 in readline (p=0x50e30e "gnuplot> ") at readline.c:420
#3  0x000000000041eaa7 in read_line (prompt=<optimized out>, start=<optimized out>) at ../../src/command.c:2669
#4  0x0000000000421e7c in com_line () at ../../src/command.c:314
#5  0x0000000000415a76 in main (argc=0, argv=0x7fffffffdf78) at ../../src/plot.c:684

After return from readline:
#0  read_line (prompt=<optimized out>, start=<optimized out>) at ../../src/command.c:2713
#1  0x0000000000421e7c in com_line () at ../../src/command.c:314
#2  0x0000000000415a76 in main (argc=0,	argv=0x7fffffffdf78) at ../../src/plot.c:684

2895        do { 
2896            /* grab some input */ 
2897            if (gp_get_string(gp_input_line + start, gp_input_line_len - start,
2898                             ((more) ? ">" : prompt))
2899                == (char *) NULL)
2900            {

(gdb) print gp_input_line
$56 = 0xa04100 "set title \"\\xc3\\xb6\""

So the line is read as it is, without interpreting.

Doing the same steps for gnuplot> set title "\303\266":

print gp_input_line
$71 = 0xa04100 "set title \"\\303\\266\""

Again, no interpretation.

Is it possible gnuplot has its own intepretation of escape sequences? From this output I don't see otherwise.

Comment 4 Frantisek Kluknavsky 2014-11-14 15:38:32 UTC
Jan, thank you very much, but in the meantime since the original bug report, gnuplot in Fedora switched from readline to libedit, your backtrace seems like coming from libedit. I will look into it further.

Mikhel, I am afraid to ask the obvious, but I have to: which font do you use? It seems like your font does not have that symbol. It is only a guess, i'm asking to be sure.

Comment 5 Mihkel Vain 2014-11-14 16:32:49 UTC
(In reply to Frantisek Kluknavsky from comment #4)
> Mikhel, I am afraid to ask the obvious, but I have to: which font do you
> use? It seems like your font does not have that symbol. It is only a guess,
> i'm asking to be sure.

I use whatever Fedora defaults to. In gnome-terminal I have monospace font (I guess it is actually Libaration Mono) and I don't specify what font to use in plotted image eg. whatever is defaults to.

When I do something like this

set encoding utf8
set term pngcairo font "sans,12"
set title "T\U+FFC3\U+FFB6\U+FFC3\U+FFB6" ## again öö was repalced
set output "graafik.png"
plot sin(x)

Same result.

Comment 6 Frantisek Kluknavsky 2014-11-19 20:22:10 UTC
Libedit + gnuplot + utf8 was discussed upstream and rejected a few years ago. http://sourceforge.net/p/gnuplot/feature-requests/265/?page=1 I did not notice any relevant change in libedit since then.

Even a minimal libedit demo does not behave nicely:
#include <histedit.h>
#include <wchar.h>
int main()
{
        EditLine *handle =  el_init("", stdin, stdout, stderr);
        int cnt = 10; 
        const wchar_t *data = el_wgets(handle, &cnt);
        el_end(handle);
        //printf("Count: %d\n", cnt);
        //for (int i=0; i<cnt; ++i) {
        //        putwc(data[i], stdout);
        //}   
        return 0;
}

Third alternative to libedit and readline is gnuplot builtin. It lacks many nifty editing tricks, but it can handle unicode. I think national characters are a priority.

I tried DragonFly BSD to see how it works there, but do not know how to handle that system. After switching keyboard layout with kbdmap, national characters did not work at all in shell. In gnuplot, \xxx octal notation instead of characters appeared. Might be my failure though.

Comment 7 Frantisek Kluknavsky 2014-11-20 08:42:15 UTC
New build in Rawhide: http://koji.fedoraproject.org/koji/taskinfo?taskID=8188549

Comment 8 Mihkel Vain 2014-11-21 07:54:21 UTC
(In reply to Frantisek Kluknavsky from comment #7)
> New build in Rawhide:
> http://koji.fedoraproject.org/koji/taskinfo?taskID=8188549

Just tested that build on my rawhide virtual machine.

gnuplot> set encoding utf8
gnuplot> set term pngcairo
Terminal type set to 'pngcairo'
Options are ' background "#ffffff" fontscale 1.0 size 640, 480 '
gnuplot> set title "Töö"              # Not that öö was not replaced
gnuplot> set output "graafik.png"  
gnuplot> plot sin(x)
gnuplot> 


IT WORKS! Good work (y)

Comment 9 Fedora Update System 2014-11-21 11:15:55 UTC
gnuplot-4.6.5-4.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/gnuplot-4.6.5-4.fc21

Comment 10 Fedora Update System 2014-11-22 20:19:45 UTC
Package gnuplot-4.6.5-4.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing gnuplot-4.6.5-4.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-15569/gnuplot-4.6.5-4.fc21
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2014-12-12 04:20:35 UTC
gnuplot-4.6.5-4.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.