Created attachment 1241368 [details]
If iso8859-x characters are found in certain positions of
input, ksh parsing may get confused.
The problem is that the parser frequently reads a byte
with fcmbget() to "peek" the next input character, and then
then calls fcseek(-LEN) where LEN is the amount of bytes read,
to reset the input.
The problem is that _fcmbget() has an local static buffer to
compose multibyte characters, and fcseek() does not know about
The code is far more complex than just needing to make the
"compose buffer" in _fcmbget() file static, make fcseek() a
The proposed patch works for the test case where it causes the
problem being reported, as well as for utf8 characters, encoding
Test cases follow with explanations, as attachments.
*Note* that this patch right now is an RFE, as it might be
required more tests to validate it.
Created attachment 1241369 [details]
# Need to use bash, or ksh with proposed patch
$ bash iso.sh > a.sh
$ ksh -x a.sh 2>&1 | tail -5
a.sh: line 8596: ": invalid variable name
with ksh with proposed patch it works.
Created attachment 1241371 [details]
This example is just to validate that with or without the
proposed patch, the output is the same, for example:
$ bash /tmp/utf.sh > a.sh
$ tail -3 a.sh
$ ksh -x a.sh > old.sh 2>&1
$ arch/linux.i386-64/src/cmd/ksh93/ksh -x a.sh > new.sh 2>&1
$ tail -3 old.sh
$ diff -u old.sh new.sh
I tried to reproduce this bug on a vanilla rhel-7.3 system, but I did not get the error 'a.sh: line 8596: ": invalid variable name'.
Here is the output from my system :
[0 root@qeos-82 bug1413716]# rpm -q ksh
[0 root@qeos-82 bug1413716]# cat iso.sh
for i in $(seq 1 65536); do
[0 root@qeos-82 bug1413716]# bash iso.sh > a.sh
[0 root@qeos-82 bug1413716]# ksh -x a.sh 2>&1 | tail -5
[0 root@qeos-82 bug1413716]#
Am I missing something in the reproducer steps ?
Somehow it got converted to utf8. Make sure
the iso.sh file has iso8859-1 characters.
For example, the "ç" character, in utf8 ksh
shows it embedded in strings as "\u[e3]", but
if in iso8859-1 it shows as "\xe3".
The original problem is due to a system that
creates ksh scripts dynamically, but uses iso
Created attachment 1241680 [details]
I think it was bugzilla that converted it because I selected text mode...
Thanks! I am able to reproduce it with attachment from comment 5.
Upstream discussion http://lists.research.att.com/pipermail/ast-users/2017q1/004806.html
Patch mentioned in comment 12 breaks if we increase size of input file to ksh by increasing length of loop in iso.sh.