Bug 496791

Summary: Error parsing $(...)
Product: [Fedora] Fedora Reporter: Michal Hlavinka <mhlavink>
Component: mkshAssignee: Robert Scheck <redhat-bugzilla>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: rawhideCC: redhat-bugzilla, tg
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 484348 Environment:
Last Closed: 2013-03-18 18:33:18 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Michal Hlavinka 2009-04-21 04:13:40 EDT
Description of problem:
Consider the following script:

#!/bin/mksh
# Comment containing '
VAR=$(
echo a
# Comment containing '
)
echo $VAR

The script is syntactically correct but mksh throws an error executing it.

Version-Release number of selected component (if applicable):
mksh-37b-1.fc10

How reproducible:
always

Steps to Reproduce:
1. run the script above
  
Actual results:
./test.sh[9]: no closing quote

Expected results:
a

Additional info:
ksh and bash are able to run the script without error.  No error appears when the $(...) command substitution is replaced with `...`.  I think there's a bug in parsing the content in the round brackets: the '\'' character is treated as special even when it appears in a comment (without '\'' it works).
Comment 1 Robert Scheck 2009-04-21 04:24:49 EDT
Michal, thanks for your report.

I'm not sure, whether this is really a bug report, thus I'm Cc'ing upstream of
mksh. Thorsten will be able to clarify this.
Comment 2 Michal Hlavinka 2009-04-21 04:46:28 EDT
(In reply to comment #1)
> Michal, thanks for your report.
> 
> I'm not sure, whether this is really a bug report, thus I'm Cc'ing upstream of
> mksh. Thorsten will be able to clarify this.  

Why this should not be bug? Mksh produces error for syntactically correct script.

FYI:
bash and ksh works
zsh was recently fixed (see bug #484348)
Comment 3 Thorsten Glaser 2009-04-21 05:40:07 EDT
I think this counts as bug. Thanks for reporting.
Note that this problem was inherited from pdksh.

I will try to fix it upstream when I have got the time.
Comment 4 Robert Scheck 2009-04-21 16:09:10 EDT
Thorsten, thank you for answering. Michal, I didn't know, whether $() really
has to work or whether it's a bash/ksh/zsh only thing. Fix will be reflected
to all active Fedora/EPEL branches with the next mksh release or similar.
Comment 5 Thorsten Glaser 2009-04-22 12:25:40 EDT
Commit ID:      10049EF448F5F89A278
CVSROOT:        /cvs
Module name:    (multiple)
Changes by:     tg@herc.mirbsd.org      2009/04/22 16:25:15 UTC

Modified files:
        bin/mksh       : mksh.1
        src            : mksh.hts

Log message:
address RedHat BZ#496791: fix currently not possible, because the code
which escapes $(…) content does not know if an imbedded ‘#’ is a comment
leader or something else like “16#foo”, “${#bla[*]}”, “${foo#bar}”, &c.
and the comment skip code does not know about nesting beforehand
⇒ document this problem in the place where it already documents that
  the current code does not properly handle nested $($(…)) expressions

patches welcome

To generate a diff of this changeset, execute the following commands:
cvs -R rdiff -kk -upr1.157 -r1.158 src/bin/mksh/mksh.1
cvs -R rdiff -kk -upr1.226 -r1.227 www/src/mksh.hts
Comment 6 Thorsten Glaser 2009-04-22 12:26:06 EDT
This means: I cannot fix this bug, while I agree it being one.
Comment 7 Bug Zapper 2009-11-18 04:13:12 EST
This message is a reminder that Fedora 10 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 10.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '10'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 10's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 10 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 8 Bug Zapper 2010-03-15 08:32:39 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 13 development cycle.
Changing version to '13'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 9 Thorsten Glaser 2010-09-05 15:06:45 EDT
I’ve got a workaround:

tg@blau:~ $ cat x
#!/bin/mksh
# Comment containing '
VAR=$(eval $(cat)) <<"EOF"
echo a
# Comment containing '
EOF
echo $VAR
tg@blau:~ $ mksh x
a

Same thing is now documented in the manual page, BUGS section:

     Some parts of the parser are not recursive; things like the following ex-
     ample will fail because of the parenthesis asymmetry (RedHat BZ#496791).
     A workaround exists; use 'x=$(cat) <<"EOF"' instead if you merely want to
     slurp the string.

           x=$(case $foo in bar) echo $bar ;; *) echo $baz ;; esac)
           # above fails; below is the workaround
           x=$(eval $(cat)) <<"EOF"
           case $foo in bar) echo $bar ;; *) echo $baz ;; esac
           EOF

This may not be a proper/real fix, but unless there will be a
rewrite of a lot of internals (not just the parser; these things
appear to be even more intertwined with each other than I originally
thought) this is the best I can do…
Comment 10 Thorsten Glaser 2011-03-05 21:34:01 EST
I’ve just committed code that fixes this bug, in the most detailed
way possible by parsing command substitutions recursively, then
inserting a tree dump into place.

This is still considered EXPERIMENTAL code, though, and as such
should not be backported or used in production systems (yet).
Comment 11 Thorsten Glaser 2011-03-13 12:53:17 EDT
As I suspected, this change (which rewrote a third of the lexer and some of the parser) needs quite some others, including fixes to the tree code (some decades old pdksh heritage) and rewrites of another half or so of the lexer to work well. I’m still trying to squash all the bugs in mksh-current, FWIW. But we have extensive testcases now, so backporting can, while difficult, be validated afterwards.
Comment 12 Robert Scheck 2013-03-18 18:33:18 EDT
This has been solved for quite a while now.