Bug 1306804 - RPM-MAVEN-PLUGIN with softlink cannot handle garbled bash function
RPM-MAVEN-PLUGIN with softlink cannot handle garbled bash function
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: maven-shared-utils (Show other bugs)
rawhide
All Linux
unspecified Severity medium
: ---
: ---
Assigned To: Michal Srb
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-11 13:01 EST by Noel McLoughlin
Modified: 2016-02-22 08:19 EST (History)
19 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1172381
Environment:
Fedora-Live-Workstation-x86_64-rawhide-20160208.iso $ cat /etc/fedora-release Fedora release 24 (Rawhide) $ uname -a Linux rawhide 4.5.0-0.rc2.git3.1.fc24.x86_64 #1 SMP Fri Feb 5 16:33:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux $ rpm -q scl-utils scl-utils-2.0.1-8.fc24.x86_64
Last Closed: 2016-02-22 08:19:36 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
RPM SPEC (1.50 KB, text/plain)
2016-02-11 13:32 EST, Noel McLoughlin
no flags Details
Sample maven rpm package to replicate issue on Fedora (1.72 KB, application/x-bzip)
2016-02-21 12:55 EST, Noel McLoughlin
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker BYTEMAN-306 Major Closed master branch fails doclint validation 2017-07-17 13:55 EDT

  None (edit)
Description Noel McLoughlin 2016-02-11 13:01:37 EST
+++ This bug was cloned from Bug #1172381  +++

The scl-utils package is causing a problem in my rawhide environment. The environment function BASH_FUNC_scl() is set by /etc/profile.d/scl-init.sh and is defined in the bash shell environment as follows:

   $ env | tail -10
   BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`
   }
   BASH_FUNC_scl()=() {  local CMD=$1;
    if [ "$CMD" = "load" -o "$CMD" = "unload" ]; then
    eval "module $@";
    else
    /usr/bin/scl "$@";
    fi
   }
   _=/usr/bin/env


Bug #1 is the syntax error shown in output below which breaks a local maven build on rawhide (not seen on ubuntu or rhel6.6). I understand sh invocation obeys different rules to bash invocation according to bash(1) man page.

$ mvn clean build
...
----------- cut -----------------

[INFO] Creating spec file /home/enoemcl/workspace/litp/<ERIClitpcli/ERIClitpcli_CXP9030420/target/rpm/ERIClitpcli_CXP9030420/SPECS/ERIClitpcli_CXP9030420.spec
[WARNING] /bin/sh: scl: line 1: syntax error: unexpected end of file
[WARNING] /bin/sh: error importing function definition for `BASH_FUNC_scl'
[INFO] Building target platforms: noarch-redhat-linux
[INFO] Building for target noarch-redhat-linux
[INFO] Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.1vHlC2
[WARNING] /bin/sh: scl: line 1: syntax error: unexpected end of file
[WARNING] error: Bad exit status from /var/tmp/rpm-tmp.1vHlC2 (%install)
[WARNING]     Bad exit status from /var/tmp/rpm-tmp.1vHlC2 (%install)

----------- cut -----------------


Bug #2 is how BASH_FUNC_xxx is not documented.

$ man -k BASH_FUNC
BASH_FUNC: nothing appropriate.


WORKAROUNDS ATTEMPTED
**********************

Remove the package solves the issue.
# rpm -e scl-utils
Comment 1 Noel McLoughlin 2016-02-11 13:03:17 EST
See also: https://issues.jboss.org/browse/BYTEMAN-306
Comment 2 Florian Weimer 2016-02-11 13:19:20 EST
(In reply to Noel McLoughlin from comment #0)
> The scl-utils package is causing a problem in my rawhide environment. The
> environment function BASH_FUNC_scl() is set by /etc/profile.d/scl-init.sh
> and is defined in the bash shell environment as follows:

In itself, this is not a problem.

> [INFO] Creating spec file
> /home/enoemcl/workspace/litp/<ERIClitpcli/ERIClitpcli_CXP9030420/target/rpm/
> ERIClitpcli_CXP9030420/SPECS/ERIClitpcli_CXP9030420.spec
> [WARNING] /bin/sh: scl: line 1: syntax error: unexpected end of file
> [WARNING] /bin/sh: error importing function definition for `BASH_FUNC_scl'
> [INFO] Building target platforms: noarch-redhat-linux
> [INFO] Building for target noarch-redhat-linux
> [INFO] Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.1vHlC2
> [WARNING] /bin/sh: scl: line 1: syntax error: unexpected end of file
> [WARNING] error: Bad exit status from /var/tmp/rpm-tmp.1vHlC2 (%install)
> [WARNING]     Bad exit status from /var/tmp/rpm-tmp.1vHlC2 (%install)

What does the RPM spec file look like?
Comment 3 Noel McLoughlin 2016-02-11 13:32 EST
Created attachment 1123278 [details]
RPM SPEC

See attached.
Comment 4 Florian Weimer 2016-02-11 13:45:54 EST
What seems to have happened here is that something has truncated the definition of the BASH_FUNC_scl() pseudo-environment variable to the first line.  Somewhere, there is code in your environment which incorrectly serializes those environment variables.

I don't see such code in scl-utils, so I suspect it's some thing else on your system.
Comment 5 Noel McLoughlin 2016-02-11 17:56:16 EST
Here is some debug output from Maven.  I am not sure how Maven invokes execution of the following command but the truncation occurs when Maven executes the command by invoking /bin/sh for *this package*.

[DEBUG] About to execute '/bin/sh -c cd /home/enoemcl/workspace/litp/ERIClitpcli/ERIClitpcli_CXP9030420/target/rpm/ERIClitpcli_CXP9030420/SPECS && rpmbuild -bb --buildroot /home/enoemcl/workspace/litp/ERIClitpcli/ERIClitpcli_CXP9030420/target/rpm/ERIClitpcli_CXP9030420/buildroot --define '_topdir /home/enoemcl/workspace/litp/ERIClitpcli/ERIClitpcli_CXP9030420/target/rpm/ERIClitpcli_CXP9030420' --target noarch-redhat-linux ERIClitpcli_CXP9030420.spec'

[WARNING] /bin/sh: scl: line 1: syntax error: unexpected end of file
[WARNING] /bin/sh: error importing function definition for `BASH_FUNC_scl'

[INFO] Building target platforms: noarch-redhat-linux
[INFO] Building for target noarch-redhat-linux
[INFO] Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.BDOiWD
[WARNING] /bin/sh: scl: line 1: syntax error: unexpected end of file
[INFO] 
[INFO] 
[INFO] RPM build errors:
[WARNING] error: Bad exit status from /var/tmp/rpm-tmp.BDOiWD (%install)
[WARNING]     Bad exit status from /var/tmp/rpm-tmp.BDOiWD (%install)
....

When I remove scl-utils package from Fedora the issue disappers.

[DEBUG] About to execute '/bin/sh -c cd /home/enoemcl/workspace/litp/ERIClitpcli/ERIClitpcli_CXP9030420/target/rpm/ERIClitpcli_CXP9030420/SPECS && rpmbuild -bb --buildroot /home/enoemcl/workspace/litp/ERIClitpcli/ERIClitpcli_CXP9030420/target/rpm/ERIClitpcli_CXP9030420/buildroot --define '_topdir /home/enoemcl/workspace/litp/ERIClitpcli/ERIClitpcli_CXP9030420/target/rpm/ERIClitpcli_CXP9030420' --target noarch-redhat-linux ERIClitpcli_CXP9030420.spec'
[INFO] Building target platforms: noarch-redhat-linux
[INFO] Building for target noarch-redhat-linux
[INFO] Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.5kXnov
[WARNING] + umask 022

I will try more troubleshooting as it seems some specific circumstance triggers the issue on certain maven packaging plugins.  I have commented on https://issues.jboss.org/browse/BYTEMAN-306 as same error is seen.
Comment 6 Noel McLoughlin 2016-02-14 07:17:41 EST
Some software cannot serialize environment containing variables which have the following format during parsing.

XXXXXXX()=() { xxxxxxxxxxxx ;

For example, the maven-rpm-plugin and maven-javadoc-plugin (BYTEMAN-206) are not able to serialize exported functions presented as follows.

  BASH_FUNC_scl()=() { local CMD=$1;   <== ';' causes warnings
      ....

  BASH_FUNC_bob()=() { local a=b;      <== ';' causes warnings
      ...

Experimenting with removing the trailing ';' seems to help.  In my specific case, the command 'mvn clean package' now works correctly.

  BASH_FUNC_scl()=() { if [ "$1" = "load" -o "$1" = "unload" ]; then

  BASH_FUNC_bob()=() { local a=b

Reviewing https://github.com/scopatz/xonsh/issues/208 there is some focus on the ';' character although they claim the bash syntax is incorrect. BTW, I can replicate xone cash issue 208 on my system regardless of the name of the exported function.

   BASH_FUNC_xxx()=() { local a=b;      <== ';' causes warnings

Some software appears to lack test coverage with exported functions.
Comment 7 Noel McLoughlin 2016-02-14 11:00:36 EST
Typo correction: I can replicate xonsh issue 208 on my system.
Comment 8 Noel McLoughlin 2016-02-14 11:53:12 EST
Also, Bash seems to be truncating single-line exported functions when imported into the running shell environment. This appears to be separate bug.

$ head -3 /usr/share/Modules/init/sh 

module() { eval `/usr/bin/modulecmd sh $*`; }     <==  ';' is defined.

$ which module
module ()
{ 
    eval `/usr/bin/modulecmd bash $*`             <= ';' truncated.
}

$ env
...
BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`   <= ';' truncated
}
Comment 9 Florian Weimer 2016-02-14 12:09:07 EST
(In reply to Noel McLoughlin from comment #8)
> Also, Bash seems to be truncating single-line exported functions when
> imported into the running shell environment. This appears to be separate bug.

> $ env
> ...
> BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`   <= ';' truncated
> }

env does not properly encode environment variables when printing them:

abc="$(printf 'ab\nc\n')" env  | head -2
abc=ab
c
$ 

env is not a shell built-in, so this is not the shell's fault.  bash does it correctly:

$ abc="$(printf 'ab\nc\n')" bash -c set  | tail -1
abc=$'ab\nc'
$
Comment 10 Noel McLoughlin 2016-02-21 12:55 EST
Created attachment 1129070 [details]
Sample maven rpm package to replicate issue on Fedora

Investigation suggests MAVEN or JAVA-JDK is truncating BASH_FUNC_scl().  

DISCUSSION

The 'getSystemEnvVars()' method in MAVEN CommandLineUtils class might be at fault: https://github.com/sonatype/plexus-utils/blob/master/src/main/java/org/codehaus/plexus/util/cli/CommandLineUtils.java#L240  

However, MAVEN domentation implicates 'System.getenv()' method in JAVA OPENJDK: https://maven.apache.org/shared/maven-shared-utils/apidocs/org/apache/maven/shared/utils/cli/CommandLineUtils.html#getSystemEnvVars%28%29
   
This garbled variable is mostly an irritation for Maven builds (see BYTEMAN-306).    However, if the POM file for RPM packaging specifies a softlinke (see: http://www.mojohaus.org/rpm-maven-plugin/source-params.html#softlinkSource) then the buildroot environment is corrupted (http://rpm5.org/docs/api/buildroot.html) and rpm building fails.

REPLICATE ISSUE:

Issue is easy to replicate with attached sample rpm maven package.

1. Install Fedora Server 23 or rawhide.
2. yum install maven rpm-build -y
3. useradd bob && password bob && su - bob
4. Extract attached demo_mvn_myapp.tar.bz2 file in $HOME
4. cd $HOME/myapp && maven clean package    # works fine.
5. su - root && yum install scl-utils -y && exit
6. cd $HOME/myapp && maven clean package    # fails, garbled scl()


Need some input from Maven developers as I'm a bit rusty on Maven/Java/Bash.
Comment 11 Florian Weimer 2016-02-21 13:02:42 EST
(In reply to Noel McLoughlin from comment #10)
> The 'getSystemEnvVars()' method in MAVEN CommandLineUtils class might be at
> fault:
> https://github.com/sonatype/plexus-utils/blob/master/src/main/java/org/
> codehaus/plexus/util/cli/CommandLineUtils.java#L240  

I think it's more likely that whatever writes out the script which contains these environment variables serializes them incorrectly.  I checked, and OpenJDK does not garble the value.  It's likely there is a function somewhere which does something like this:

  out.print(e.getKey());
  out.print("=\"");
  out.print(e.getValue());
  out.println('"');

That's quite incorrect even for environment variables which are plain strings.
Comment 12 Noel McLoughlin 2016-02-21 13:36:06 EST
I agree. The maven properties string is not garbled (below).  

---
DEBUG properties used { java.vendor=Oracle Corporation, .... etc ... , 
env.BASH_FUNC_scl()=() {  local CMD=$1;
 if [ "$CMD" = "load" -o "$CMD" = "unload" ]; then
 eval "module $@";
 else
 /usr/bin/scl "$@";
 fi
}, ..... etc .....
---

So I think maven-shared-utils is likely culprit.
Comment 13 Mikolaj Izdebski 2016-02-22 01:15:59 EST
I don't think that RPM Maven plugin is packaged in Fedora, so I assume you are using upstream version. In this case you should report this issue in upstream bug tracker at https://github.com/mojohaus/rpm-maven-plugin/issues/
Comment 14 Noel McLoughlin 2016-02-22 02:52:21 EST
Thanks Mikolaj, Florian.

I raised https://github.com/sonatype/plexus-utils/issues/21 as I think the issue lies there, not rpm maven plugin.
Comment 15 Michal Srb 2016-02-22 04:29:13 EST
Just out of curiosity, which version of RPM Maven plugin are you using Noel?
Comment 16 Mikolaj Izdebski 2016-02-22 04:52:36 EST
From attached reproducer it looks like it's 2.1-alpha-1 (latest version is 2.1.5).
Comment 17 Noel McLoughlin 2016-02-22 04:57:00 EST
Yes, Mikolaj is correct.
Comment 18 Michal Srb 2016-02-22 05:03:35 EST
Hmm... version 2.1-alpha-1 was released on 25-May-2010. Could you please try with latest version of the plugin?
Comment 19 Noel McLoughlin 2016-02-22 05:15:30 EST
After fresh install of fedora, the first invocation of 'maven' pulls down the configured plugins from repos.  So it is default behaviour to use 2.1-alpah-1
Comment 20 Noel McLoughlin 2016-02-22 05:29:23 EST
Is there a correct way to update maven plugins? Or do I manually download and update $HOME/.m2/repository/ with later plugin?

Mvn manpage is not helpful and '-up' (update plugins) is depreciated.
Comment 21 Mikolaj Izdebski 2016-02-22 05:36:38 EST
(In reply to Noel McLoughlin from comment #20)
> Is there a correct way to update maven plugins? Or do I manually download
> and update $HOME/.m2/repository/ with later plugin?

In your POM file (myapp/mytest/pom.xml) you specified plugin version as 2.1-alpha-1. You can change it to 2.1.5 to use the latest version.
Comment 22 Noel McLoughlin 2016-02-22 05:45:09 EST
Hi Mikolaj,

Great, that is the root cause. I updated pom.xml to specify 2.1.5 and added the following line to fix an error.

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

My colleagues use ubuntu with maven so did not see this issue & I do not know maven well so did not see this.

thanks
Noel

Note You need to log in before you can comment on or make changes to this bug.