Bug 1147043

Summary: Regression from bash function mangling patch breaks "at" jobs
Product: Red Hat Enterprise Linux 6 Reporter: Paul Victor Novarese <pvn>
Component: atAssignee: Tomas Mraz <tmraz>
Status: CLOSED CURRENTRELEASE QA Contact: qe-baseos-daemons
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.6CC: Anthony.Thyssen, bhu, billcarlson, carnil, cwawak, cww, eblake, fkrska, fweimer, hartsjc, heuristik, jherrman, jkurik, john.haxby, jonstanley, koen.schram, lars, mavigano, michele, myllynen, ngalvin, nkim, pdwyer, poulacou2, psklenar, rcyriac, redhat, richard.ryder, rikard.oberg, sardella, seldridg, s.kieske, sreber, tbowling, tmraz, toracat, tscherf, vanhoof, wayne.johnson, wburrows, wnix
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: at-3.1.10-45.el6 Doc Type: Bug Fix
Doc Text:
Due to a security issue fix in Bash, "at" jobs failed to run because the "atd" daemon exported environment variables with an incorrect syntax to the Bash shell running the jobs. With this update, "atd" filters out environment variables that cannot be parsed by the Bash shell, thus allowing the "at" jobs to run properly.
Story Points: ---
Clone Of:
: 1148730 1148731 1148837 1148844 1149276 1149363 1152434 (view as bug list) Environment:
Last Closed: 2015-10-05 14:05:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1075802, 1148730, 1148731, 1149276, 1149277, 1149363, 1152434    
Attachments:
Description Flags
at-skip-names.diff none

Description Paul Victor Novarese 2014-09-26 16:28:18 UTC
Description of problem:  Customer applied shellshock update and now at is broken.


Version-Release number of selected component (if applicable):

at-3.1.10-43.el6_2.1.x86_64
bash-4.1.2-15.el6_5.1.x86_64 (_5.2 also affected)
environment-modules-3.2.10-1.el6_5.x86_64


How reproducible:
100%

Steps to Reproduce:

[root@example ~]# at now
at> ls
at> <EOT>
job 6 at 2014-09-26 12:05

Actual results:

From root  Fri Sep 26 12:05:38 2014
Return-Path: <root>
X-Original-To: root
Delivered-To: root
Subject: Output from your job        6
To: root.com
Date: Fri, 26 Sep 2014 12:05:37 -0400 (EDT)
From: root.com (root)
Status: R

sh: line 37: syntax error near unexpected token `=\(\)\ {\ \ eval\ \`/usr/bin/modulecmd\ bash\ \$\*\`"
"}'
sh: line 37: `"}; export BASH_FUNC_module()'

& 




Expected results:

If I downgrade to 4.1.2-15.el6_4 (latest bash before shellshock), I get this

[root@example ~]# rpm -q bash
bash-4.1.2-15.el6_4.x86_64
[root@example ~]# at now
at> ls
at> <EOT>
job 7 at 2014-09-26 12:07
From root  Fri Sep 26 12:08:00 2014
Return-Path: <root>
X-Original-To: root
Delivered-To: root.com
Subject: Output from your job        7
To: root
Date: Fri, 26 Sep 2014 12:07:59 -0400 (EDT)
From: root (root)
Status: R

01181684
01200315
anaconda-ks.cfg
install.log
install.log.syslog
samptrace2.sh
samptrace.sh

Additional info:

Comment 2 Florian Weimer 2014-09-26 16:43:58 UTC
This is likely caused by the environment export code in at, see writefile() in at.c:

    /* Write out the environment. Anything that may look like a
     * special character to the shell is quoted, except for \n, which is
     * done with a pair of ""'s.  Dont't export the no_export list (such
     * as TERM or DISPLAY) because we don't want these.
     */

Comment 3 Florian Weimer 2014-09-26 16:48:15 UTC
at generates invalid shell code when serializing the environment:

BASH_FUNC_module()=\(\)\ {\ \ eval\ \`/usr/bin/modulecmd\ bash\ \$\*\`"
"}; export BASH_FUNC_module()

In retrospect, () was a poor choice as the suffix, but I used it because it was already used by bash before (see previous comments in the file).

Comment 5 Paul Victor Novarese 2014-09-26 17:38:43 UTC
FWIW, I have not been able to reproduce with cron.

Comment 8 Florian Weimer 2014-09-27 07:23:49 UTC
Limited workaround: Add the following to /etc/profiled/modules.sh.  This will help for use from interactive shells and shell scripts (as long as they use at and not /usr/bin/at), but there are ways to call the at command bypassing the function, so it is not a complete workaround.

if test -n "$BASH_VERSINFO"; then
  at () {
    (
      declare -f +x module at
      /usr/bin/at "$@"
    )
  }
  export -f at
fi

This also seems to work, and its effect is complete at least as far as environment-modules is concerned.  It reverts the “export -f module” in /usr/share/Modules/init/bash.  It may have a functionality impact on environment-modules, so it needs more testing.

if test -n "$BASH_VERSINFO"; then
  declare -f +x module
fi

Comment 12 Rikard 2014-09-29 12:34:50 UTC
@Florian Weimer thanks for the "workaround" so we can use "at jobs" while a real fix are being made. Is it possible to add so that the workaround also works when called from the script "/usr/bin/batch" which also comes from the "at" package ? 

[testuser@test001 ~]$ batch
at> echo hi
at> <EOT>
still results in:
 
sh: line 36: syntax error near unexpected token `=\(\)\ {\ \ \(\ declare\ -f\ +x\ module\ at\;"
"\ /usr/bin/at\ \"\$@\"\ \)"
"}'
sh: line 36: `"}; export BASH_FUNC_at()'

Comment 16 Florian Weimer 2014-09-30 11:42:20 UTC
Created attachment 942671 [details]
at-skip-names.diff

Upstream fix for at, from at 3.1.16.

http://anonscm.debian.org/cgit/collab-maint/at.git/commit/?id=482f5962d

Comment 20 Matt Temple 2014-10-01 01:07:03 UTC
(In reply to Florian Weimer from comment #8)
> Limited workaround: Add the following to /etc/profiled/modules.sh.  This
> will help for use from interactive shells and shell scripts (as long as they
> use at and not /usr/bin/at), but there are ways to call the at command
> bypassing the function, so it is not a complete workaround.
> 
> if test -n "$BASH_VERSINFO"; then
>   at () {
>     (
>       declare -f +x module at
>       /usr/bin/at "$@"
>     )
>   }
>   export -f at
> fi
> 
> This also seems to work, and its effect is complete at least as far as
> environment-modules is concerned.  It reverts the “export -f module” in
> /usr/share/Modules/init/bash.  It may have a functionality impact on
> environment-modules, so it needs more testing.
> 
> if test -n "$BASH_VERSINFO"; then
>   declare -f +x module
> fi
-------------------------------------
Environment: CentOS6
I've just inserted the test from the top of this comment into /etc/profile.d/modules.sh, and tested the fix using a tiny script that just touches a file.   I get the same error message as I did before I added  this.   Command was "at -f ./touchfile now"
Script is:

#!/bin/bash:
touch x.y

Comment 21 Matt Temple 2014-10-01 01:45:26 UTC
I got hold of the at source rpm for at.
I added the patch mentioned above, <at-skip-names.diff>
to the spec file and the sources, and rebuilt the rpm from the new
source.
   
Command:  rpmbuild -ba "at.spec"

I ran rpm --force to overwrite the old "at."

Fixed the problem.

Environment -- CentOS 6.5 -- 2.6.32-431.23.3.el6.x86_64

Comment 22 Florian Weimer 2014-10-01 06:46:20 UTC
(In reply to Matt Temple from comment #20)
> I've just inserted the test from the top of this comment into
> /etc/profile.d/modules.sh, and tested the fix using a tiny script that just
> touches a file.   I get the same error message as I did before I added 
> this.

You need to re-login, so that the changed profile scripts run.

Comment 23 Florian Weimer 2014-10-01 06:48:09 UTC
I think we should consider fixing this in the “at” package.

Comment 36 Tomas Mraz 2014-10-04 07:16:18 UTC
*** Bug 1146958 has been marked as a duplicate of this bug. ***

Comment 37 Hans-Georg Bork 2014-10-10 13:47:55 UTC
It seem that at-3.1.10-45.el6 is not yet available; however, installing at-3.1.10-44.el6 solve the issue already. Has the version noted in "Fixed in Version" probably a typo or am I getting something wrong?

Comment 38 Tomas Mraz 2014-10-10 13:50:12 UTC
No, the -45 version just corrects release date in the changelog of the rpm.

Comment 39 Hans-Georg Bork 2014-10-10 14:02:28 UTC
So the issue is fixed within -44? If so, that version should be mentioned.

Sorry that I'm picky with that, but I've trouble to explain why I installed "only" -44 on RHEL 6.5 and got it fixed, while the fix is only as of -45

Comment 40 Tomas Mraz 2014-10-10 14:09:41 UTC
I have mislead you with the comment. Because the RHEL-6.5 fixed version is completely different. You cannot depend on Fixed in field here, I am sorry. This field is used for our internal process for tracking fixes on various RHEL release branches.

Comment 43 Bill Carlson 2014-11-11 20:15:59 UTC
What is delaying this fix? Not allowed to see #1075802, why?
This has been out for CentOS for some time, makes no sense.

Comment 44 Chris Williams 2014-11-11 20:46:59 UTC
Hi Bill,

We have shipped errata for:
6.6.z
https://rhn.redhat.com/errata/RHBA-2014-1637.html
6.5.z
https://rhn.redhat.com/errata/RHBA-2014-1362.html
6.4.z
https://rhn.redhat.com/errata/RHBA-2014-1638.html

This BZ, 1147043, if for an errat that will ship in RHEL 6.7.

BZ 1075802 is an internal tracker for the 6.7 minor release.

HTH,

cww

Comment 45 Bill Carlson 2014-11-11 21:00:07 UTC
Chris,

Thanks, evidently my rhn source is not seeing these at this time, will pursue there.

Comment 47 antofthy 2015-09-09 04:53:30 UTC
I have also run into this problem but it has nothing to do with "at" command.

The problem I have is the "BASH_FUNC_???" environment exports when you call bash with the verbose option.

For example the command...

echo "echo hi" | bash -v

Outputs....

module () {  eval `/usr/bin/modulecmd bash $*`
}
scl () {  local CMD=$1;
 if [ "$CMD" = "load" -o "$CMD" = "unload" ]; then
 eval "module $@";
 else
 /usr/bin/scl "$@";
 fi
}
echo hi
hi

Which are functions exported by the /etc/profiled/modules.sh  I have NO idea what these functions are ment to provide, and really I don't care.   All I know it they stuff up any program that uses the verbose flas (presumably "at" command does this).

Note using  sh -v  also does this when 'sh' is a link to 'bash'.

Solutions so far (at least for my shell script...

1/ Use a restricted shell, so that it doe not try to execute the exported functions.   However this means you can not use  "sh -v" which is needed if the script may be usable on older Bourne shell systems (a possibility in my case).

   bash --restricted -v

2/ unset the exports beforehand.  This   

      unset -f `env | sed -n 's/^BASH_FUNC_\([a-z]*\)()=.*/\1/p'`



QUESTION: why does BASH manpage no list BASH_FUNC_???  With all the stuff about shellshock I would have thought they would at least improve the bash manpage with some more info about it!

Comment 48 antofthy 2015-09-09 06:03:09 UTC
PS: the last was with Fedora 22

Comment 51 Tomas Mraz 2015-10-05 14:05:05 UTC
Fixed in the released version.

Comment 52 antofthy 2015-10-06 00:50:37 UTC
How was it fixed?

It is very bad (both manners and in documenting) to just say 'It is fixed!' without actually given the reporters an idea as to 'How was it fixed!'.

We do not have an 'inside track' to the developers, so at least a minimal bit of feedback would be appreciated.

Comment 53 Tomas Mraz 2015-10-06 08:20:43 UTC
The fix is described here:
https://access.redhat.com/errata/RHBA-2014:1637