Bug 698670

Hmm, I don't see any rule that bash should provide these commands.

Anyway, few builtins already are regular commands:
false, kill, newgrp, pwd
(newgrp is not bash builtin)

but more are not:
alias, bg, cd, command, fc, fg, getopts, jobs, read, true, umask, unalias, wait

Regular commands are provided by:
false: coreutils
kill: util-linux-ng
newgrp: shadow-utils
pwd: coreutils

Your resolution seems to be good workaround, but not good for consistency. I guess the best resolution is to provide some new package with those missing commands...

(In reply to comment #2)
> Hmm, I don't see any rule that bash should provide these commands.
> 
> Anyway, few builtins already are regular commands:
> false, kill, newgrp, pwd

and true

> (newgrp is not bash builtin)
> 
> but more are not:
> alias, bg, cd, command, fc, fg, getopts, jobs, read, true, umask, unalias, wait

drop true, add hash, type, ulimit

> 
> Regular commands are provided by:
> false: coreutils
> kill: util-linux-ng
> newgrp: shadow-utils
> pwd: coreutils

true: coreutils

> 
> Your resolution seems to be good workaround, but not good for consistency. I
> guess the best resolution is to provide some new package with those missing
> commands...

Whether this is done in bash's specfile or via a new package, I don't care, as long as that package automatically gets installed when I want POSIX compliance.

Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Many of the builtins you listed work by changing state in the shell process or control its child processes: "cd" changes the shell's working directory, "read" assigns to the environment variables to be passed to children, alias changes aliases, wait reaps child exit status, ...

How could these builtins be implemented outside the shell process itself?  A regular command runs in a separate process, and as such can't access the shell's state without some form of IPC, nor can it control the shell's child processes.

POSIX requires that they exist outside the shell, but also specifically documents that they are not as useful when exec'd rather than run as a builtin.  That is, 'exec cd' is required to work, but it changes the directory of the exec'd process and not of the shell itself.  'read' is required to parse data, but that data is not visible to the parent shell.  For most of the builtins, the effect of having them be something that can be found by exec is for the side effects (with read, the side effect of consuming a line of stdin can be useful; with cd, the side effect of getting an exit status of whether or not the directory can be changed into), and not for affecting the parent shell as is the case when the shell builtin is executed.

If you need more proof, look at a Solaris machine:

$ uname -sr
SunOS 5.10
$ type -a cd
cd is a shell builtin
cd is /usr/bin/cd
cd is /bin/cd
cd is /usr/5bin/cd
$ file /usr/bin/cd
/usr/bin/cd: Korn shell script text executable
$ cat /usr/bin/cd
#!/bin/ksh -p
#
#ident  "@(#)alias.sh   1.2     00/02/15 SMI"
#
# Copyright (c) 1995 by Sun Microsystems, Inc.
#
cmd=`basename $0`
$cmd "$@"

If Solaris cares enough about standards compliance to provide a shim script hard-linked to the names of all the builtins, then it should be easy to do likewise in RHEL.

$ rpm -q ksh
ksh-20110208-3.fc14.x86_64

$ rpm -ql ksh | grep cd


How many shells are in any POSIX distribution/system? How many of them are posix compliance? Is this really the shell duty to satisfy this POSIX requirement? Is this POSIX requirement usefull/needed? Does really all POSIX shells implement all Regular Built-In Utilities?

What I mean, I'm not against to satisfy this bug report. I'm just not sure if bash really should(!) provide exec-able alternatives. Is this really a good solution? If bash will provide this alternatives, then all shells could provide them...

Right now, bash provides /bin/sh, which _is_ the RHEL posix shell.  There are other shells available in RHEL, but none of them are installed as /bin/sh.  If we switch to dash (and debian already switched to dash), then I would have filed this against dash.  Likewise for ksh.

I don't care if you do this fix via bash, or via a new package independently of which shell is providing /bin/sh, as long as there is _some_ package that gets installed when I want POSIX compliance which provides these shims.  Only one package has to provide the shims, and preferably in association with providing /bin/sh.  Out of the box, upstream bash does NOT install itself as /bin/sh, so we're already doing things in the bash rpm to install POSIX files.

Regardless of what a standard (POSIX in this case) says, I conceive having executable shims which are called the same as shell built-ins, but can't provide the expected functionality as harmful. If -- by whatever freak circumstances -- I'd use a "cd" executable rather than the shell-builtin, I might overwrite or delete valuable data (think of something like "cd /; tar -cf - [...] . | (cd /my_backup_patch && tar -xf - [...]").

I can't think of any problem these executables would solve, the shells need them built-in anyway for their main purpose of e.g. changing the working directory in the shell. IMO, this is a case where the standard needs to be fixed rather than that it should be followed by the letter.

(In reply to comment #9)
> Regardless of what a standard (POSIX in this case) says, I conceive having
> executable shims which are called the same as shell built-ins, but can't
> provide the expected functionality as harmful. If -- by whatever freak
> circumstances -- I'd use a "cd" executable rather than the shell-builtin, I
> might overwrite or delete valuable data (think of something like "cd /; tar -cf
> - [...] . | (cd /my_backup_patch && tar -xf - [...]").

What you typed is guaranteed to get the shell builtin, and not the shim executable.  POSIX states this in XCU 2.9.1.1:

| If the command name matches the name of a utility listed in the following
| table, that utility shall be invoked.
| Shell Commands
| alias     false    jobs     read     wait
| bg        fc       kill     true
| cd        fg       newgrp   umask
| command   getopts  pwd      unalias

So these built-ins are favored above executable shims for all except usages like 'nice cd', 'nohup cd', 'exec cd', 'find . -exec cd \;', all of which are unlikely to appear in scripts in the first place.  But because they are not special built-ins, they do not qualify for the exemption in XCU 2.14 that only special built-ins need not have an exec'able alternative:

| The special built-in utilities in this section need not be provided in a
| manner accessible via the exec family of functions defined in the System
| Interfaces volume of POSIX.1-2008.

Finally, POSIX explicitly states this for cd (and similar wording for most of the other built-ins from the above list), in XCU cd APPLICATION USAGE:

| Since cd affects the current shell execution environment, it is always
| provided as a shell regular built-in. If it is called in a subshell or
| separate utility execution environment, such as one of the following:
|    (cd /tmp)
|    nohup cd
|    find . −exec cd {} \;
| it does not affect the working directory of the caller’s environment.

so shell programmers already know to be aware of the issues.

> I can't think of any problem these executables would solve,

Then you haven't read this whole thread.  For example, 'wait' is one of the built-ins in question.  While 'exec wait %1' would fail when run as a shim (because the executed environment does not inherit job specs from the parent), 'exec wait 123' is still valid.

> the shells need
> them built-in anyway for their main purpose of e.g. changing the working
> directory in the shell. IMO, this is a case where the standard needs to be
> fixed rather than that it should be followed by the letter.

Then please submit such a proposal to change the standards, at http://austingroupbugs.net/.  But until that is submitted and accepted, I don't see any harm in adding the shims to comply with POSIX 2008 (and portable shell programmers are already aware of the shims existing on Solaris, and haven't complained about them existing there).

This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.

This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.

(In reply to comment #9)
> Regardless of what a standard (POSIX in this case) says, I conceive having
> executable shims which are called the same as shell built-ins, but can't
> provide the expected functionality as harmful. If -- by whatever freak
> circumstances -- I'd use a "cd" executable rather than the shell-builtin, I
> might overwrite or delete valuable data (think of something like "cd /; tar -cf
> - [...] . | (cd /my_backup_patch && tar -xf - [...]").
Isn't pushd and popd better?
> 
> I can't think of any problem these executables would solve, the shells need
> them built-in anyway for their main purpose of e.g. changing the working
> directory in the shell. IMO, this is a case where the standard needs to be
> fixed rather than that it should be followed by the letter.

Nils, did you send report for this? Or should I?


Anyway, I don't see any gain in providing workaround scripts. If people want to call shell builtins via exec, they can call them like `sh -c 'builtin ...''

I've cloned this bug to Fedora. I think it's better to start with this change (if ever) in Fedora and then in RHEL 7.

(In reply to comment #14)
> (In reply to comment #9)
> > - [...] . | (cd /my_backup_patch && tar -xf - [...]").
> Isn't pushd and popd better?

The pushd/popd builtins are different. Interestingly, the (cd ... && ...) way does things how a cd executable would do it: change to the directory in a new process, which doesn't influence the parent (which is why I use it here -- no need to remembed to popd later on).

> > I can't think of any problem these executables would solve, the shells need
> > them built-in anyway for their main purpose of e.g. changing the working
> > directory in the shell. IMO, this is a case where the standard needs to be
> > fixed rather than that it should be followed by the letter.
> 
> Nils, did you send report for this? Or should I?

I wouldn't have an idea whom to contact about this.

(In reply to comment #16)

> > > I can't think of any problem these executables would solve, the shells need
> > > them built-in anyway for their main purpose of e.g. changing the working
> > > directory in the shell. IMO, this is a case where the standard needs to be
> > > fixed rather than that it should be followed by the letter.
> > 
> > Nils, did you send report for this? Or should I?
> 
> I wouldn't have an idea whom to contact about this.

I can go ahead and tackle the upstream issue of raising a bug report against POSIX to see if we can avoid requiring these to be external executables in the next version of POSIX; but the point still remains that POSIX 2008 requires them to exist.

Adding needinfo, to remind myself to create an Austin Group bug and post the URL here.

(In reply to comment #19)
> Adding needinfo, to remind myself to create an Austin Group bug and post the
> URL here.

Eric, is it done? I'm going to close this bug deferred.

Filed: http://austingroupbugs.net/view.php?id=698
I'll follow up with status according to what the Austin Group does when reviewing this bug.

Sorry; the Austin Group didn't change their stance; POSIX still requires the shim executables, with this rationale:

"This was discussed during the May 23, 2013 conference call.
XRAT section C.1.7 clearly states why the authors of this section of the standard decided to require these built-ins to be exec-able. We do not believe that the arguments presented in this change request override the concerns stated by the original authors.

Therefore, we reject this proposed change."
http://austingroupbugs.net/view.php?id=698#c1626

For reference, c.1.7 states:
"There were originally three justifications for allowing the omission of exec-able versions:

    It would require wasting space in the file system, at the expense of very small systems. However, it has been pointed out that all 16 utilities in the table can be provided with 16 links to a single-line shell script:

    $0 "$@"

    It is not logical to require invocation of utilities such as cd because they have no value outside the shell environment or cannot be useful in a child process. However, counter-examples always seemed to be available for even the most unusual cases:

    find . -type d -exec cd {} \; -exec foo {} \;
        (which invokes "foo" on accessible directories)

    ps ... | sed ... | xargs kill


    find . -exec true \; -a ...
        (where "true" is used for temporary debugging)

    It is confusing to have a utility such as kill that can easily be in the file system in the base standard, but that requires built-in status for the User Portability Utilities option (for the % job control job ID notation). It was decided that it was more appropriate to describe the required functionality (rather than the implementation) to the system implementors and let them decide how to satisfy it.

On the other hand, it was realized that any distinction like this between utilities was not useful to applications, and that the cost to correct it was small. These arguments were ultimately the most effective."
http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap01.html

(In reply to Roman Rakus from comment #15)
> I've cloned this bug to Fedora. I think it's better to start with this
> change (if ever) in Fedora and then in RHEL 7.

Since the change has landed in Fedora 18, I'm fine with closing this as WONTFIX for RHEL 6, and waiting until RHEL 7 to get the functionality.

(In reply to Eric Blake from comment #23)
> (In reply to Roman Rakus from comment #15)
> > I've cloned this bug to Fedora. I think it's better to start with this
> > change (if ever) in Fedora and then in RHEL 7.
> 
> Since the change has landed in Fedora 18, I'm fine with closing this as
> WONTFIX for RHEL 6, and waiting until RHEL 7 to get the functionality.

OK, but deferred is better I think.