Description of problem: The default path for root after login by "su -" or ssh is /usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin The problem is /usr/local/sbin and /usr/local/bin, where /usr/local is shared over nfs with other hosts. When the nfs server for /usr/local is down, then the shell hangs because it want to search the path for the command and waits for /usr/local/sbin and /usr/local/bin for a read answer. This means that if that fileserver is down, no user can login oder do something (because the also have /usr/local/bin in their path), and also root cannot do anything because the shell hangs for the same reason. But at least root should be able to do something, for example to change the systems into normal operational state again. I found the following: [root@xxx ~]# strings /sbin/init |grep /usr/local /usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin [root@xxx ~]# strings /usr/sbin/sshd |grep /usr/local /usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin /usr/local/bin:/bin:/usr/bin [root@yxxx ~]# strings /bin/bash |grep /usr/local /usr/local/share/bashdb/bashdb-main.inc /usr/local/bin:/bin:/usr/bin I also looked in /proc/PID/environ of a process running mingetty on tty? and found the same path as shown above from grep of /sbin/init. It seems that /usr/local/sbin and /usr/local/bin are hardcoded in the binaries. I checked all the init scripts in /etc and searched for /usr/local and found nothing which sets /usr/local/{sbin,bin} for root. I think the defaults should be /sbin:/bin:/usr/sbin:/usr/bin for root /bin:/usr/bin for normal users. More paths can be added by scripts in /etc/profile.d/ which will be user configurable. Question: Is there a clean and welldefined way to set PATH early enought for system and login programs to overwrite the hardcoded defaults, for example in a config file in /etc/* ? The default for root logins und normal user logins should be set separatly. In previous versions of fedora and redhat there was a file /etc/default/login where the following variables could be set: ROOT=/bin:/usr/bin SUROOT=/sbin:/bin:/usr/sbin:/usr/bin PASSREQ=YES But it seems that this file isn't used in the current fedora release. If this problem would be a FAQ I will be very sorry but I found no description in the documentation and internet. Version-Release number of selected component (if applicable): I don't know which programs / package sets the path. Maybe it affects /sbin/init, /bin/bash, /usr/sbin/sshd, and possibly other programs that use /usr/local/{sbin,bin} . I have choosen openssh because bugzilla requires a choose, but upstart and bash may also affected. If someone can confirm the problem as described, then we can also open bugreports against these components. How reproducible: Allways. Steps to Reproduce: 1. Login by "su -" or "ssh -l root localhost". 2. Enter "echo $PATH" Actual results: PATH contains /usr/local/{sbin,bin} . Expected results: PATH should not contain /usr/local/{sbin,bin} .
I should probably WONTFIX this right away but if the default should be changed somewhere it would be in the bash first so I'm reassigning it to bash maintainer for consideration. But IMO there is no reason why /usr/local/... should not be in the default paths. You can override the default in /etc/profile anyway.
Bash only use set environment, but not set it. I don't know which exact package add /usr/local/anything into PATH. /etc/profile and /etc/bashrc are in hand of setup, so I will change this to setup.
Thanks for reassigning, both... Right component is really setup (if any) ... it's funny opposite ... nonlocal directories /usr/local/bin and /usr/local/sbin ... I would recommend to use /mnt/nfs or something like that for nfs share - and then you could add setting path to /mnt/nfs/{bin,sbin} into ~/.bashrc for any user which should be able to execute nfs binaries. Changing default PATH set would affect almost everyone (everyone who expect to have /usr/local/{bin,sbin} searched for binary will be affected) for very rare benefit for users with nfs-mounted /usr/local ... and after that someone will come up with proposel to remove /usr/bin and /usr/sbin from default path - as you could have /usr as nfs mount as well - and there could be another nfs timeout and very long hang-out ... File you are searching for (and where you could modify default paths) is /etc/profile - there you could delete line with pathmunge /usr/local/bin and /usr/local/sbin - and you will not have /usr/local/bin or /usr/local/sbin in default PATH. Note: This doesn't have immediate effect - as envvar PATH is already set with those locations. Setting to assigned, although I'll probably close it WONTFIX/NOTABUG soon as I don't think it's way to go.
Thanks for the answers. Ok, one right component is setup - as /etc/profile and /etc/csh.login contain lines that include /usr/local/* into PATH. But these files are not the real problem for me - I can overwrite them, which I already have done. But after doing "su -" and "ssh -l root localhost" (for example) PATH is as follows: /usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin So - changing the initscripts is not enough. Or at least I don't know for the moment - I will try it. I will modify /etc/profile to remove /usr/local/* from PATH and try with /usr/local unavailable. But I guess that ssh, bash, su or even init may use PATH before reading /etc/profile? If this is true, then that process will hang waiting for hard-mounted /usr/local (hard-mounting is right here because user programs should continue after a short disruption of nfs service). So my suggestion is be to change be precompiled buildin default PATH of these binaries (init, bash, su, ssh) to /bin:/usr/bin (and maybe, /sbin:/usr/sbin:/bin:/usr/bin for root) and to leave the job of adding /usr/local/sbin and /usr/local/bin to the setup scripts in /etc. The scripts in /etc may contain /usr/local as long as they can be changed by system admins and not automatically overwritten by new versions of rpm packages. So also others are (normally) not affected. (I surely agree that the change should not break the working system for others.) About the question if /usr/local should be local and not a nfs share: Ok, if /usr/local needs to be on a local file system is the question. We use /usr/local nfs shared since nearly twenty years, first on SunOS, and later the same concept for Linux. Most software package, for example nearly any GNU software, configures as default to /usr/local if you install it by hand and use "configure" or a preconfigured makefile. And in the past we don't want to compile and install every software on every machine as this would be much work - so /usr/local was the place to install software that the operation system doesn't supply. Ok, things changed over the time, and nearly all software is available as package for linux, so /usr/local gets smaller and contains only our special application software. But they also default to /usr/local - both our local software and downloaded software from others. Changing this to /mnt/something may cause that some of this software may not work anymore. May be we can change and modify all these software to use another path - but many programs need to be changed and recompiled (if we have the source...). Question: Is there a standard path for a (nfs) shared /usr/local instead of real /usr/local - e.g. an standard /mnt/xyz or /usr/site-share or /usr/site-local or something like that that is default for linux or unix systems and used around the world? (I haven't seen such a directory / partition by now.) I thought that /opt is for host local additional software and /usr/local may be used for shared site local software. About your comment about /usr: No, I don't think that /usr should be excluded from PATH because it may be nfs mounted, for example on a diskless machine. If /usr is missing, then many programs are missing, and the system does not work right anyway. So I think /usr/local should be considered differently from /usr. Independent of this: My suggestion to exclude /usr/local from the precompiled buildin default PATH and to add it with setup scripts allows both to work: /usr/local shared and non-shared, and it does not break other user. Thanks in advance!
I mentioned /bin/su because it also contains /usr/local: [root@xxx ~]# strings /bin/su |grep /usr/local /usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin /usr/local/bin:/bin:/usr/bin Now I have searched the paths in the sources. Here the lines I think that may put /usr/local/bin or /usr/local/sbin into the binaries. bash patches: bash-2.03-paths.patch:- "/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:." bash-2.03-paths.patch:+ "/usr/local/bin:/bin:/usr/bin" coreutils patches: sh-utils-1.16-paths.patch:+#define DEFAULT_LOGIN_PATH "/usr/local/bin:/bin:/usr/bin" sh-utils-1.16-paths.patch:+ "/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin" In upstart-0.3.9.tar.bz2: ./upstart-0.3.9/init/paths.h:#define PATH "/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin" In bash-3.2.tar.gz: ./bash-3.2/config-top.h: "/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:." In openssh-5.1p1-noacss.tar.bz2: ./openssh-5.1p1/buildpkg.sh.in:/usr/local/bin \ ./openssh-5.1p1/buildpkg.sh.in:/usr/local/sbin \ ./openssh-5.1p1/buildpkg.sh.in:[ -d /usr/local/bin ] && { ./openssh-5.1p1/buildpkg.sh.in: echo $PATH | grep ":/usr/local/bin" > /dev/null 2>&1 ./openssh-5.1p1/buildpkg.sh.in: [ $? -ne 0 ] && PATH=$PATH:/usr/local/bin There are also occurences of /usr/local in documentation files, test programs, etc., and other references to /usr/local like /usr/local/lib, /usr/local/include, /usr/local/share, and more. I have not mentioned these in the listing above. If you agree that this may be changed: I think the patch files are from fedora (?) and can be changed by fedora / redhat (is there a new bugreport necceccary for each component?). The occurences in the tar file may be changed by fedora with a patch and eventually brought to the upstream sources?
Thanks for detailed clarification and searching for possible culprits. Now I understand what's the meaning of that bugzilla. I'll suggest to summarize required/suggested change and make proposal on fedora-devel-list . As it has many experienced readers, someone may have some objections (if there is something what will be broken by this new behaviour/set of default PATHs). Then - if accepted community-wide (or at least if no real objection found) it could be used as a tracker bug and those minor changes in components should be reported separately (although bash, coreutils and openssh maintainer is already in CC, it would be better to track progress and possibly report/reopen bugs in the case of broken behaviour of something).
Note: coreutils added /usr/local/bin and /usr/local/sbin for compatibility with util-linux-ng (#102567) - so those should be listed as well...
As nothing was sent to fedora-devel-list so far and as it seems /usr/local/bin is already part of LSB, I'm going to make some conclusion. Best way to avoid issues with nonlocal NFS mount could be to make /usr/local/ (or whatever directory) as symlink to NFS share. This should prevent issues with horrible timeouts - as broken symlink should be skipped in the case of unavailable share. I tend to close this bugzilla NOTABUG, as in fact is not a bug - as in Linux filesystem Standard 1.2 ( e.g. http://linux.tnc.edu.tw/techdoc/linux-books/rute/node38.html )is /usr/local described as a directory for "local hierarchy" . Anyway feel free to discuss the issue on fedora-devel-list if you think that the change to default behaviour/path set is required.
Dear Ondrej, sorry for the long "not response" in the meantime. I was thinking about the problem and I'm not sure what will be the best solution. We are thinking about creating a new directory for a shared local hierarchy with a name other than /usr/local and leaving /usr/local non-shared host-local directory. But this implies more changes in our infrastructure. Regarding Linux filesystem Standard 1.2 ( e.g. http://linux.tnc.edu.tw/techdoc/linux-books/rute/node38.html ) part 35.4.9.1: "It may be used for programs and data that are shareable amongst a group of hosts, but not found in /usr." Doesn't the word "shareable" mean that /usr/local can be on a network filessystem (NFS)? It is ok to close the bug as "notabug". I will regard to discuss the problem on the mailing list. (Do I need to subscribe to fedora-devel-list to get the answers to my mail or will answers automatically copied to me without being subscribed?) Thanks for your comments and your time!
You could subscribe easily on http://www.redhat.com/mailman/listinfo/fedora-devel-list - but be prepared for tons of emails everyday. I think the mailing list is not restricted to subscribers only, so you could send the email without subscription - although it's not guaranteed that everyone will use reply-to-all (most of users will...) and keep you informed about replies. Anyway - you could always check the thread in the archives (link on subscription page), so you'll be informed about such replies anyway.