Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 563759

Summary: A combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault
Product: Red Hat Enterprise Linux 5 Reporter: Tim Dyce <tjdyce>
Component: glibcAssignee: Jeff Law <law>
Status: CLOSED DUPLICATE QA Contact: qe-baseos-tools-bugs
Severity: high Docs Contact:
Priority: low    
Version: 5.4CC: ewan, fweimer, law, rrakus, tjdyce
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-01-25 16:54:03 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Shell script to generate the segfault described none

Description Tim Dyce 2010-02-11 02:25:17 UTC 
Description of problem:

I think I have found a combination of specific LD_PRELOAD and LD_LIBRARY_PATH which causes core binaries (ls, rm etc.) to segfault in bash and sh.
This only occurs for a LD_LIBRARY_PATH 8 character length window (at around 2690 chars), with a very specific LD_PRELOAD path.

This was causing issues on our worker nodes for some ATLAS (CERN) jobs.
Note that these LD_PRELOAD and LD_LIBRARY_PATH variables contain library paths that do not exist on normal nodes, but this should not generate a segfault in itself. When run on a standard RHEL 5.3 x86_64 machine with none of the paths present the error still occurs.
It should also no only occur for only an 8 character length window of the LD_LIBRARY_PATH variable.

Version-Release number of selected component (if applicable):
RHEL 5.3 x86_64 and 5.4 x86_64 (no issue on i686)

How reproducible:

Run the attached script .... it will work fine and just ls the path until the LD_LIBRARY path grows to a certain point, then ls will segfault for 8 characters and then recover.

Note: It performs an ls at each LD_LIBRARY_PATH length, so there is less output when run in a relatively empty directory 

Actual results:

ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
Segmentation fault
Segmentation fault
Segmentation fault
Segmentation fault
Segmentation fault
Segmentation fault
Segmentation fault
Segmentation fault
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9
ERROR: ld.so: object '/opt/exp_soft/SL5/atlas/local/$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so' from LD_PRELOAD cannot be preloaded: ignored.
dsmerror.log  dsmsched.log  ld_path_length_specific_segfault_example.sh  segfault_ld_paths_random.sh  setup.sh  setup.sh_15.5.1-orig  simple_seg.sh  tags  torque-2.3.9


Expected results:
 No segfaults

Additional info:
Comment 1 Tim Dyce 2010-02-11 02:26:56 UTC 
Created attachment 390151 [details]
Shell script to generate the segfault described
Comment 2 Roman Rakus 2010-03-10 14:51:25 UTC 
Simplified reproducer:
LD_LIBRARY_PATH='/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/AtlasOffline/15.5.1/InstallArea/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/AtlasSimulation/15.5.1/InstallArea/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/AtlasAnalysis/15.5.1/InstallArea/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/AtlasTrigger/15.5.1/InstallArea/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/AtlasReconstruction/15.5.1/InstallArea/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/AtlasEvent/15.5.1/InstallArea/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/AtlasConditions/15.5.1/InstallArea/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/AtlasCore/15.5.1/InstallArea/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/GAUDI/v20r4p5/InstallArea/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/DetCommon/15.5.1/InstallArea/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/external/slc4compat/1.0.2/i686-slc4-gcc34/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/sw/lcg/external/MCGenerators/rivet/1.1.3/slc4_ia32_gcc34/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/sw/lcg/external/MCGenerators/pythia8/120/slc4_ia32_gcc34/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/sw/lcg/external/valgrind/3.4.1/slc4_amd64_gcc34/lib/valgrind/amd64-linux:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/sw/lcg/external/qt/4.4.2/slc4_ia32_gcc34/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/sw/lcg/external/castor/2.1.7-24/slc4_ia32_gcc34/usr/lib:/lib:/opt/exp_soft/SL5/atlas/local/lib:/opt/d-cache/dcap/lib:/opt/glite/lib:/opt/glite/lib64:/opt/globus/lib:/opt/lcg/lib:/opt/lcg/lib64:/opt/classads/lib64/:/opt/c-ares/lib/:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/sw/lcg/external/coin3d/2.4.6/slc4_ia32_gcc34/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/sw/lcg/external/soqt/1.4.1_qt4.4.2_coin3d2.4.6/slc4_ia32_gcc34/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/sw/lcg/external/Python/2.5.4/slc4_ia32_gcc34/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/sw/lcg/external/mysql++/1.7.40_mysql.5.0.18/slc4_ia32_gcc34/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/tdaq-common/tdaq-common-01-14-00/installed/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/dqm-common/dqm-common-00-10-00/installed/i686-slc4-gcc34-opt/lib:/imports/ExpSoft/SL5/atlas/prod/releases/rel_15-8/sw/lcg/external/qwt/5.1.1_qt4.4.2/slc4_ia32_gcc34/lib:/uxxxxxxx' LD_PRELOAD="/opt/exp_soft/SL5/atlas/local/\$LIB/libglobus_gssapi_gsi_gcc32dbgpthr.so"  ls
Comment 3 Roman Rakus 2010-03-11 13:24:25 UTC 
I'm pretty sure, the problem is on the linker side. Reassigning to glibc.
Comment 4 Tim Dyce 2010-06-29 15:34:28 UTC 
This bug is now affecting other WLCG grid sites as well, is there any update on the fix?
Thanks,
Tim
Comment 5 Andreas Schwab 2010-06-29 15:46:27 UTC 
4c35fb65c9ea221561bc5698358944a73383b3d6
Comment 6 Jeff Law 2012-01-25 16:54:03 UTC 
Unobvious duplicate, but these two bugs have the same underlying root cause, an underestimation of the length of DST substitution.

*** This bug has been marked as a duplicate of bug 694655 ***