Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 574941[details]
Reproducer
Description of problem:
Fortran77 requires static arrays. These end up as BSS sections in the binary.
This particular test has a large array in memory:
[25] .bss NOBITS 0000000000608380 00008374
00000004fb4d1528 0000000000000000 WA 0 0 32
This 0x4fb4d1528 ends up being 19GB
When this process is being linked and it gets to the part where it it calculates the build-id. It allocates all that space (which is nothing but zeros) and then calculates the buildid including that. This greatly increases the time needed to link the binary, can cause this shared diskless machine to oomkill, and impacts other users who are trying to do something.
When we attach to the bloated process with gdb:
(gdb) where
#0 0x00000000004282a3 in sha1_process_block (buffer=0x2aaf56f99bde, len=<value optimized out>, ctx=0x7fffffffd2c0) at ./sha1.c:355
#1 0x000000000042925b in sha1_process_bytes (buffer=<value optimized out>, len=<value optimized out>, ctx=0x7fffffffd2c0) at ./sha1.c:245
#2 0x00002aaaaad1beca in bfd_elf64_checksum_contents (abfd=0x6a5be0, process=0x4291b0 <sha1_process_bytes>, arg=0x7fffffffd2c0) at elfcode.h:1206
#3 0x0000000000420ee7 in gldelf_x86_64_write_build_id_section (abfd=0x6a5be0) at eelf_x86_64.c:906
#4 0x00002aaaaad26a2f in _bfd_elf_write_object_contents (abfd=0x6a5be0) at elf.c:5155
#5 0x00002aaaaad01437 in bfd_close (abfd=0x6a5be0) at opncls.c:692
#6 0x0000000000417f7c in main (argc=46, argv=0x7fffffffd6b8) at ./ldmain.c:515
Looking at where the problem seems to be:
#2 0x00002aaaaad1beca in bfd_elf64_checksum_contents (abfd=0x6a5be0, process=0x4291b0 <sha1_process_bytes>, arg=0x7fffffffd2c0) at elfcode.h:1206
1206 (*process) (sec->contents, i_shdr.sh_size, arg);
(gdb) p *i_shdr
$3 = {sh_name = 240, sh_type = 8, sh_flags = 3, sh_addr = 6325120, sh_offset = 0, sh_size = 21396002088, sh_link = 0, sh_info = 0, sh_addralign = 32, sh_entsize = 0, bfd_section = 0xbd2b30, contents = 0x0}
There is our 19GB.
Thus we can see clearly that the problem is when it is calculating the checksum of for the BSS.
We have a work around passing in -Wl,--build-id={none,uuid} but we believe that it would better if we had an optimized buildid calculation which didn't allocate the bss when it calculates the checksum.
The problem appears to only happen on rhel6 not rhel5 or F16.
It seems to be geared toward mpich2 rather than openmpi.
Version-Release number of selected component (if applicable):
binutils-2.20.51.0.2-5.28.el6.x86_64
How reproducible:
100%
Steps to Reproduce:
The attached reproducer requires MPI.
Actual results:
Excessive memory utilization.
Expected results:
Appropriate memory utilization.
Additional info:
This bug has also been reported against the FSF binutils:
http://sourceware.org/bugzilla/show_bug.cgi?id=12451
The uploaded patch is a simplified version of the patch that fixes that PR. Once some internal networking problems are resolved I will add it to the RHEL6.3 binutils rpm.
Cheers
Nick
Thanks Nick,
I never would have made the association between 12451 and the reported bug. The patch looks really simple and I think that the users are OK with the current workaround for a bit but if you would like some testing on some alpha/beta test packages for 6.3 just let us know where to find the packages and we'll run them through our env for a while.
(In reply to comment #0)
> Created attachment 574941[details]
> Reproducer
$ sh REPRODUCER_SCRIPT.sh
[...]
mpif77 -c -O3 -fPIC cg.f
make[1]: mpif77: Command not found
[...]
I have openmpi and mpich2 packages installed. How do I compile it on RHEL6?
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
http://rhn.redhat.com/errata/RHBA-2012-0872.html
Created attachment 574941 [details] Reproducer Description of problem: Fortran77 requires static arrays. These end up as BSS sections in the binary. This particular test has a large array in memory: [25] .bss NOBITS 0000000000608380 00008374 00000004fb4d1528 0000000000000000 WA 0 0 32 This 0x4fb4d1528 ends up being 19GB When this process is being linked and it gets to the part where it it calculates the build-id. It allocates all that space (which is nothing but zeros) and then calculates the buildid including that. This greatly increases the time needed to link the binary, can cause this shared diskless machine to oomkill, and impacts other users who are trying to do something. When we attach to the bloated process with gdb: (gdb) where #0 0x00000000004282a3 in sha1_process_block (buffer=0x2aaf56f99bde, len=<value optimized out>, ctx=0x7fffffffd2c0) at ./sha1.c:355 #1 0x000000000042925b in sha1_process_bytes (buffer=<value optimized out>, len=<value optimized out>, ctx=0x7fffffffd2c0) at ./sha1.c:245 #2 0x00002aaaaad1beca in bfd_elf64_checksum_contents (abfd=0x6a5be0, process=0x4291b0 <sha1_process_bytes>, arg=0x7fffffffd2c0) at elfcode.h:1206 #3 0x0000000000420ee7 in gldelf_x86_64_write_build_id_section (abfd=0x6a5be0) at eelf_x86_64.c:906 #4 0x00002aaaaad26a2f in _bfd_elf_write_object_contents (abfd=0x6a5be0) at elf.c:5155 #5 0x00002aaaaad01437 in bfd_close (abfd=0x6a5be0) at opncls.c:692 #6 0x0000000000417f7c in main (argc=46, argv=0x7fffffffd6b8) at ./ldmain.c:515 Looking at where the problem seems to be: #2 0x00002aaaaad1beca in bfd_elf64_checksum_contents (abfd=0x6a5be0, process=0x4291b0 <sha1_process_bytes>, arg=0x7fffffffd2c0) at elfcode.h:1206 1206 (*process) (sec->contents, i_shdr.sh_size, arg); (gdb) p *i_shdr $3 = {sh_name = 240, sh_type = 8, sh_flags = 3, sh_addr = 6325120, sh_offset = 0, sh_size = 21396002088, sh_link = 0, sh_info = 0, sh_addralign = 32, sh_entsize = 0, bfd_section = 0xbd2b30, contents = 0x0} There is our 19GB. Thus we can see clearly that the problem is when it is calculating the checksum of for the BSS. We have a work around passing in -Wl,--build-id={none,uuid} but we believe that it would better if we had an optimized buildid calculation which didn't allocate the bss when it calculates the checksum. The problem appears to only happen on rhel6 not rhel5 or F16. It seems to be geared toward mpich2 rather than openmpi. Version-Release number of selected component (if applicable): binutils-2.20.51.0.2-5.28.el6.x86_64 How reproducible: 100% Steps to Reproduce: The attached reproducer requires MPI. Actual results: Excessive memory utilization. Expected results: Appropriate memory utilization. Additional info: