Bug 496432
| Summary: | EventLog.lock fd leak and submission events are bogus | ||
|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Matthew Farrellee <matt> |
| Component: | condor | Assignee: | Matthew Farrellee <matt> |
| Status: | CLOSED ERRATA | QA Contact: | Jan Sarenik <jsarenik> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 1.1.1 | CC: | jsarenik, lbrindle |
| Target Milestone: | 1.2 | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Grid bug fix
C: When the EventLog was configured in the Schedd, on each submission a file descriptor to the EventLog.lock would be leaked.
C: The Schedd would eventually run out of file descriptors and exit with an error to that effect.
F: The leak has been located and corrected
R: The schedd no longer runs out of file descriptors. Subsequently, the error no longer occurs
When the EventLog was configured in the schedd, on each submission a file descriptor to the EventLog.lock would be leaked. This caused the schedd to eventually run out of file descriptors and exit with an error. The leak has been located and corrected so that the schedd no longer runs out of file descriptors. Subsequently, the error no longer occurs.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2009-12-03 09:18:50 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 527551 | ||
This is tracked upstream... http://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=395 It has been resolved. Verify with: 1) env _CONDOR_USE_PROCD=FALSE _CONDOR_EVENT_LOG=<some nfs mount/EventLog.test condor_schedd -t -f 2) echo -e "executable = /bin/sleep\narguments = 2400\nshould_transfer_files = true\nwhen_to_transfer_output = on_exit\nqueue 5\n" | condor_submit 3) ls /proc/`pidof condor_schedd`/fd Failure will show fds open to <some nfs mount>/EventLog.test.lock It should be verified that fds are never leaked when _CONDOR_EVENT_LOG=/tmp/EventLog.test too, for good measure. Release note added. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: please see bug summary. I can not reproduce the bug as condor-7.3.1-0.3.el5 is already not available from Brew. The bug is not reproducible on previous stable condor-7.2.2-0.9.el5. When I follow the steps given in comments above, I see no file leaks even on the latest -devel condor-7.4.1-0.2.el5 so I consider it being fixed. Please verify that /mnt/pool/spool/EventLog exists and contains data related to jobs in the queue as a baseline. What confuses me is that there is no mention of NFS in the initial comment and later in second comment there is ``<some nfs mount/EventLog.test'' This is how I tested (mynfs is NFS mounted, I have tried the same in local /tmp): On clear RHEL system I installed condor. # echo 'EVENT_LOG = /mnt/mynfs/EventLog' >> ~condor/condor_config.local # service condor start # su user -c 'echo -e "executable = /bin/echo\narguments = ahoj\nshould_transfer_files = true\nwhen_to_transfer_output = on_exit\nqueue 2\n" | condor_submit' # ls /mnt/mynfs/ EventLog EventLog.lock Sorry for the confusion. This should be verified on and off NFS. Condor uses different code paths depending on if a file is on NFS or not. Sorry, I'm struggling to interpret this one. Could someone please dumb this down a bit for the tech writer? ;) Thanks, LKB Release note updated. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1,6 @@ -please see bug summary.+Grid bug fix + +C: +C: +F: +R: When the EventLog was configured in the Schedd, on each submission a file descriptor to the EventLog.lock would be leaked. The Schedd would eventually run out of fds and exit with an error to that effect. The leak has been fixed. Release note updated. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,6 +1,8 @@ Grid bug fix -C: +C: When the EventLog was configured in the Schedd, on each submission a file descriptor to the EventLog.lock would be leaked. -C: +C: The Schedd would eventually run out of file descriptors and exit with an error to that effect. -F: +F: The leak has been located and corrected -R:+R: The schedd no longer runs out of file descriptors. Subsequently, the error no longer occurs + +When the EventLog was configured in the schedd, on each submission a file descriptor to the EventLog.lock would be leaked. This caused the schedd to eventually run out of file descriptors and exit with an error. The leak has been located and corrected so that the schedd no longer runs out of file descriptors. Subsequently, the error no longer occurs. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHEA-2009-1633.html |
It appears that an fd to the EventLog.lock is leaked for each submission, and the submission events are bogus in the EventLog itself. condor-7.3.1-0.3.el5 EVENT_LOG = /mnt/pool/spool/EventLog Terminal 0: $ echo -e "executable = /bin/sleep\narguments = 5m\nshould_transfer_files = true\nwhen_to_transfer_output = on_exit\nqueue 2000\n" | condor_submit Terminal 1: # date; ls -al /proc/`pidof condor_schedd`/fd/ | awk '{print $11}' | sort | uniq -c | sort -n | tail -n 1 Sun Apr 19 00:11:12 EDT 2009 3 /dev/null Sun Apr 19 00:11:14 EDT 2009 20 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:11:15 EDT 2009 69 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:11:16 EDT 2009 118 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:11:17 EDT 2009 152 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:11:19 EDT 2009 208 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:11:21 EDT 2009 210 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:11:23 EDT 2009 313 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:11:25 EDT 2009 431 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:11:51 EDT 2009 ls: /proc//fd/: No such file or directory # cat /var/log/condor/dprintf_failure.SCHEDD dprintf() had a fatal error in pid 4542 **** PANIC -- OUT OF FILE DESCRIPTORS at line 821 in dprintf.ceuid: 64, ruid: 0 # cat /mnt/pool/spool/EventLog [snip] 000 (017.990.000) 04/19 00:11:42 Job submitted from host: ... 000 (017.991.000) 04/19 00:11:42 Job submitted from host: ... 000 (017.992.000) 04/19 00:11:43 Job submitted from host: ... 000 (017.993.000) 04/19 00:11:43 Job submitted from host: ... 000 (017.994.000) 04/19 00:11:43 Job submitted from host: ... Attempt 2: Terminal 0: Submit 300 jobs instead... Terminal 1: Sun Apr 19 00:15:22 EDT 2009 3 /dev/null Sun Apr 19 00:15:27 EDT 2009 86 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:28 EDT 2009 127 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:29 EDT 2009 157 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:30 EDT 2009 192 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:31 EDT 2009 221 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:31 EDT 2009 250 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:32 EDT 2009 277 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:33 EDT 2009 300 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:33 EDT 2009 299 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:34 EDT 2009 299 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:35 EDT 2009 299 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:37 EDT 2009 299 /mnt/pool/spool/EventLog.lock Sun Apr 19 00:15:39 EDT 2009 299 /mnt/pool/spool/EventLog.lock