Bug 534073
| Summary: | condor scheduler / negotiator crash during condor shutdown process | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Frantisek Reznicek <freznice> | ||||||||
| Component: | condor | Assignee: | Matthew Farrellee <matt> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Tomas Rusnak <trusnak> | ||||||||
| Severity: | medium | Docs Contact: | |||||||||
| Priority: | medium | ||||||||||
| Version: | Development | CC: | esammons, matt, mkudlej, pmackinn, trusnak | ||||||||
| Target Milestone: | 1.3 | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | All | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | Type: | --- | |||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | 533213, 544976, 610773, 625450 | ||||||||||
| Bug Blocks: | 596210 | ||||||||||
| Attachments: |
|
||||||||||
Created attachment 368430 [details]
Detailed info about bz
The issue was retested, attached file contains the transcript + all condor logs + job output and job log file
Created attachment 369018 [details] The issue reproducer Changing the bugzilla title to "condor scheduler / negotiator crash during condor shutdown process" This bug is close to the bug 528015. However it seems to not be the same. I cannot get the core files even if ulimit + DAEMON_COREFILE_LIMIT='unlimited' + CREATE_CORE_FILES = True specified. There are crashes mentioned in MasterLog, for instance: /var/log/condor/MasterLog:11/11 07:20:55 The SCHEDD (pid 9619) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:21:00 The NEGOTIATOR (pid 11589) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:23:46 The SCHEDD (pid 14243) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:23:51 The NEGOTIATOR (pid 15924) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:26:18 The NEGOTIATOR (pid 20416) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:29:26 The SCHEDD (pid 22695) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:29:31 The NEGOTIATOR (pid 24401) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:31:12 The NEGOTIATOR (pid 26795) died due to signal 11 (Segmentation fault) ... The attached reproducer needs to have installed packages of qpidd, qmf, condor and condor-qmf-plugins. Run it following way: bash test.sh 1000 1 1 ^ number of loops ^ verbose on ^ clean condor logs before real start There were 4 instances of the test run on RHEL 4.8 i386 / x86_64 and 5.4 386 / x86_64. Just RHEL 4.8 i386 / x86_64 tests show this behavior, this might indicate RHEL4 only issue. The test itself randomize parameters which jitters the time needed to trigger those crashes, negotiator crashes are seen much often than scheduler ones: [root@hp-dl360-06 bz]# bash test_cndr_restart.sh 100 1 1 grep: /var/log/condor/*Log: No such file or directory DAEMON_COREFILE_LIMIT='unlimited' CREATE_CORE_FILES = True MAX_COLLECTOR_LOG = 100000000 MAX_STARTD_LOG = 100000000 MAX_STARTER_LOG = 100000000 MAX_SCHEDD_LOG = 100000000 MAX_MASTER_LOG = 100000000 MAX_NEGOTIATOR_LOG = 100000000 SCHEDD.PLUGINS = $(LIB)/plugins/MgmtScheddPlugin-plugin.so COLLECTOR.PLUGINS = $(LIB)/plugins/MgmtCollectorPlugin-plugin.so NEGOTIATOR.PLUGINS = $(LIB)/plugins/MgmtNegotiatorPlugin-plugin.so MASTER.PLUGINS = $(LIB)/plugins/MgmtMasterPlugin-plugin.so QMF_BROKER_HOST = hp-dl360-06.rhts.englab.brq.redhat.com qpidd dead but pid file exists Starting Qpid AMQP daemon: [ OK ] qpidd (pid 5378) is running... [07:38:39] .restart_mode:3,cpids:5182 5188 5223 5224 5225 5226 ,qpid:5378,wait:5162[msec] Stopping Qpid AMQP daemon: [ OK ] [07:38:40] qpidd toggled [07:38:40] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:38:41] .waiting: done Starting Qpid AMQP daemon: [ OK ] Stopping Qpid AMQP daemon: [ OK ] [07:38:43] .starting condor daemons Starting Condor daemons: [ OK ] [07:38:43] .wait for condor settled - jobs will be sent ^[OM[07:39:00] .job[s] running phase entered (total:1 job[s]) [07:39:00] .running job dmesg [07:39:11] ..passed (ecode:0) [07:39:14] tst_cnt:0, err_cnt:0, log_segs:0 (wait:2935mili-sec[s], qpidd:1) [07:39:14] .restart_mode:1,cpids:5560 5581 5592 5593 5594 5595 7139 ,qpid:,wait:425[msec] Starting Qpid AMQP daemon: [ OK ] [07:39:15] qpidd toggled [07:39:15] waiting: done Stopping Qpid AMQP daemon: [ OK ] [07:39:16] qpidd toggled [07:39:16] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:39:16] .waiting: done [07:39:17] .starting condor daemons Starting Condor daemons: [ OK ] [07:39:18] tst_cnt:1, err_cnt:0, log_segs:0 (wait:1248mili-sec[s], qpidd:0) [07:39:18] .restart_mode:0,cpids:7291 7297 ,qpid:,wait:14244[msec] [07:39:18] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:39:32] .waiting: done [07:39:33] .starting condor daemons Starting Condor daemons: [ OK ] [07:39:34] tst_cnt:2, err_cnt:0, log_segs:1 (wait:1060mili-sec[s], qpidd:0) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) [07:39:34] .restart_mode:3,cpids:7501 7507 ,qpid:,wait:29451[msec] Starting Qpid AMQP daemon: [ OK ] [07:39:35] qpidd toggled [07:39:35] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:39:48] .waiting: done Stopping Qpid AMQP daemon: [ OK ] Starting Qpid AMQP daemon: [ OK ] [07:39:50] .starting condor daemons Starting Condor daemons: [ OK ] [07:39:51] tst_cnt:3, err_cnt:0, log_segs:2 (wait:856mili-sec[s], qpidd:0) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:39:51] .restart_mode:2,cpids:7822 7829 ,qpid:7798,wait:3914[msec] [07:39:51] qpidd stopped (kill-STOP 7798) [07:39:51] waiting: done [07:39:55] qpidd stop released (kill-CONT 7798) [07:39:55] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:40:06] .waiting: done [07:40:06] .starting condor daemons Starting Condor daemons: [ OK ] [07:40:06] .wait for condor settled - jobs will be sent [07:40:27] .job[s] running phase entered (total:1 job[s]) [07:40:27] .running job dmesg [07:40:32] ..passed (ecode:0) [07:40:35] tst_cnt:4, err_cnt:0, log_segs:2 (wait:2872mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:40:35] .restart_mode:0,cpids:8020 8037 8047 8048 8049 8050 9910 ,qpid:7798,wait:9963[msec] [07:40:35] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:40:36] .waiting: done [07:40:36] .starting condor daemons Starting Condor daemons: [ OK ] [07:40:36] .wait for condor settled - jobs will be sent [07:40:54] .job[s] running phase entered (total:3 job[s]) [07:40:54] .running job uname -a Linux hp-dl360-06.rhts.englab.brq.redhat.com 2.6.9-89.0.16.ELsmp #1 SMP Tue Oct 27 03:49:28 EDT 2009 i686 i686 i386 GNU/Linux [07:41:24] ..passed (ecode:0) [07:41:24] .running job uname -a Linux hp-dl360-06.rhts.englab.brq.redhat.com 2.6.9-89.0.16.ELsmp #1 SMP Tue Oct 27 03:49:28 EDT 2009 i686 i686 i386 GNU/Linux [07:41:44] ..passed (ecode:0) [07:41:44] .running job df -ah / Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 15G 2.6G 12G 19% / [07:42:04] ..passed (ecode:0) [07:42:05] tst_cnt:5, err_cnt:0, log_segs:2 (wait:438mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:42:05] .restart_mode:1,cpids:9990 10011 10021 10022 10023 10024 11560 ,qpid:7798,wait:33375[msec] Stopping Qpid AMQP daemon: [ OK ] [07:42:06] qpidd toggled [07:42:06] waiting: done Starting Qpid AMQP daemon: [ OK ] [07:42:41] qpidd toggled [07:42:41] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:42:41] .waiting: done [07:42:41] .starting condor daemons Starting Condor daemons: [ OK ] [07:42:41] .wait for condor settled - jobs will be sent [07:43:01] .job[s] running phase entered (total:2 job[s]) [07:43:01] .running job dmesg [07:43:07] ..passed (ecode:0) [07:43:07] .running job rpm -q condor qpidd condor-7.4.1-0.5.el4 qpidd-0.5.752581-33.el4 [07:43:27] ..passed (ecode:0) [07:43:30] tst_cnt:6, err_cnt:0, log_segs:2 (wait:2693mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:43:30] .restart_mode:3,cpids:11814 11835 11845 11846 11847 11848 13698 ,qpid:11745,wait:16599[msec] Stopping Qpid AMQP daemon: [ OK ] [07:43:31] qpidd toggled [07:43:31] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:43:32] .waiting: done Starting Qpid AMQP daemon: [ OK ] [07:43:33] .starting condor daemons Starting Condor daemons: [ OK ] [07:43:34] tst_cnt:7, err_cnt:0, log_segs:2 (wait:878mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:43:35] .restart_mode:4,cpids:13889 13895 ,qpid:13849,wait:7421[msec] [07:43:35] qpidd stopped (kill-STOP 13849) [07:43:35] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:43:49] .waiting: done Stopping Qpid AMQP daemon: [ OK ] [07:43:50] .starting condor daemons Starting Condor daemons: [ OK ] [07:43:51] .wait for condor settled - jobs will be sent [07:44:11] .job[s] running phase entered (total:3 job[s]) [07:44:11] .running job df -ah / Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 15G 2.6G 12G 19% / [07:44:36] ..passed (ecode:0) [07:44:36] .running job df -ah / Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 15G 2.6G 12G 19% / [07:44:57] ..passed (ecode:0) [07:44:57] .running job uname -a Linux hp-dl360-06.rhts.englab.brq.redhat.com 2.6.9-89.0.16.ELsmp #1 SMP Tue Oct 27 03:49:28 EDT 2009 i686 i686 i386 GNU/Linux [07:45:17] ..passed (ecode:0) [07:45:18] tst_cnt:8, err_cnt:0, log_segs:2 (wait:1030mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:45:18] .restart_mode:3,cpids:14112 14129 14141 14142 14143 14144 16019 ,qpid:,wait:21736[msec] Starting Qpid AMQP daemon: [ OK ] [07:45:19] qpidd toggled [07:45:19] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:45:20] .waiting: done Stopping Qpid AMQP daemon: [ OK ] [07:45:21] .starting condor daemons Starting Condor daemons: [ OK ] [07:45:21] .wait for condor settled - jobs will be sent [07:45:42] .job[s] running phase entered (total:1 job[s]) [07:45:42] .running job dmesg [07:46:07] ..passed (ecode:0) [07:46:10] tst_cnt:9, err_cnt:0, log_segs:2 (wait:2589mili-sec[s], qpidd:0) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:46:10] .restart_mode:4,cpids:16255 16272 16287 16288 16289 16290 18168 ,qpid:,wait:12891[msec] [07:46:10] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:46:12] .waiting: done [07:46:12] .starting condor daemons Starting Condor daemons: [ OK ] [07:46:12] .wait for condor settled - jobs will be sent [07:46:32] .job[s] running phase entered (total:1 job[s]) [07:46:32] .running job free total used free shared buffers cached Mem: 1034580 859520 175060 0 126384 634636 -/+ buffers/cache: 98500 936080 Swap: 2031608 0 2031608 [07:46:38] ..passed (ecode:0) [07:46:40] tst_cnt:10, err_cnt:0, log_segs:2 (wait:2007mili-sec[s], qpidd:0) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:46:40] .restart_mode:4,cpids:18249 18266 18282 18283 18284 18290 20152 ,qpid:,wait:16790[msec] [07:46:40] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:46:41] .waiting: done Starting Qpid AMQP daemon: [ OK ] [07:46:42] .starting condor daemons Starting Condor daemons: [ OK ] [07:46:42] .wait for condor settled - jobs will be sent [07:47:03] .job[s] running phase entered (total:3 job[s]) [07:47:03] .running job rpm -q condor qpidd condor-7.4.1-0.5.el4 qpidd-0.5.752581-33.el4 [07:47:28] ..passed (ecode:0) [07:47:28] .running job free total used free shared buffers cached Mem: 1034580 861936 172644 0 126388 634892 -/+ buffers/cache: 100656 933924 Swap: 2031608 0 2031608 [07:47:48] ..passed (ecode:0) [07:47:48] .running job free total used free shared buffers cached Mem: 1034580 862056 172524 0 126388 634892 -/+ buffers/cache: 100776 933804 Swap: 2031608 0 2031608 [07:48:09] ..passed (ecode:0) [07:48:10] tst_cnt:11, err_cnt:0, log_segs:2 (wait:1667mili-sec[s], qpidd:0) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:48:11] .restart_mode:0,cpids:20275 20296 20306 20307 20308 20309 22185 ,qpid:20251,wait:12814[msec] [07:48:11] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:48:12] .waiting: done [07:48:12] .starting condor daemons Starting Condor daemons: [ OK ] [07:48:14] tst_cnt:12, err_cnt:0, log_segs:2 (wait:2403mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:48:15] .restart_mode:2,cpids:22347 22353 ,qpid:20251,wait:18012[msec] [07:48:15] qpidd stopped (kill-STOP 20251) [07:48:15] waiting: done [07:48:33] qpidd stop released (kill-CONT 20251) [07:48:33] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:48:34] .waiting: done [07:48:34] .starting condor daemons Starting Condor daemons: [ OK ] [07:48:34] .wait for condor settled - jobs will be sent [07:48:55] .job[s] running phase entered (total:3 job[s]) [07:48:55] .running job df -ah / Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 15G 2.6G 12G 19% / [07:49:20] ..passed (ecode:0) [07:49:20] .running job rpm -q condor qpidd condor-7.4.1-0.5.el4 qpidd-0.5.752581-33.el4 [07:49:40] ..passed (ecode:0) [07:49:40] .running job free total used free shared buffers cached Mem: 1034580 862052 172528 0 126392 634888 -/+ buffers/cache: 100772 933808 Swap: 2031608 0 2031608 [07:50:01] ..passed (ecode:0) [07:50:01] tst_cnt:13, err_cnt:0, log_segs:2 (wait:391mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:50:01] .restart_mode:3,cpids:22500 22519 22531 22532 22533 22534 24400 ,qpid:20251,wait:7673[msec] Stopping Qpid AMQP daemon: [ OK ] [07:50:03] qpidd toggled [07:50:03] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:50:04] .waiting: done Starting Qpid AMQP daemon: [ OK ] [07:50:05] .starting condor daemons Starting Condor daemons: [ OK ] [07:50:07] tst_cnt:14, err_cnt:0, log_segs:2 (wait:1730mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:50:07] .restart_mode:3,cpids:24631 24637 ,qpid:24591,wait:26565[msec] Stopping Qpid AMQP daemon: [ OK ] [07:50:08] qpidd toggled [07:50:08] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:50:17] .waiting: done Starting Qpid AMQP daemon: [ OK ] Stopping Qpid AMQP daemon: [ OK ] [07:50:20] .starting condor daemons Starting Condor daemons: [ OK ] [07:50:21] tst_cnt:15, err_cnt:0, log_segs:2 (wait:1410mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) [07:50:21] .restart_mode:4,cpids:24904 24910 ,qpid:,wait:23218[msec] [07:50:21] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:50:36] .waiting: done Starting Qpid AMQP daemon: [ OK ] [07:50:37] .starting condor daemons Starting Condor daemons: [ OK ] [07:50:37] .wait for condor settled - jobs will be sent [07:50:57] .job[s] running phase entered (total:3 job[s]) [07:50:57] .running job rpm -q condor qpidd condor-7.4.1-0.5.el4 qpidd-0.5.752581-33.el4 [07:51:03] ..passed (ecode:0) [07:51:03] .running job dmesg [07:51:23] ..passed (ecode:0) [07:51:23] .running job free total used free shared buffers cached Mem: 1034580 862424 172156 0 126404 635136 -/+ buffers/cache: 100884 933696 Swap: 2031608 0 2031608 [07:51:43] ..passed (ecode:0) [07:51:46] tst_cnt:16, err_cnt:0, log_segs:3 (wait:2277mili-sec[s], qpidd:0) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:50:23 The NEGOTIATOR (pid 24961) died due to signal 11 (Segmentation fault) [07:51:46] .restart_mode:1,cpids:25162 25183 25193 25194 25195 25196 27055 ,qpid:25138,wait:26645[msec] Stopping Qpid AMQP daemon: [ OK ] [07:51:47] qpidd toggled [07:51:47] waiting: done Starting Qpid AMQP daemon: [ OK ] [07:52:15] qpidd toggled [07:52:15] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:52:16] .waiting: done Stopping Qpid AMQP daemon: [ OK ] [07:52:17] .starting condor daemons Starting Condor daemons: [ OK ] [07:52:19] tst_cnt:17, err_cnt:0, log_segs:3 (wait:1793mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:50:23 The NEGOTIATOR (pid 24961) died due to signal 11 (Segmentation fault) [07:52:19] .restart_mode:4,cpids:27319 27326 ,qpid:,wait:33852[msec] [07:52:19] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:52:33] .waiting: done Starting Qpid AMQP daemon: [ OK ] [07:52:34] .starting condor daemons Starting Condor daemons: [ OK ] [07:52:34] .wait for condor settled - jobs will be sent [07:52:54] .job[s] running phase entered (total:3 job[s]) [07:52:55] .running job uname -a Linux hp-dl360-06.rhts.englab.brq.redhat.com 2.6.9-89.0.16.ELsmp #1 SMP Tue Oct 27 03:49:28 EDT 2009 i686 i686 i386 GNU/Linux [07:53:00] ..passed (ecode:0) [07:53:00] .running job dmesg [07:53:20] ..passed (ecode:0) [07:53:20] .running job uname -a Linux hp-dl360-06.rhts.englab.brq.redhat.com 2.6.9-89.0.16.ELsmp #1 SMP Tue Oct 27 03:49:28 EDT 2009 i686 i686 i386 GNU/Linux [07:53:40] ..passed (ecode:0) [07:53:41] tst_cnt:18, err_cnt:0, log_segs:4 (wait:523mili-sec[s], qpidd:0) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:50:23 The NEGOTIATOR (pid 24961) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:52:21 The NEGOTIATOR (pid 27371) died due to signal 11 (Segmentation fault) [07:53:41] .restart_mode:0,cpids:27572 27593 27603 27604 27605 27606 29460 ,qpid:27548,wait:17902[msec] [07:53:41] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:53:42] .waiting: done [07:53:43] .starting condor daemons Starting Condor daemons: [ OK ] [07:53:43] tst_cnt:19, err_cnt:0, log_segs:4 (wait:405mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:50:23 The NEGOTIATOR (pid 24961) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:52:21 The NEGOTIATOR (pid 27371) died due to signal 11 (Segmentation fault) [07:53:44] .restart_mode:3,cpids:29621 29627 ,qpid:27548,wait:32202[msec] Stopping Qpid AMQP daemon: [ OK ] [07:53:45] qpidd toggled [07:53:45] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:53:56] .waiting: done Starting Qpid AMQP daemon: [ OK ] [07:53:57] .starting condor daemons Starting Condor daemons: [ OK ] [07:53:59] tst_cnt:20, err_cnt:0, log_segs:4 (wait:1600mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:50:23 The NEGOTIATOR (pid 24961) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:52:21 The NEGOTIATOR (pid 27371) died due to signal 11 (Segmentation fault) [07:53:59] .restart_mode:4,cpids:29892 29898 ,qpid:29852,wait:7764[msec] [07:53:59] qpidd stopped (kill-STOP 29852) [07:53:59] .stopping condor daemons Stopping Condor daemons: [ OK ] [07:54:14] .waiting: done Stopping Qpid AMQP daemon: [ OK ] [07:54:15] .starting condor daemons Starting Condor daemons: [ OK ] [07:54:16] tst_cnt:21, err_cnt:0, log_segs:5 (wait:449mili-sec[s], qpidd:1) /var/log/condor/MasterLog:11/11 07:39:20 The NEGOTIATOR (pid 7345) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:39:36 The NEGOTIATOR (pid 7604) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:50:23 The NEGOTIATOR (pid 24961) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:52:21 The NEGOTIATOR (pid 27371) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:54:01 The NEGOTIATOR (pid 29943) died due to signal 11 (Segmentation fault) [07:54:16] .restart_mode:0,cpids:30115 30122 ,qpid:,wait:28059[msec] [07:54:16] .stopping condor daemons Stopping Condor daemons: [ OK ] Warning: condor_master may not have exited, start/restart may fail [07:54:31] .waiting: done [07:54:31] .starting condor daemons Starting Condor daemons: [ OK ] [07:54:31] .wait for condor settled - jobs will be sent [07:54:52] .job[s] running phase entered (total:3 job[s]) [07:54:52] .running job dmesg [07:55:17] ..passed (ecode:0) [07:55:17] .running job free I am quite surprised I cannot get core files from crashing condor daemons. Seeing this: [root@hp-dl360-06 ~]# grep -i "segmentation fault" /var/log/condor/*Log /var/log/condor/MasterLog:11/11 07:20:55 The SCHEDD (pid 9619) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:21:00 The NEGOTIATOR (pid 11589) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:23:46 The SCHEDD (pid 14243) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:23:51 The NEGOTIATOR (pid 15924) died due to signal 11 (Segmentation fault) /var/log/condor/MasterLog:11/11 07:26:18 The NEGOTIATOR (pid 20416) died due to signal 11 (Segmentation fault) [root@hp-dl360-06 ~]# date Wed Nov 11 07:28:15 EST 2009 [root@hp-dl360-06 ~]# condor_config_val -dump |grep CORE CREATE_CORE_FILES = True DETECTED_CORES = 2 UNICORE_GAHP = $(SBIN)/unicore_gahp [root@hp-dl360-06 ~]# cd bz [root@hp-dl360-06 bz]# ls core* ~condor/core.* /var/log/condor/core* ls: core*: No such file or directory ls: /var/lib/condor/core.*: No such file or directory ls: /var/log/condor/core*: No such file or directory I did following steps: ulimit -c unlimited echo "DAEMON_COREFILE_LIMIT='unlimited'" >>/etc/sysconfig/init and [root@hp-dl360-06 bz]# grep CORE ~condor/condor_config.local CREATE_CORE_FILES = True The condor detects it [root@hp-dl360-06 ~]# condor_config_val -dump |grep CORE CREATE_CORE_FILES = True DETECTED_CORES = 2 ... but nothing is generated. This seems like another bug to me. Can someone else check whether my settings above are enough, please? An update: This is both (common) RHEL 4.8 and RHEL 5.4 issue currently RHEL 4.8 i386 / x86_64 and RHEL 5.4 i386 are showing the defect. Retested with corrected core dump options and core files got for both scheduler and negotiator:
negotiator crash
~~~~~~~~~~~~~~~~
GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
...
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
warning: core file may not match specified executable file.
Core was generated by `condor_negotiator -f'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib64/tls/libpthread.so.0...done.
...
Loaded symbols for /usr/lib64/sasl2/libgssapiv2.so.2
#0 std::_Rb_tree<qpid::sys::Mutex*, qpid::sys::Mutex*, std::_Identity<qpid::sys::Mutex*>, std::less<qpid::sys::Mutex*>, std::allocator<qpid::sys::Mutex*> >::insert_unique (this=0x2a95e1e340, __v=@0x41dfb2d8)
at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_function.h:227
227 { return __x < __y; }
(gdb) 4 process 1596 0x0000003526cfd115 in _dl_addr () from /lib64/tls/libc.so.6
3 process 1625 0x0000003527908b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/tls/libpthread.so.0
2 process 1626 0x0000003526cc2f56 in __select_nocancel ()
from /lib64/tls/libc.so.6
* 1 process 1627 std::_Rb_tree<qpid::sys::Mutex*, qpid::sys::Mutex*, std::_Identity<qpid::sys::Mutex*>, std::less<qpid::sys::Mutex*>, std::allocator<qpid::sys::Mutex*> >::insert_unique (this=0x2a95e1e340, __v=@0x41dfb2d8)
at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_function.h:227
(gdb)
Thread 4 (process 1596):
#0 0x0000003526cfd115 in _dl_addr () from /lib64/tls/libc.so.6
#1 0x0000003526cdb798 in backtrace_symbols_fd () from /lib64/tls/libc.so.6
#2 0x00000000004c0288 in dprintf_dump_stack () at dprintf.c:1344
#3 0x00000000004c0487 in sig_backtrace_handler (signum=6)
at dprintf_config.c:62
#4 <signal handler called>
#5 0x0000003526c2e26d in raise () from /lib64/tls/libc.so.6
#6 0x0000003526c2fa6e in abort () from /lib64/tls/libc.so.6
#7 0x00000037d64d26b4 in __tcf_1 () at ./qpid/sys/posix/Mutex.h:112
#8 0x0000003526c30c95 in exit () from /lib64/tls/libc.so.6
#9 0x00000000004a9801 in __wrap_exit (status=0) at daemon_core.cpp:5758
#10 0x00000000004b5044 in DC_Exit (status=0, shutdown_program=0x0)
at daemon_core_main.cpp:314
#11 0x000000000048b97a in main_shutdown_fast () at main.cpp:58
#12 0x00000000004b706c in handle_dc_sigquit () at daemon_core_main.cpp:1443
#13 0x00000000004a2feb in DaemonCore::Driver (this=0x8b4ab0)
at daemon_core.cpp:2879
#14 0x00000000004b88b3 in main (argc=1, argv=0x7fbffffd20)
at daemon_core_main.cpp:2249
Thread 3 (process 1625):
#0 0x0000003527908b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/tls/libpthread.so.0
#1 0x00000037d64e3952 in qpid::client::StateManager::waitFor (this=0x90bf00,
desired=
{_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<int> >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<int> >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<int,int,bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x90cbe0, _M_left = 0x90cc40, _M_right = 0x90cc10}, _M_node_count = 3}}})
at ./qpid/sys/posix/Condition.h:63
#2 0x00000037d6471837 in qpid::client::ConnectionHandler::waitForOpen (
this=0x90be98)
at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/ext/new_allocator.h:69
#3 0x00000037d647da60 in qpid::client::ConnectionImpl::open (this=0x90bdb0)
at qpid/client/ConnectionImpl.cpp:162
#4 0x00000037d646f76e in qpid::client::Connection::open (this=0x2a957c7240,
settings=Variable "settings" is not available.
) at /usr/include/boost/shared_ptr.hpp:252
#5 0x00000037d621ba3c in qpid::management::ManagementAgentImpl::ConnectionThread::run (this=0x2a957c7218) at qpid/agent/ManagementAgentImpl.cpp:795
#6 0x0000002a95986faa in qpid::sys::(anonymous namespace)::runRunnable (
p=0x90bf34) at qpid/sys/posix/Thread.cpp:35
#7 0x0000003527906137 in start_thread () from /lib64/tls/libpthread.so.0
#8 0x0000003526cc9f03 in clone () from /lib64/tls/libc.so.6
Thread 2 (process 1626):
#0 0x0000003526cc2f56 in __select_nocancel () from /lib64/tls/libc.so.6
#1 0x00000000004c2962 in sleep (seconds=1) at sleep.c:36
#2 0x00000037d621d960 in qpid::management::ManagementAgentImpl::PublishThread::run (this=0x2a957c7440) at qpid/agent/ManagementAgentImpl.cpp:924
#3 0x0000002a95986faa in qpid::sys::(anonymous namespace)::runRunnable (p=0x0)
at qpid/sys/posix/Thread.cpp:35
#4 0x0000003527906137 in start_thread () from /lib64/tls/libpthread.so.0
#5 0x0000003526cc9f03 in clone () from /lib64/tls/libc.so.6
Thread 1 (process 1627):
#0 std::_Rb_tree<qpid::sys::Mutex*, qpid::sys::Mutex*, std::_Identity<qpid::sys::Mutex*>, std::less<qpid::sys::Mutex*>, std::allocator<qpid::sys::Mutex*> >::insert_unique (this=0x2a95e1e340, __v=@0x41dfb2d8)
at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_function.h:227
#1 0x00000037d64cfc4c in qpid::client::SaslFactory::mutex_alloc ()
at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_set.h:314
#2 0x0000002a962d74c9 in gssapiv2_client_plug_init ()
from /usr/lib64/sasl2/libgssapiv2.so.2
#3 0x000000352e80746c in sasl_client_add_plugin ()
from /usr/lib64/libsasl2.so.2
#4 0x000000352e810883 in _sasl_load_plugins () from /usr/lib64/libsasl2.so.2
#5 0x000000352e8076ef in sasl_client_init () from /usr/lib64/libsasl2.so.2
#6 0x00000037d64d20fd in qpid::client::SaslFactory::start (this=Variable "this" is not available.
)
at qpid/client/SaslFactory.cpp:122
#7 0x00000037d64d261a in qpid::client::SaslFactory::getInstance ()
at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285
#8 0x00000037d6472554 in qpid::client::ConnectionHandler::start (
this=0x90be98, mechanisms=@0x90e740)
at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:366
#9 0x0000002a95954847 in qpid::framing::AMQP_ClientOperations::ConnectionHandler::Invoker::visit (this=0x41e00fe0, body=@0x90e700)
at gen/qpid/framing/ConnectionStartBody.h:79
#10 0x00000037d6477879 in qpid::framing::invoke<qpid::framing::AMQP_ClientOperations::ConnectionHandler> (target=Variable "target" is not available.
) at ./qpid/framing/Invoker.h:80
#11 0x00000037d6473198 in qpid::client::ConnectionHandler::incoming (
this=0x90be98, frame=@0x41e013c0) at ./qpid/framing/Invoker.h:47
#12 0x00000037d64887c8 in qpid::client::TCPConnector::decode (this=0x90c5e0,
buffer=Variable "buffer" is not available.
) at qpid/client/Connector.cpp:384
#13 0x00000037d6488230 in qpid::client::TCPConnector::readbuff (this=Variable "this" is not available.
)
at qpid/client/Connector.cpp:355
#14 0x00000037d64961cd in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::_mfi::mf2<bool, qpid::client::TCPConnector, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::client::TCPConnector*>, boost::arg<1>, boost::arg<2> > >, bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr=Variable "function_obj_ptr" is not available.
)
at /usr/include/boost/bind/mem_fn_template.hpp:251
#15 0x0000002a95981c07 in boost::function2<bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
this=0x2a95e01210, a0=@0x41dfb2d8, a1=0x0)
at /usr/include/boost/function/function_template.hpp:581
#16 0x0000002a95980b70 in qpid::sys::posix::AsynchIO::readable (this=0x90d640,
h=@0x90d648) at qpid/sys/posix/AsynchIO.cpp:448
#17 0x0000002a95984901 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (
function_obj_ptr=Variable "function_obj_ptr" is not available.
) at /usr/include/boost/bind/mem_fn_template.hpp:149
#18 0x0000002a959eb377 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x2a95e01210,
a0=@0x41dfb2d8) at /usr/include/boost/function/function_template.hpp:581
#19 0x0000002a959e9e76 in qpid::sys::DispatchHandle::processEvent (
this=0x90d648, type=qpid::sys::Poller::WRITABLE)
at qpid/sys/DispatchHandle.cpp:447
#20 0x0000002a95992f3c in qpid::sys::Poller::run (this=0x90c9d0)
at ./qpid/sys/Poller.h:122
#21 0x00000037d64904d9 in qpid::client::TCPConnector::run (this=0x90c5e0)
at qpid/client/Connector.cpp:408
#22 0x0000002a95986faa in qpid::sys::(anonymous namespace)::runRunnable (
p=0x2a95e1e340) at qpid/sys/posix/Thread.cpp:35
#23 0x0000003527906137 in start_thread () from /lib64/tls/libpthread.so.0
#24 0x0000003526cc9f03 in clone () from /lib64/tls/libc.so.6
(gdb)
scheduler crash type A
~~~~~~~~~~~~~~~~~~~~~~~
GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
Copyright 2004 Free Software Foundation, Inc.
...
Core was generated by `condor_schedd -f'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib64/tls/libpthread.so.0...done.
...
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
#0 0x00000000004d1508 in DaemonCore::getpid (this=0x0)
at ./condor_daemon_core.h:923
923 inline pid_t getpid() { return this->mypid; };
(gdb) 4 process 28104 0x000000352790b09d in __lll_mutex_unlock_wake ()
from /lib64/tls/libpthread.so.0
3 process 28131 0x0000003527908b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/tls/libpthread.so.0
2 process 28132 0x0000003526cc2f56 in __select_nocancel ()
from /lib64/tls/libc.so.6
* 1 process 28138 0x00000000004d1508 in DaemonCore::getpid (this=0x0)
at ./condor_daemon_core.h:923
(gdb)
Thread 4 (process 28104):
#0 0x000000352790b09d in __lll_mutex_unlock_wake ()
from /lib64/tls/libpthread.so.0
#1 0x0000003526b15300 in _rtld_local () from /lib64/ld-linux-x86-64.so.2
#2 0x0000000000000000 in ?? ()
Thread 3 (process 28131):
#0 0x0000003527908b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/tls/libpthread.so.0
#1 0x00000037d64a9cb2 in qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop (this=0xa8b8c0, result=@0x409ffbd0, timeout=Variable "timeout" is not available.
)
at ./qpid/sys/posix/Condition.h:63
#2 0x00000037d64a56cf in qpid::client::Dispatcher::run (this=0xa6c4c0)
at ./qpid/sys/BlockingQueue.h:71
#3 0x00000037d621c3ba in qpid::management::ManagementAgentImpl::ConnectionThread::run (this=0x2a957e5218) at /usr/include/boost/shared_ptr.hpp:252
#4 0x0000002a959a4faa in qpid::sys::(anonymous namespace)::runRunnable (
p=0xa8b8ec) at qpid/sys/posix/Thread.cpp:35
#5 0x0000003527906137 in start_thread () from /lib64/tls/libpthread.so.0
#6 0x0000003526cc9f03 in clone () from /lib64/tls/libc.so.6
Thread 2 (process 28132):
#0 0x0000003526cc2f56 in __select_nocancel () from /lib64/tls/libc.so.6
#1 0x000000000056612a in sleep (seconds=1) at sleep.c:36
#2 0x00000037d621d960 in qpid::management::ManagementAgentImpl::PublishThread::run (this=0x2a957e5440) at qpid/agent/ManagementAgentImpl.cpp:924
#3 0x0000002a959a4faa in qpid::sys::(anonymous namespace)::runRunnable (p=0x0)
at qpid/sys/posix/Thread.cpp:35
#4 0x0000003527906137 in start_thread () from /lib64/tls/libpthread.so.0
#5 0x0000003526cc9f03 in clone () from /lib64/tls/libc.so.6
Thread 1 (process 28138):
#0 0x00000000004d1508 in DaemonCore::getpid (this=0x0)
at ./condor_daemon_core.h:923
#1 0x0000000000556ee5 in unix_sigchld () at daemon_core_main.cpp:1280
#2 <signal handler called>
#3 0x0000003526cca2dc in epoll_wait () from /lib64/tls/libc.so.6
#4 0x0000002a959af95c in qpid::sys::Poller::wait (this=0x2a95e01890, timeout=Variable "timeout" is not available.
)
at qpid/sys/epoll/EpollPoller.cpp:439
#5 0x0000002a959b0f07 in qpid::sys::Poller::run (this=0x2a95e01890)
at qpid/sys/epoll/EpollPoller.cpp:405
#6 0x00000037d64904d9 in qpid::client::TCPConnector::run (this=0x2a95e014a0)
at qpid/client/Connector.cpp:408
#7 0x0000002a959a4faa in qpid::sys::(anonymous namespace)::runRunnable (p=0xf)
at qpid/sys/posix/Thread.cpp:35
#8 0x0000003527906137 in start_thread () from /lib64/tls/libpthread.so.0
#9 0x0000003526cc9f03 in clone () from /lib64/tls/libc.so.6
(gdb)
scheduler crash type B
~~~~~~~~~~~~~~~~~~~~~~~
GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
...
Core was generated by `condor_schedd -f'.
Program terminated with signal 11, Segmentation fault.
Loaded symbols for /usr/lib64/libplc4.so
...
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
#0 0x000000352e8084f1 in _sasl_client_mechs () from /usr/lib64/libsasl2.so.2
(gdb) 4 process 20248 0x000000352b00a319 in __deregister_frame ()
from /lib64/libgcc_s.so.1
3 process 20266 0x0000003527908b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/tls/libpthread.so.0
2 process 20267 0x0000003526cc2f56 in __select_nocancel ()
from /lib64/tls/libc.so.6
* 1 process 20268 0x000000352e8084f1 in _sasl_client_mechs ()
from /usr/lib64/libsasl2.so.2
(gdb)
Thread 4 (process 20248):
#0 0x000000352b00a319 in __deregister_frame () from /lib64/libgcc_s.so.1
#1 0x0000003526cfcfa2 in dl_iterate_phdr () from /lib64/tls/libc.so.6
#2 0x000000352b00a592 in _Unwind_Find_FDE () from /lib64/libgcc_s.so.1
#3 0x000000352b007cf3 in _Unwind_FindEnclosingFunction ()
from /lib64/libgcc_s.so.1
#4 0x000000352b008e37 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#5 0x0000003526cdb3df in backtrace () from /lib64/tls/libc.so.6
#6 0x000000000056296d in dprintf_dump_stack () at dprintf.c:1335
#7 0x0000000000562bc7 in sig_backtrace_handler (signum=11)
at dprintf_config.c:62
#8 <signal handler called>
#9 0x0000003526c2fb83 in abort () from /lib64/tls/libc.so.6
#10 0x00000037d64d26b4 in __tcf_1 () at ./qpid/sys/posix/Mutex.h:112
#11 0x0000003526c30c95 in exit () from /lib64/tls/libc.so.6
#12 0x0000000000549de1 in __wrap_exit (status=0) at daemon_core.cpp:5758
#13 0x00000000005551d8 in DC_Exit (status=0, shutdown_program=0x0)
at daemon_core_main.cpp:314
#14 0x00000000004cbb18 in Scheduler::shutdown_fast (this=0x86ae00)
at schedd.cpp:11042
#15 0x00000000004da385 in main_shutdown_fast () at schedd_main.cpp:209
#16 0x0000000000557200 in handle_dc_sigquit () at daemon_core_main.cpp:1443
#17 0x00000000005435cb in DaemonCore::Driver (this=0xa17c80)
at daemon_core.cpp:2879
#18 0x0000000000558a47 in main (argc=1, argv=0x7fbffffd20)
at daemon_core_main.cpp:2249
Thread 3 (process 20266):
#0 0x0000003527908b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/tls/libpthread.so.0
#1 0x00000037d64e3952 in qpid::client::StateManager::waitFor (this=0xa6beb0,
desired=
{_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<int> >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<int> >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<int,int,bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x2a95e00c80, _M_left = 0x2a95e01130, _M_right = 0x2a95e00cb0}, _M_node_count = 3}}}) at ./qpid/sys/posix/Condition.h:63
#2 0x00000037d6471837 in qpid::client::ConnectionHandler::waitForOpen (
this=0xa6be48)
at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/ext/new_allocator.h:69
#3 0x00000037d647da60 in qpid::client::ConnectionImpl::open (this=0xa6bd60)
at qpid/client/ConnectionImpl.cpp:162
#4 0x00000037d646f76e in qpid::client::Connection::open (this=0x2a957e5240,
settings=Variable "settings" is not available.
) at /usr/include/boost/shared_ptr.hpp:252
#5 0x00000037d621ba3c in qpid::management::ManagementAgentImpl::ConnectionThread::run (this=0x2a957e5218) at qpid/agent/ManagementAgentImpl.cpp:795
#6 0x0000002a959a4faa in qpid::sys::(anonymous namespace)::runRunnable (
p=0xa6bee4) at qpid/sys/posix/Thread.cpp:35
#7 0x0000003527906137 in start_thread () from /lib64/tls/libpthread.so.0
#8 0x0000003526cc9f03 in clone () from /lib64/tls/libc.so.6
Thread 2 (process 20267):
#0 0x0000003526cc2f56 in __select_nocancel () from /lib64/tls/libc.so.6
#1 0x000000000056612a in sleep (seconds=1) at sleep.c:36
#2 0x00000037d621d960 in qpid::management::ManagementAgentImpl::PublishThread::run (this=0x2a957e5440) at qpid/agent/ManagementAgentImpl.cpp:924
#3 0x0000002a959a4faa in qpid::sys::(anonymous namespace)::runRunnable (p=0x0)
at qpid/sys/posix/Thread.cpp:35
#4 0x0000003527906137 in start_thread () from /lib64/tls/libpthread.so.0
#5 0x0000003526cc9f03 in clone () from /lib64/tls/libc.so.6
Thread 1 (process 20268):
#0 0x000000352e8084f1 in _sasl_client_mechs () from /usr/lib64/libsasl2.so.2
#1 0x000000352e80ad3d in _sasl_build_mechlist () from /usr/lib64/libsasl2.so.2
#2 0x000000352e80776e in sasl_client_init () from /usr/lib64/libsasl2.so.2
#3 0x00000037d64d20fd in qpid::client::SaslFactory::start (this=Variable "this" is not available.
)
at qpid/client/SaslFactory.cpp:122
#4 0x00000037d64d261a in qpid::client::SaslFactory::getInstance ()
at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285
#5 0x00000037d6472554 in qpid::client::ConnectionHandler::start (
this=0xa6be48, mechanisms=@0xa6e420)
at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:366
#6 0x0000002a95972847 in qpid::framing::AMQP_ClientOperations::ConnectionHandler::Invoker::visit (this=0x41e00fe0, body=@0xa6e3e0)
at gen/qpid/framing/ConnectionStartBody.h:79
#7 0x00000037d6477879 in qpid::framing::invoke<qpid::framing::AMQP_ClientOperations::ConnectionHandler> (target=Variable "target" is not available.
) at ./qpid/framing/Invoker.h:80
#8 0x00000037d6473198 in qpid::client::ConnectionHandler::incoming (
this=0xa6be48, frame=@0x41e013c0) at ./qpid/framing/Invoker.h:47
#9 0x00000037d64887c8 in qpid::client::TCPConnector::decode (this=0xa6c540,
buffer=Variable "buffer" is not available.
) at qpid/client/Connector.cpp:384
#10 0x00000037d6488230 in qpid::client::TCPConnector::readbuff (this=Variable "this" is not available.
)
at qpid/client/Connector.cpp:355
#11 0x00000037d64961cd in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::_mfi::mf2<bool, qpid::client::TCPConnector, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::client::TCPConnector*>, boost::arg<1>, boost::arg<2> > >, bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr=Variable "function_obj_ptr" is not available.
)
at /usr/include/boost/bind/mem_fn_template.hpp:251
#12 0x0000002a9599fc07 in boost::function2<bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
this=0x2a964e1320, a0=@0x40, a1=0x2a95e03960)
at /usr/include/boost/function/function_template.hpp:581
#13 0x0000002a9599eb70 in qpid::sys::posix::AsynchIO::readable (this=0xa6d650,
h=@0xa6d658) at qpid/sys/posix/AsynchIO.cpp:448
#14 0x0000002a959a2901 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (
function_obj_ptr=Variable "function_obj_ptr" is not available.
) at /usr/include/boost/bind/mem_fn_template.hpp:149
#15 0x0000002a95a09377 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x2a964e1320, a0=@0x40)
at /usr/include/boost/function/function_template.hpp:581
#16 0x0000002a95a07e76 in qpid::sys::DispatchHandle::processEvent (
this=0xa6d658, type=qpid::sys::Poller::READABLE)
at qpid/sys/DispatchHandle.cpp:447
#17 0x0000002a959b0f3c in qpid::sys::Poller::run (this=0xa6c930)
at ./qpid/sys/Poller.h:122
#18 0x00000037d64904d9 in qpid::client::TCPConnector::run (this=0xa6c540)
at qpid/client/Connector.cpp:408
#19 0x0000002a959a4faa in qpid::sys::(anonymous namespace)::runRunnable (
p=0x1c9f0) at qpid/sys/posix/Thread.cpp:35
#20 0x0000003527906137 in start_thread () from /lib64/tls/libpthread.so.0
#21 0x0000003526cc9f03 in clone () from /lib64/tls/libc.so.6
(gdb)
This recent backtraces look very much related to BZ528015, a suspected side-effect of QMF_DELETE_ON_SHUTDOWN=FALSE I've submitted 10000 jobs "sleep 100", wait until jobs have submitted correctly. After that I tried "condor_q -better" and before end of this command, I've restart it by "service condor restart" and I can see condor_schedd segmentation fault in Schedlog: 11/16 05:11:44 (pid:26360) Got SIGQUIT. Performing fast shutdown. 11/16 05:11:44 (pid:26360) Sent signal 9 to shadow [pid 27888] for job 1.3 11/16 05:11:44 (pid:26360) Sent signal 9 to shadow [pid 27804] for job 1.10 11/16 05:11:44 (pid:26360) Sent signal 9 to shadow [pid 27928] for job 1.1 11/16 05:11:44 (pid:26360) Sent signal 9 to shadow [pid 27899] for job 1.0 11/16 05:11:44 (pid:26360) All shadows have been killed, exiting. 11/16 05:11:44 (pid:26360) **** condor_schedd (condor_SCHEDD) pid 26360 EXITING WITH STATUS 0 11/16 05:11:44 (pid:26360) Canceling request for claim slot1@ <ip:49567> for condor@ 1.2 Stack dump for process 26360 at timestamp 1258366304 (18 frames) condor_schedd(dprintf_dump_stack+0xb7)[0x5820a1] condor_schedd[0x58230e] /lib64/libpthread.so.0[0x3d85a0e7c0] condor_schedd(_ZN10DaemonCore13Cancel_SocketEP6Stream+0x65)[0x56131b] condor_schedd(_ZN20TransferQueueRequestD1Ev+0x2f)[0x520323] condor_schedd(_ZN20TransferQueueManagerD1Ev+0x48)[0x5209d2] condor_schedd(_ZN9SchedulerD1Ev+0x7b7)[0x4db867] condor_schedd[0x4f2210] /lib64/libc.so.6(exit+0xe5)[0x3d852333a5] condor_schedd(__wrap_exit+0x28)[0x55dd08] condor_schedd(_Z16global_dc_sinfulv+0x0)[0x577966] condor_schedd(_ZN9Scheduler13shutdown_fastEv+0x157)[0x4ca157] condor_schedd(_Z18main_shutdown_fastv+0x1d)[0x4f2253] condor_schedd(_Z17handle_dc_sigquitP7Servicei+0x59)[0x573ee1] condor_schedd(_ZN10DaemonCore6DriverEv+0x7ac)[0x55ecfe] condor_schedd(main+0x1917)[0x577261] /lib64/libc.so.6(__libc_start_main+0xf4)[0x3d8521d994] condor_schedd[0x4c4eb9] Masterlog: 11/16 05:11:44 Got SIGQUIT. Performing fast shutdown. 11/16 05:11:44 Sent SIGQUIT to COLLECTOR (pid 26344) 11/16 05:11:44 Sent SIGQUIT to NEGOTIATOR (pid 26359) 11/16 05:11:44 Sent SIGQUIT to SCHEDD (pid 26360) 11/16 05:11:44 Sent SIGQUIT to STARTD (pid 26361) 11/16 05:11:44 The COLLECTOR (pid 26344) exited with status 0 11/16 05:11:44 The NEGOTIATOR (pid 26359) exited with status 0 11/16 05:11:44 The SCHEDD (pid 26360) died due to signal 11 (Segmentation fault) 11/16 05:12:14 The STARTD (pid 26361) exited with status 0 11/16 05:12:14 All daemons are gone. Exiting. 11/16 05:12:14 **** condor_master (condor_MASTER) pid 26340 EXITING WITH STATUS 0 corefile stacktrace: GNU gdb Fedora (6.8-37.el5) Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"... (no debugging symbols found) Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpcre.so.0 Reading symbols from /usr/lib64/libgsoapssl++.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libgsoapssl++.so.0 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /lib64/libssl.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /usr/lib64/libclassad.so.0... (no debugging symbols found)...done. Loaded symbols for /usr/lib64/libclassad.so.0 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/ld-linux-x86-64.so.2... (no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libz.so.1 Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libsepol.so.1 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /usr/lib64/condor/plugins/MgmtScheddPlugin-plugin.so... (no debugging symbols found)...done. Loaded symbols for /usr/lib64/condor/plugins/MgmtScheddPlugin-plugin.so Reading symbols from /usr/lib64/libqpidcommon.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libqpidcommon.so.0 Reading symbols from /usr/lib64/libqpidclient.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libqpidclient.so.0 Reading symbols from /usr/lib64/libqmfagent.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libqmfagent.so.0 Reading symbols from /usr/lib64/libboost_program_options.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libboost_program_options.so.2 Reading symbols from /usr/lib64/libboost_filesystem.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libboost_filesystem.so.2 Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /usr/lib64/libsasl2.so.2... (no debugging symbols found)...done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_dns.so.2 Core was generated by `condor_schedd -f'. Program terminated with signal 11, Segmentation fault. [New process 26360] [New process 26404] [New process 26403] #0 0x00000000004e96ec in DaemonCore::getpid () (gdb) thread apply all bt full Thread 3 (process 26403): #0 0x0000003d852cced2 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x0000000000585be2 in sleep () No symbol table info available. #2 0x00002ac39c3f8b44 in qpid::management::ManagementAgentImpl::ConnectionThread::run () from /usr/lib64/libqmfagent.so.0 No symbol table info available. #3 0x00002ac39be3cbca in ?? () from /usr/lib64/libqpidcommon.so.0 No symbol table info available. #4 0x0000003d85a064a7 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x0000003d852d3c2d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 2 (process 26404): #0 0x0000003d852cced2 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x0000000000585be2 in sleep () No symbol table info available. #2 0x00002ac39c3f9fb2 in qpid::management::ManagementAgentImpl::PublishThread::run () from /usr/lib64/libqmfagent.so.0 No symbol table info available. #3 0x00002ac39be3cbca in ?? () from /usr/lib64/libqpidcommon.so.0 No symbol table info available. #4 0x0000003d85a064a7 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x0000003d852d3c2d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 1 (process 26360): #0 0x00000000004e96ec in DaemonCore::getpid () No symbol table info available. #1 0x0000000000573dc7 in unix_sigchld () No symbol table info available. #2 <signal handler called> No symbol table info available. #3 0x0000003d85a0b8b5 in pthread_sigmask () from /lib64/libpthread.so.0 No symbol table info available. #4 0x0000000000582361 in ?? () No symbol table info available. #5 <signal handler called> No symbol table info available. #6 0x000000000056131b in DaemonCore::Cancel_Socket () No symbol table info available. #7 0x0000000000520323 in TransferQueueRequest::~TransferQueueRequest () No symbol table info available. #8 0x00000000005209d2 in TransferQueueManager::~TransferQueueManager () No symbol table info available. #9 0x00000000004db867 in Scheduler::~Scheduler () No symbol table info available. #10 0x00000000004f2210 in ?? () No symbol table info available. #11 0x0000003d852333a5 in exit () from /lib64/libc.so.6 No symbol table info available. #12 0x000000000055dd08 in __wrap_exit () No symbol table info available. #13 0x0000000000577966 in DC_Exit () No symbol table info available. #14 0x00000000004ca157 in Scheduler::shutdown_fast () No symbol table info available. #15 0x00000000004f2253 in main_shutdown_fast () No symbol table info available. #16 0x0000000000573ee1 in handle_dc_sigquit () No symbol table info available. #17 0x000000000055ecfe in DaemonCore::Driver () No symbol table info available. #18 0x0000000000577261 in main () No symbol table info available. I think this is "scheduler crash type A" as Frantisek described it above. Is this reproducible with: QMF_DELETE_ON_SHUTDOWN=TRUE QMF_DELETE_ON_SHUTDOWN=TRUE is back to being the default in builds after 7.4.1-0.7.1. The underlying issue has been errata'd. Such as 7.4.2-0.1 Created attachment 405940 [details]
The issue reproducer - new qpid package name
Verfied on RHEL5:
$CondorVersion: 7.4.3 Mar 29 2010 BuildID: RH-7.4.3-0.8.el5 PRE-RELEASE $
$CondorPlatform: I386-LINUX_RHEL5 $
[06:35:52] tst_cnt:99, err_cnt:0, log_segs:0 (wait:360mili-sec[s], qpidd:0)
$CondorVersion: 7.4.3 Mar 29 2010 BuildID: RH-7.4.3-0.8.el5 PRE-RELEASE $
$CondorPlatform: X86_64-LINUX_RHEL5 $
[07:01:31] tst_cnt:99, err_cnt:0, log_segs:0 (wait:2773mili-sec[s], qpidd:0)
Script from Frantisek was patched to use new qpid package name. See attachment.
Waiting for RHEL4 condor packages.
Verified over current condor and qmf plugins on RHEL4:
condor-7.4.4-0.4.el4
python-qmf-0.7.946106-5.el4
qpid-cpp-client-ssl-0.7.946106-6.el4
qmf-0.7.946106-6.el4
python-qpid-0.7.946106-4.el4
qpid-java-common-0.7.946106-5.el4
qpid-cpp-server-devel-0.7.946106-6.el4
qpid-tests-0.7.946106-1.el4
qmf-devel-0.7.946106-6.el4
rh-tests-distribution-MRG-Grid-grid_test_segfault_condor_with_qmf_bz534073-1.0-3
qpid-cpp-client-0.7.946106-6.el4
qpid-dotnet-0.4.738274-2.el4
qpid-tools-0.7.946106-6.el4
condor-kbdd-7.4.4-0.4.el4
condor-debuginfo-7.4.4-0.4.el4
qpid-cpp-server-0.7.946106-6.el4
qpid-cpp-client-devel-0.7.946106-6.el4
qpid-cpp-server-store-0.7.946106-6.el4
qpid-cpp-server-ssl-0.7.946106-6.el4
condor-qmf-7.4.4-0.4.el4
qpid-cpp-mrg-debuginfo-0.7.946106-6.el4
condor-test-7.4.4-0.4.el4
qpid-java-client-0.7.946106-5.el4
qpid-cpp-server-xml-0.7.946106-6.el4
$CondorVersion: 7.4.4 Jun 30 2010 BuildID: RH-7.4.4-0.4.el4 PRE-RELEASE $
$CondorPlatform: X86_64-LINUX_RHEL4
$CondorVersion: 7.4.4 Jun 30 2010 BuildID: RH-7.4.4-0.4.el4 PRE-RELEASE $
$CondorPlatform: I386-LINUX_RHEL4 $
>>> VERIFIED
Regression found in packages: condor-7.4.4-0.9.el4 condor-qmf-7.4.4-0.9.el4 qmf-0.7.946106-11.el4 RHEL4/i386: root 7917 1.0 0.1 51468 8116 ? Rsl 08:26 0:00 condor_master -pidfile /var/run/condor/condor_master.pid [08:27:03] pstack 7917 Thread 2 (Thread -1232077920 (LWP 7953)): #0 0x006fb7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x007e160e in epoll_wait () from /lib/tls/libc.so.6 #2 0x00f2ba60 in qpid::sys::Poller::wait () from /usr/lib/libqpidcommon.so.3 #3 0x00f2cb26 in qpid::sys::Poller::run () from /usr/lib/libqpidcommon.so.3 #4 0x00f1e4d1 in qpid::sys::(anonymous namespace)::runRunnable () #5 0x0097a5cc in start_thread () from /lib/tls/libpthread.so.0 #6 0x007e0f8e in clone () from /lib/tls/libc.so.6 RHEL4/x86_64: condor 3166 0.1 0.8 93436 8480 ? Dsl 08:18 0:00 condor_master -pidfile /var/run/condor/condor_master.pid condor 3189 1.3 0.0 0 0 ? Ds 08:18 0:01 [condor_collecto] [08:20:11] pstack 3166 Thread 3 (Thread 1094719840 (LWP 3187)): #0 0x00000030c8bc2dd6 in __select_nocancel () from /lib64/tls/libc.so.6 #1 0x0000000000485503 in sleep () #2 0x00000030c9f230c0 in qpid::management::ManagementAgentImpl::PublishThread::run () from /usr/lib64/libqmf.so.2 #3 0x00000030c935ffda in qpid::sys::(anonymous namespace)::runRunnable () #4 0x00000030c9006317 in start_thread () from /lib64/tls/libpthread.so.0 #5 0x00000030c8bc9d83 in clone () from /lib64/tls/libc.so.6 Thread 2 (Thread 1105209696 (LWP 3188)): #0 0x00000030c8bca15c in epoll_wait () from /lib64/tls/libc.so.6 #1 0x00000030c936b315 in qpid::sys::Poller::wait () #2 0x00000030c936be7b in qpid::sys::Poller::run () #3 0x00000030c935ffda in qpid::sys::(anonymous namespace)::runRunnable () #4 0x00000030c9006317 in start_thread () from /lib64/tls/libpthread.so.0 #5 0x00000030c8bc9d83 in clone () from /lib64/tls/libc.so.6 Retest once more on latest MRG-M: qpid-tests-0.7.946106-1.el4 qpid-cpp-server-ssl-0.7.946106-12.el4 condor-qmf-7.4.4-0.9.el4 qpid-cpp-client-devel-0.7.946106-12.el4 python-qmf-0.7.946106-9.el4 qpid-cpp-server-0.7.946106-12.el4 python-qpid-0.7.946106-12.el4 qpid-cpp-server-store-0.7.946106-12.el4 qpid-cpp-server-devel-0.7.946106-12.el4 qpid-java-client-0.7.946106-7.el4 qpid-tools-0.7.946106-8.el4 rh-tests-distribution-MRG-Grid-grid_test_segfault_condor_with_qmf_bz534073-1.0-3 qpid-cpp-client-ssl-0.7.946106-12.el4 qpid-cpp-client-devel-docs-0.7.946106-12.el4 qmf-devel-0.7.946106-12.el4 qpid-java-common-0.7.946106-7.el4 qmf-0.7.946106-12.el4 qpid-cpp-mrg-debuginfo-0.7.946106-11.el4 qpid-cpp-client-0.7.946106-12.el4 qpid-cpp-server-xml-0.7.946106-12.el4 Thread 3 (Thread -1219654752 (LWP 15186)): #0 0x006fb7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x007d99d1 in ___newselect_nocancel () from /lib/tls/libc.so.6 #2 0x080c2e34 in sleep () #3 0x008c691f in qpid::management::ManagementAgentImpl::PublishThread::run () #4 0x00f12f11 in qpid::sys::(anonymous namespace)::runRunnable () #5 0x0097a5cc in start_thread () from /lib/tls/libpthread.so.0 #6 0x007e0f8e in clone () from /lib/tls/libc.so.6 Thread 2 (Thread -1232077920 (LWP 15188)): #0 0x006fb7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x007e160e in epoll_wait () from /lib/tls/libc.so.6 #2 0x00f204a0 in qpid::sys::Poller::wait () from /usr/lib/libqpidcommon.so.3 #3 0x00f21566 in qpid::sys::Poller::run () from /usr/lib/libqpidcommon.so.3 #4 0x00f12f11 in qpid::sys::(anonymous namespace)::runRunnable () #5 0x0097a5cc in start_thread () from /lib/tls/libpthread.so.0 #6 0x007e0f8e in clone () from /lib/tls/libc.so.6 How to reproduce: 1. install RHEL4, MRG 2. configure qmf plugins 3. service qpidd start 4. service condor start 5. kill -STOP qpidd 6. service condor stop The message from init script followed: Stopping Condor daemons: [ OK ] Warning: condor_master may not have exited, start/restart may fail The condor_master will stay alive forever. The problem is 100% reproducible. The issue is tracked as bug 625450 pstack for a stuck master on mrg42... Thread 4 (Thread -1208812640 (LWP 9924)): #0 0x0030e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x0058fef6 in pthread_cond_wait@@GLIBC_2.3.2 () #2 0x0029a0ed in qpid::client::StateManager::waitFor () #3 0x00238115 in qpid::client::ConnectionHandler::close () #4 0x00240fc9 in qpid::client::ConnectionImpl::close () #5 0x00230bdc in qpid::client::Connection::close () #6 0x0016143e in qpid::management::ManagementAgentImpl::ConnectionThread::run #7 0x0096ff11 in qpid::sys::(anonymous namespace)::runRunnable () #8 0x0058d5cc in start_thread () from /lib/tls/libpthread.so.0 #9 0x003f3f8e in clone () from /lib/tls/libc.so.6 Thread 3 (Thread -1219302496 (LWP 9925)): #0 0x0030e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x0058fef6 in pthread_cond_wait@@GLIBC_2.3.2 () #2 0x00291f10 in qpid::client::SessionImpl::waitForCompletionImpl () #3 0x00292081 in qpid::client::SessionImpl::waitForCompletion () #4 0x00270976 in qpid::client::Future::wait () #5 0x002305e5 in qpid::client::Completion::wait () #6 0x00225387 in qpid::client::no_keyword::Session_0_10::messageTransfer () #7 0x0015447d in qpid::management::ManagementAgentImpl::ConnectionThread::sendMessage () from /usr/lib/libqmf.so.2 #8 0x0015522f in qpid::management::ManagementAgentImpl::ConnectionThread::sendBuffer () from /usr/lib/libqmf.so.2 #9 0x0015e44d in qpid::management::ManagementAgentImpl::sendHeartbeat () #10 0x0015f788 in qpid::management::ManagementAgentImpl::periodicProcessing () #11 0x001608fa in qpid::management::ManagementAgentImpl::PublishThread::run () #12 0x0096ff11 in qpid::sys::(anonymous namespace)::runRunnable () #13 0x0058d5cc in start_thread () from /lib/tls/libpthread.so.0 #14 0x003f3f8e in clone () from /lib/tls/libc.so.6 Thread 2 (Thread -1229792352 (LWP 9927)): #0 0x0030e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x003f460e in epoll_wait () from /lib/tls/libc.so.6 #2 0x0097d4a0 in qpid::sys::Poller::wait () from /usr/lib/libqpidcommon.so.3 #3 0x0097e566 in qpid::sys::Poller::run () from /usr/lib/libqpidcommon.so.3 #4 0x0096ff11 in qpid::sys::(anonymous namespace)::runRunnable () #5 0x0058d5cc in start_thread () from /lib/tls/libpthread.so.0 #6 0x003f3f8e in clone () from /lib/tls/libc.so.6 Comment 14-19 were tracked as different issue in BZ625450, which is now verified. This should be marked as verified as described in comment #13. >>> VERIFIED |
Description of problem: There is pretty easy way how to crash sheduler / negotiator, using 'dmesg' job as described below. Interesting is that no core file (even is enabled) is generated runuser -s /bin/bash -l condor -c "echo -e 'Executable = /bin/dmesg\nUniverse = vanilla\nLog = /tmp/job4.log\nOutput = /tmp/job4.out\nQueue' | condor_submit" or runuser -s /bin/bash -l condor -c "condor_run $(which dmesg)" causes shedd coredump messages, but no real core file got even if ulimit -c unlimited DAEMON_COREFILE_LIMIT='unlimited' and condor specific: CREATE_CORE_FILES = True given/set-up. Observed on 8 core RHEL 4.8 x86_64: [06:39:09] get_cpu_info():CPU information: processor : 0 1 2 3 4 5 6 7 vendor_id : AuthenticAMD model name : Quad-Core AMD Opteron(tm) Processor 2376 cpu MHz : 800.000 cpu cores : 4 bogomips : 4592.42 4588.53 4588.56 4588.49 4588.47 4588.38 4588.42 4588.55 Feel free to push for 1.3 if needed. Version-Release number of selected component (if applicable): [root@mrg-qe-03 bz]# rpm -qa | egrep '(condor|qmf|qpid|rhm)' | sort -u condor-7.4.1-0.4.el4 condor-debuginfo-7.4.1-0.4.el4 condor-qmf-plugins-7.4.1-0.4.el4 condor-remote-configuration-1.0-23.el4 python-qpid-0.5.760500-6.el4 qmf-0.5.752581-32.el4 qmf-devel-0.5.752581-32.el4 qpidc-0.5.752581-32.el4 qpidc-debuginfo-0.5.752581-32.el4 qpidc-devel-0.5.752581-32.el4 qpidc-ssl-0.5.752581-32.el4 qpidd-0.5.752581-32.el4 qpidd-acl-0.5.752581-32.el4 qpidd-devel-0.5.752581-32.el4 qpid-dotnet-0.4.738274-2.el4 qpidd-ssl-0.5.752581-32.el4 qpidd-xml-0.5.752581-32.el4 qpid-java-client-0.5.751061-9.el4 qpid-java-common-0.5.751061-9.el4 rhm-0.5.3206-25.el4 rhm-debuginfo-0.5.3206-23.el4 rhm-docs-0.5.756148-1.el4 How reproducible: 100% Steps to Reproduce: 1. install packages and set-up personal condor 2. start condor services 3. runuser -s /bin/bash -l condor -c "condor_run $(which dmesg)" or eventually runuser -s /bin/bash -l condor -c "echo -e 'Executable = /bin/dmesg\nUniverse = vanilla\nLog = /tmp/job4.log\nOutput = /tmp/job4.out\nQueue' | condor_submit" Actual results: There is dump of the segfaults printed on stdout by condor or kernel/glibc, job not successfully executed. Expected results: Job should succeed, no segfaults Additional info: [root@mrg-qe-03 bz]# runuser -s /bin/bash -l condor -c "echo -e 'Executable = /bin/dmesg\nUniverse = vanilla\nLog = /tmp/job4.log\nOutput = /tmp/job4.out\nQueue' | condor_submit" Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 102. [root@mrg-qe-03 bz]# runuser -s /bin/bash -l condor -c "condor_run $(which dmesg)" condor_schedd[2714]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 00000000409fec10 error 4 condor_schedd[4214]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea40 error 4 condor_schedd[22628]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01420 error 4 condor_schedd[6190]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[6497]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[8558]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[9007]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[9716]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[11056]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[11370] general protection rip:3526c2fb83 rsp:7fbffff1c0 error:0 condor_schedd[11669]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[12424]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[12724]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[13467]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[14460]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[15235]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[15850]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[15994]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[16603]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[19268]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[19424]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[20635]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[20775]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[21249]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 00000000409ff160 error 4 condor_schedd[22254]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[23585] general protection rip:3526c2fb83 rsp:7fbffff1f0 error:0 condor_schedd[24186]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[28968]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[29100]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[30441]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[30570]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[30889]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[31169]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[32234]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01420 error 4 condor_schedd[32664]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[352]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffe930 error 4 condor_schedd[518]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea80 error 4 condor_schedd[668]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[2325]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[2748]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[3223]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[3484] general protection rip:3526c2fb83 rsp:7fbffff1f0 error:0 condor_schedd[3975]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[5477]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[5758]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[9069]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[9213]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[10425]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[11742]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[13364] general protection rip:3526c2fb83 rsp:7fbffff1f0 error:0 condor_schedd[13678]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[13956]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[15938]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[17283]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[19056]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[19358]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[19799]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[20275]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[20855]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[21133]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[21727]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[22622]: segfault at 000000000000000a rip 0000003526c688fa rsp 0000007fbffff0a0 error 4 condor_schedd[23509]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[23669]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[23819]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[24994]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[25022]: segfault at 0000002a96204320 rip 000000352e8084f1 rsp 0000000041e00be0 error 4 condor_schedd[25887]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[27219]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[28590]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[29058]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[30250]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[30845]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[30996]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[31500]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01420 error 4 condor_schedd[32227]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[1876]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[2313]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[2450]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[3635]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[3785]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[3957]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[4134]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01420 error 4 condor_schedd[5357]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[5925]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[6202]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[6363]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[6829]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[7260]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[7719]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 00000000409ff160 error 4 condor_schedd[8130]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[8741]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[8769]: segfault at 0000002a95d498a0 rip 000000352e8084f1 rsp 0000000041e00be0 error 4 condor_schedd[9057]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01420 error 4 condor_schedd[9487]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[9656]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[10099]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[10663]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[11280] general protection rip:3526c2fb83 rsp:7fbffff1c0 error:0 condor_schedd[13498]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[14130]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[14605]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[14913]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[15356]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[17414]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[18901]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[19509]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[19936]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[21517]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[21813]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[22884]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea40 error 4 condor_schedd[25283]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[25560]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[27645]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[28694]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[29112]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[29840]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[3072]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[4892]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[5061] general protection rip:3526c2fb83 rsp:7fbffff1f0 error:0 condor_schedd[5361]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[5510]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[6835]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[8226]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[8363]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[8514]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[9845]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[11798]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[12826]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[14335]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[14768]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[18488]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[19525]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[22054]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[22367]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[23409]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[24139]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[24294]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[25261]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[25944]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[28804]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[3572]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[4924]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[6420]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[7303]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[9226]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[9783]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[9938]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000007fbfffea70 error 4 condor_schedd[12656]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 condor_schedd[31758]: segfault at 0000000000000110 rip 00000000004d14a6 rsp 0000000041e01430 error 4 audit(1257860173.093:2): avc: granted { setenforce } for pid=9606 comm="setenforce" scontext=root:system_r:unconfined_t tcontext=system_u:object_r:security_t tclass=security [root@mrg-qe-03 bz]# rpm -qa | egrep '(condor|qmf|qpid|rhm)' | sort -u condor-7.4.1-0.4.el4 condor-debuginfo-7.4.1-0.4.el4 condor-qmf-plugins-7.4.1-0.4.el4 condor-remote-configuration-1.0-23.el4 python-qpid-0.5.760500-6.el4 qmf-0.5.752581-32.el4 qmf-devel-0.5.752581-32.el4 qpidc-0.5.752581-32.el4 qpidc-debuginfo-0.5.752581-32.el4 qpidc-devel-0.5.752581-32.el4 qpidc-ssl-0.5.752581-32.el4 qpidd-0.5.752581-32.el4 qpidd-acl-0.5.752581-32.el4 qpidd-devel-0.5.752581-32.el4 qpid-dotnet-0.4.738274-2.el4 qpidd-ssl-0.5.752581-32.el4 qpidd-xml-0.5.752581-32.el4 qpid-java-client-0.5.751061-9.el4 qpid-java-common-0.5.751061-9.el4 rhm-0.5.3206-25.el4 rhm-debuginfo-0.5.3206-23.el4 rhm-docs-0.5.756148-1.el4 [root@mrg-qe-03 bz]# uname -a Linux mrg-qe-03.lab.eng.brq.redhat.com 2.6.9-89.0.16.ELsmp #1 SMP Tue Oct 27 03:51:52 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux MasterLog contains this: ... 11/10 07:20:43 Sent SIGQUIT to NEGOTIATOR (pid 14582) 11/10 07:20:43 Sent SIGQUIT to SCHEDD (pid 14583) 11/10 07:20:43 Sent SIGQUIT to STARTD (pid 14584) 11/10 07:20:43 The COLLECTOR (pid 14487) exited with status 0 11/10 07:20:43 The SCHEDD (pid 14583) exited with status 0 11/10 07:20:43 The NEGOTIATOR (pid 14582) died due to signal 11 (Segmentation fault) 11/10 07:20:48 attempt to connect to <10.34.33.56:9618> failed: Connection refused (connect errno = 111). 11/10 07:20:48 ERROR: SECMAN:2004:Failed to create security session to <10.34.33.56:9618> with TCP.|SECMAN:2003:TCP connection to <10.34.33.56:96 18> failed. 11/10 07:20:48 Failed to start non-blocking update to <10.34.33.56:9618>. 11/10 07:20:55 The STARTD (pid 14584) exited with status 0 11/10 07:20:55 All daemons are gone. Exiting. ...