Bug 507957

Description of problem:

schedd configured with MAX_JOBS_RUNNING=200
500 concurrent DAG jobs submitted

However the condor_dagman jobs count against the total jobs running. So, it seems schedd will happily accept the 500 dag submissions and THEN realize it is busted well after those dags have submitted their own jobs. Thus, schedd gets stuck.

Version-Release number of selected component (if applicable):
$CondorVersion: 7.3.2 Jun  8 2009 BuildID: RH-7.3.2-0.2.el5 PRE-RELEASE-UWCS $
$CondorPlatform: X86_64-LINUX_RHEL5 $

How reproducible:
100%

Steps to Reproduce:
1. log into ha-schedd (mrg27) as bigmonkey
2. cd dagman
3. ./dag_driver.sh
4. wait for 500 submits to complete
5. condor_q -dag until it summary stops updating running, idle count
  
Actual results:

Job status doesn't change:
[16:08:03][bigmonkey@mrg27:~/dagman]$ condor_q -dag | tail -1
678 jobs; 178 idle, 500 running, 0 held
[16:09:19][bigmonkey@mrg27:~/dagman]$ condor_q -dag | tail -1
678 jobs; 178 idle, 500 running, 0 held

Job outputs doesn't change:
[15:57:04][bigmonkey@mrg27:~/dagman]$ ls -1 /tmp/dag_test/out/* | wc -l
1350
[16:05:41][bigmonkey@mrg27:~/dagman]$ ls -1 /tmp/dag_test/out/* | wc -l
1350

Expected results:

Dunno...schedd puts new top-level dagmans into Held until under limit? Maybe introduce a configurable buffer amount which kicks in and holds new jobs as we approach % of limit?

Additional info: