Description of problem: schedd configured with MAX_JOBS_RUNNING=200 500 concurrent DAG jobs submitted However the condor_dagman jobs count against the total jobs running. So, it seems schedd will happily accept the 500 dag submissions and THEN realize it is busted well after those dags have submitted their own jobs. Thus, schedd gets stuck. Version-Release number of selected component (if applicable): $CondorVersion: 7.3.2 Jun 8 2009 BuildID: RH-7.3.2-0.2.el5 PRE-RELEASE-UWCS $ $CondorPlatform: X86_64-LINUX_RHEL5 $ How reproducible: 100% Steps to Reproduce: 1. log into ha-schedd (mrg27) as bigmonkey 2. cd dagman 3. ./dag_driver.sh 4. wait for 500 submits to complete 5. condor_q -dag until it summary stops updating running, idle count Actual results: Job status doesn't change: [16:08:03][bigmonkey@mrg27:~/dagman]$ condor_q -dag | tail -1 678 jobs; 178 idle, 500 running, 0 held [16:09:19][bigmonkey@mrg27:~/dagman]$ condor_q -dag | tail -1 678 jobs; 178 idle, 500 running, 0 held Job outputs doesn't change: [15:57:04][bigmonkey@mrg27:~/dagman]$ ls -1 /tmp/dag_test/out/* | wc -l 1350 [16:05:41][bigmonkey@mrg27:~/dagman]$ ls -1 /tmp/dag_test/out/* | wc -l 1350 Expected results: Dunno...schedd puts new top-level dagmans into Held until under limit? Maybe introduce a configurable buffer amount which kicks in and holds new jobs as we approach % of limit? Additional info:
Perhaps this is intended to be managed by combinations of? -maxidle <number> (Maximum number of idle nodes to allow) -maxjobs <number> (Maximum number of jobs ever submitted at once) Will experiment...
Examining the submit code, -maxidle/-maxjobs provides no relief since they only count per dagman. Have to look at schedd to figure out if there is a way the submit client can get the "overall" picture at time of submission.
New BZ created (526480) will promote workaround of capacity planning prior to large multi-dag deployments and setting an appropriately high MAX_JOBS_RUNNING (estimate # of concurrent dags X # of nodes for largest dag). Next level of analysis on this BZ will focus on why schedd appears to not be doing bookkeeping of ALL jobs (dagman+nodes). Will also try to solicit input/opinions from UW.
Referenced upstream at http://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=642. Ultimate resolution in that ticket. Write a KB article that will be given to Lana for reference in next user Guide. Consult Mike Cressman or John Thomas for KB article tips.
KB article http://kbase.redhat.com/faq/docs/DOC-33345 submitted to SME jthomas for tech review