Bug 491038 - No jobs processed when Negotiator reporting: Over submitter resource limit (0) ... only consider startd ranks
No jobs processed when Negotiator reporting: Over submitter resource limit (0...
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: condor (Show other bugs)
1.1
All Linux
low Severity medium
: 1.2
: ---
Assigned To: Matthew Farrellee
ppecka
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-03-18 23:09 EDT by Charlie Wyse
Modified: 2010-10-23 04:24 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-03-16 10:31:43 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Charlie Wyse 2009-03-18 23:09:42 EDT
Description of problem:
After enabling dynamic provisioning we ran into an issue where machines with resources were not taking jobs.  It looked like our user was hitting a resource limit.

Version-Release number of selected component (if applicable):
condor-7.2.2-0.8.el5

How reproducible:
Very

Steps to Reproduce:
1. Submit a few hundred jobs.
2. Watch them idle.
3. Turn on full debugging in Negotiator log and see the resource limit error
  
Actual results:
jobs just sitting around and doing nothing.

Expected results:
jobs running about, free and happy.

Additional info:
Found release notes saying this was fixed in the UW version.
http://www.cs.wisc.edu/condor/manual/v7.2/8_5Stable_Release.html
Fixed a problem in the condor_negotiator in which machines go unassigned when user priorities result in the machines getting split into shares that are rounded down to 0. For example if there are 10 machines and 100 equal priority submitters, then each submitter was getting 0.1 machines, which got rounded down to 0, so no machines were assigned to anybody. The message in the condor_negotiator log in this case was this:

Over submitter resource limit (0) ... only consider startd ranks
Comment 1 Charlie Wyse 2009-03-18 23:10:35 EDT
After setting the following config variable on our negotiator everything began to work as normal.
NEGOTIATOR_IGNORE_USER_PRIORITIES = True
Comment 2 Matthew Farrellee 2009-10-28 16:02:53 EDT
This is believed fixed in the 7.4 series
Comment 4 ppecka 2009-11-19 11:07:37 EST
with condor-7.4.1-0.5.el5

In /var/log/condor/NegotiatorLog there is:
Over submitter resource limit (0.000000, used 0.000000) ... only consider startd ranks


with condor-7.2.2-0.9.el5
In /var/log/condor/NegotiatorLog there is:
Over submitter resource limit (0) ... only consider startd ranks
Comment 6 ppecka 2009-11-24 08:49:34 EST
The issue has been fixed on RHEL 4.8 / 5.4 i386 / x86_64 on packages:

# rpm -qa | grep condor | sort
condor-7.4.1-0.7.el5
condor-qmf-plugins-7.4.1-0.7.el5

-> VERIFIED
Comment 8 Jeff Needle 2010-03-16 10:31:43 EDT
The fix for this bug was included in the MRG 1.2 release.
Comment 9 Jeff Needle 2010-03-16 10:35:37 EDT
The fix for this bug was included in the MRG 1.2 release.

Note You need to log in before you can comment on or make changes to this bug.