Bug 505923 - dedicated scheduler may be inappropriately reusing claims
dedicated scheduler may be inappropriately reusing claims
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: condor (Show other bugs)
1.1.1
All Linux
medium Severity medium
: 1.3
: ---
Assigned To: Erik Erlandson
Lubos Trilety
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-06-14 14:27 EDT by Matthew Farrellee
Modified: 2010-10-14 12:11 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, claims were not released after a parallel universe job finished, because claim re-use did not handle concurrency limits properly. With this update, all concurrency limits of jobs can be checked.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-10-14 12:11:35 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Matthew Farrellee 2009-06-14 14:27:30 EDT
Description of problem:

From condor-users -

	My test Pool: 1 Dedicated Schedd,2 Startd
	I set a concurrency limit in Negotiator Config "license1_LIMIT=2".
	Then I submit 3 parallel jobs, each job requests 2 slots "machine_count = 2":
		first: concurrency_limits=license1:3
		second: concurrency_limits=license1:2
		third: concurrency_limits=license1:3

	First job could not run  because the concurrency limits exceed,and I removed first job from schedd,the second job started to run,but after the 2nd job completed,the 3rd job started running !!!.
	When setting NEGOTIATOR_DEBUG to D_FULLDEBUG, I found sth wrong in logs, after 2nd job completed ,the SCHEDD would not communicate with NEGOTIATOR, and concurrency limits of jobs could not be checked.


Version-Release number of selected component (if applicable):

condor 7.2
Comment 1 Erik Erlandson 2010-05-21 18:07:09 EDT
Claims do not appear to be released after a parallel universe job finishes.  After my parallel job completed, my slots remained in 'claimed' state.   These claims blocked execution of non-parallel job, but the slots were reusable by another parallel job.

[eje@rorschach ~]$ condor_status

Name               OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime

slot1@rorschach.lo LINUX      X86_64 Claimed   Idle     0.360   951  0+00:01:13
slot2@rorschach.lo LINUX      X86_64 Claimed   Idle     0.000   951  0+00:01:14
                     Total Owner Claimed Unclaimed Matched Preempting Backfill

        X86_64/LINUX     2     0       2         0       0          0        0

               Total     2     0       2         0       0          0        0
Comment 2 Erik Erlandson 2010-05-25 18:47:43 EDT
(In reply to comment #1)
> Claims do not appear to be released after a parallel universe job finishes. 
> After my parallel job completed, my slots remained in 'claimed' state.

This behavior is intended, and governed by config parameter UNUSED_CLAIM_TIMEOUT.

Problem seems to be that claim re-use is not properly handling concurrency limits.  In the repro example, third job should not be eligible since it exceeds concurrency limits.
Comment 3 Erik Erlandson 2010-06-11 16:54:32 EDT
pushed a fix to branch: V7_4-BZ505923-Ded-Schedd-Concurrency-Limits-branch
Comment 4 Lubos Trilety 2010-07-28 09:32:39 EDT
Tested with (version):
condor-7.4.4-0.4

Test Scenario:
  Test Pool: 1 Dedicated Schedd,1 Startd
  Set a concurrency limit in Negotiator Config "license1_LIMIT=2".
1. Submit 3 parallel jobs, each job requests 2 slots "machine_count = 1":
   first: concurrency_limits=license1:3
   second: concurrency_limits=license1:2
   third: concurrency_limits=license1:3
2. First job could not run because the concurrency limits exceed.
3. Remove first job. Second job started to run.
4. After the 2nd job completed, check that the 3rd job could not run because the concurrency limits exceed (see logs).


Tested on:
RHEL4 x86_64  - passed
RHEL4 i386    - passed
RHEL5 x86_64  - passed
RHEL5 i386    - passed

>>> VERIFIED
Comment 5 Florian Nadge 2010-10-07 13:06:08 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, claims were not released after a parallel universe job finished, because claim re-use  did not handle concurrency limits properly. With this update, all concurrency limits of jobs can be checked.
Comment 7 errata-xmlrpc 2010-10-14 12:11:35 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html

Note You need to log in before you can comment on or make changes to this bug.