Bug 617709

The easiest way to see this bug is to start a set of jobs that have a few levels of groups..such as a1.a2.a1, a1.a2.a2 and turn on D_ACCOUNTANT. You may want to clean the accountant log.

Let the system spin up and look at the second or later iteration of negotiation (after jobs get matched and run). With the bug you will see something like:


8/03 14:42:51 group a1 dynamic quota for 41 slots = 0.400
08/03 14:42:51 Group Table : group a1 quota 0.400 usage 8.000 prio 2000.00
08/03 14:42:51 negotiationtime: slots 41 group a1 autoregroup true
08/03 14:42:51 group b1 dynamic quota for 41 slots = 0.400
08/03 14:42:51 Group Table : group b1 quota 0.400 usage 9.000 prio 2250.00
08/03 14:42:51 negotiationtime: slots 41 group b1 autoregroup true
08/03 14:42:51 group a1.a2 dynamic quota for 41 slots = 0.200
08/03 14:42:51 Group Table : group a1.a2 quota 0.200 usage 0.000 prio 0.00
08/03 14:42:51 negotiationtime: slots 41 group a1.a2 autoregroup false
08/03 14:42:51 group a1.a1 dynamic quota for 41 slots = 0.200
08/03 14:42:51 Group Table : group a1.a1 quota 0.200 usage 0.000 prio 0.00
08/03 14:42:51 negotiationtime: slots 41 group a1.a1 autoregroup false
08/03 14:42:51 group a1.a3 dynamic quota for 41 slots = 0.600
08/03 14:42:51 Group Table : group a1.a3 quota 0.600 usage 0.000 prio 0.00
08/03 14:42:51 negotiationtime: slots 41 group a1.a3 autoregroup true
08/03 14:42:51 group a1.a3.a1 dynamic quota for 41 slots = 0.400
08/03 14:42:51 Group Table : group a1.a3.a1 quota 0.400 usage 0.000 prio 0.00
08/03 14:42:51 negotiationtime: slots 41 group a1.a3.a1 autoregroup true
08/03 14:42:51 group a1.a3.a2 dynamic quota for 41 slots = 0.200
08/03 14:42:51 Group Table : group a1.a3.a2 quota 0.200 usage 0.000 prio 0.00
08/03 14:42:51 negotiationtime: slots 41 group a1.a3.a2 autoregroup false
08/03 14:42:51 group a1.a3.a3 dynamic quota for 41 slots = 0.200
08/03 14:42:51 Group Table : group a1.a3.a3 quota 0.200 usage 0.000 prio 0.00
08/03 14:42:51 negotiationtime: slots 41 group a1.a3.a3 autoregroup false
08/03 14:42:51 group b1.b2 dynamic quota for 41 slots = 0.600
08/03 14:42:51 Group Table : group b1.b2 quota 0.600 usage 0.000 prio 0.00
08/03 14:42:51 negotiationtime: slots 41 group b1.b2 autoregroup true
08/03 14:42:51 group b1.b1 dynamic quota for 41 slots = 0.400
08/03 14:42:51 Group Table : group b1.b1 quota 0.400 usage 0.000 prio 0.00


Note that only the top tier groups have usage and prio values above zero. That is because the groupname is truncated after the first ".". Hence values for a1.a2.a1 get recorded for a1. In the fixed version you will see usage and prio values for each group name that reflects their own usage/prio.

This is the easiest observable way to see the bug. It has an impact in the addmatch and removematch statistic handling and changes behavior later in negotiation.

Created attachment 436380 [details]
same patch ported to condor-7_4_4-0_4_el5

I ran a config with the following:

#####################
# HFS testing related parameters
#####################
NEGOTIATOR_DEBUG = D_FULLDEBUG | D_ACCOUNTANT
SCHEDD_INTERVAL	= 15
NEGOTIATOR_USE_SLOT_WEIGHTS = FALSE

NUM_CPUS = 100

GROUP_NAMES = a1, b1, a1.a1, a1.a2, a1.a2.a1, a1.a2.a2, b1.b2, b1.b1

GROUP_QUOTA_DYNAMIC_a1 = .4
GROUP_QUOTA_DYNAMIC_b1 = .4
GROUP_QUOTA_DYNAMIC_a1.a1 = .5
GROUP_QUOTA_DYNAMIC_a1.a2 = .5
GROUP_QUOTA_DYNAMIC_b1.b1 = .5
GROUP_QUOTA_DYNAMIC_b1.b2 = .5
GROUP_QUOTA_DYNAMIC_a1.a2.a1 = .5
GROUP_QUOTA_DYNAMIC_a1.a2.a2 = .5




And ran this submission:

universe = vanilla
executable = /bin/sleep
arguments = 10m
+AccountingGroup = "b1.b1.user"
queue 100
+AccountingGroup = "b1.b2.user"
queue 100
+AccountingGroup = "a1.a2.a1.user"
queue 100
+AccountingGroup = "a1.a2.a2.user"
queue 100
+AccountingGroup = "zz_none.user"
queue 100


I let the system spool up for a few iterations, and I'm seeing this:

08/06 16:22:33 Group Table : group a1 quota 0.400 usage 0.000 prio 0.00
08/06 16:22:33 Group Table : group b1 quota 0.400 usage 8.000 prio 2000.00
08/06 16:22:33 Group Table : group a1.a1 quota 0.500 usage 0.000 prio 0.00
08/06 16:22:33 Group Table : group a1.a2 quota 0.500 usage 0.000 prio 0.00
08/06 16:22:33 Group Table : group a1.a2.a1 quota 0.500 usage 10.000 prio 2000.00
08/06 16:22:33 Group Table : group a1.a2.a2 quota 0.500 usage 10.000 prio 2000.00
08/06 16:22:33 Group Table : group b1.b2 quota 0.500 usage 20.000 prio 4000.00
08/06 16:22:33 Group Table : group b1.b1 quota 0.500 usage 20.000 prio 4000.00


I notice that group 'b1' seems to have a nonzero usage (8.0), but 'a1' has a zero usage, which seems inconsistent, because both groups have subgroups with submissions running (and neither has any submissions directly against it).

I'm assuming results are with the patch. They are strange results. Did you nuke the accountant log?

Other than that the only thing I see is perhaps some code needs to be added in Initialize() when it uses "thisUser".

char const *thisUser = &(key[CustomerRecord.Length()]);

I have no idea what form &(key[CustomerRecord.Length()]) is in,  but apparently it's matched against the groupnamelist.  This code doesn't do any filtering or string trimming of thisUser and matching directly against the groupnamelist seems odd.

The older code truncated at the first ".", so I'm not sure why this might produce values for b1, but not a1 or for a1.a2.  Add to that why is b1 usage 8 and b1.b1 and b1.b2 sum to 40. 

I'll have to see if I can repro your results.

well, I can't duplicate your results with the patch. I get the expected "0" value for both b1 and a1.

Verified that patch works when acct log is actually nuked (note to self: acct log lives in 'spool', not 'log')

Pushed branch V7_4-BZ617709-accountant-subgroup-stats to the FH repo.

Reproduced with (version):
condor-7.4.4-0.8

08/19/10 09:25:14 Group Table : group a1 quota 0.400 usage 20.000 prio 5000.00
08/19/10 09:25:14 Group Table : group b1 quota 0.400 usage 40.000 prio 10000.00
08/19/10 09:25:14 Group Table : group a1.a1 quota 0.500 usage 0.000 prio 0.00
08/19/10 09:25:14 Group Table : group a1.a2 quota 0.500 usage 0.000 prio 0.00
08/19/10 09:25:14 Group Table : group a1.a2.a1 quota 0.500 usage 0.000 prio 0.00
08/19/10 09:25:14 Group Table : group a1.a2.a2 quota 0.500 usage 0.000 prio 0.00
08/19/10 09:25:14 Group Table : group b1.b2 quota 0.500 usage 0.000 prio 0.00
08/19/10 09:25:14 Group Table : group b1.b1 quota 0.500 usage 0.000 prio 0.00

Tested with (version):
condor-7.4.4-0.9

Tested on:
RHEL5 i386,x86_64  - passed
RHEL4 i386,x86_64  - passed

>>> VERIFIED

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, HFS group names were handled incorrectly in the accountant. Groupname was parsed from the beginning of the group string leading to some stats being assigned to the wrong group. With this update, the group, to which the customer is submitted against, is parsed from the end of the customername string.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html