Bug 707771

Summary:	RFE: option for condor_userprio to report acctgrp stats hierarchically
Product:	Red Hat Enterprise MRG	Reporter:	Erik Erlandson <eerlands>
Component:	condor	Assignee:	Erik Erlandson <eerlands>
Status:	CLOSED ERRATA	QA Contact:	Lubos Trilety <ltrilety>
Severity:	low	Docs Contact:
Priority:	low
Version:	Development	CC:	jthomas, ltrilety, matt, mkudlej, tstclair
Target Milestone:	2.1	Keywords:	FutureFeature
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	condor-7.6.4-0.5	Doc Type:	Enhancement
Doc Text:	With this update, two new options, "-grouporder" and "-grouprollup", have been added to the condor_userprio utility to allow accounting group hierarchies to be displayed and group statistics to be rolled up by hierarchy, respectively.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-01-23 17:26:58 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	743350

Description Erik Erlandson 2011-05-25 21:35:34 UTC

Description of problem:
With the new hierarchical accounting group support (#1393), users may find it useful to have condor_userprio report stats such that stats for a group include stats for its children (subtree). For example, if the usage for groups "a","a.b","a.c" are 1, 2, 3 respectively, the usage reported for "a" would be 6. (Currently, condor_userprio reports statistics for accounting groups independently of any children). Proposal is to provide condor_userprio with an option to report stats rolled up by accounting group hierarchy.


Expected results:
ability to roll up stats hierarchically with condor_userprio

Additional info:
upstream https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=1926

Comment 1 Erik Erlandson 2011-09-26 19:00:31 UTC

Available on tracking branch:
UPSTREAM-7.7.3-BZ707771-userprio-rollup

Comment 2 Erik Erlandson 2011-09-26 19:06:20 UTC

REPRO/TEST

This is a new feature, and so there is no repro condition.

Consider the following accounting group configuration, with a simple hierarchy:

NEGOTIATOR_DEBUG = D_FULLDEBUG
NEGOTIATOR_INTERVAL = 30
SCHEDD_INTERVAL	= 15
NEGOTIATOR_USE_SLOT_WEIGHTS = FALSE
GROUP_QUOTA_MAX_ALLOCATION_ROUNDS = 1

NUM_CPUS = 10

GROUP_NAMES = a, a.b, a.c
GROUP_QUOTA_a = 10
GROUP_QUOTA_a.b = 5
GROUP_QUOTA_a.c = 5
GROUP_ACCEPT_SURPLUS = TRUE

Submit the following to populate the acct groups with some jobs:

universe = vanilla
cmd = /bin/sleep
args = 6000
should_transfer_files = if_needed
when_to_transfer_output = on_exit
+AccountingGroup="a.c.u3"
queue 1
+AccountingGroup="a.b.u1"
queue 1
+AccountingGroup="a.b.u2"
queue 1

Wait for the jobs to spin up, and test output of condor_userprio. The updated usage statement shows two new options "-grouporder" and "-grouprollup"

$ condor_userprio -h
usage: condor_userprio [ -pool hostname ] [ -all | -usage | { -setprio | -setfactor | -setaccum | -setbegin | -setlast }  user value | -resetusage user | -resetall | -getreslist user | -delete user ] [-allusers | -activefrom month day year] [-l] [-grouprollup] [-grouporder]

Legacy default behavior is preserved (sorting by prio, no rollup, etc):

$ condor_userprio -all -allusers
Last Priority Update:  9/22 11:36
                                    Effective   Real     Priority   Res   Total Usage       Usage            Last
User Name                           Priority  Priority    Factor    Used (wghted-hrs)    Start Time       Usage Time
------------------------------      --------- -------- ------------ ----  ----------- ---------------- ----------------
<none>                                   0.50     0.50         1.00    0         0.00 12/31/1969 17:00        ???
a                                        0.50     0.50         1.00    0         0.00 12/31/1969 17:00        ???
a.c                                      0.50     0.50         1.00    1         0.02  9/22/2011 11:35  9/22/2011 11:36
a.c.u3@localdomain                       0.50     0.50         1.00    1         0.02  9/22/2011 11:35  9/22/2011 11:36
a.b.u2@localdomain                       0.50     0.50         1.00    1         0.02  9/22/2011 11:35  9/22/2011 11:36
a.b.u1@localdomain                       0.50     0.50         1.00    1         0.02  9/22/2011 11:35  9/22/2011 11:36
a.b                                      0.50     0.50         1.00    2         0.03  9/22/2011 11:35  9/22/2011 11:36
------------------------------      --------- -------- ------------ ----  ----------- ---------------- ----------------
Number of users: 3                                                     3         0.05  9/22/2011 11:35        ???

Using the new "-grouporder" will place acct groups at top of output, in breadthfirst order (no rollup). Individual submitters are still sorted by priority as usual:

$ condor_userprio -all -allusers -grouporder
Last Priority Update:  9/22 11:37
                                    Effective   Real     Priority   Res   Total Usage       Usage            Last
User Name                           Priority  Priority    Factor    Used (wghted-hrs)    Start Time       Usage Time
------------------------------      --------- -------- ------------ ----  ----------- ---------------- ----------------
<none>                                   0.50     0.50         1.00    0         0.00 12/31/1969 17:00        ???
a                                        0.50     0.50         1.00    0         0.00 12/31/1969 17:00        ???
a.b                                      0.50     0.50         1.00    2         0.09  9/22/2011 11:35  9/22/2011 11:37
a.c                                      0.50     0.50         1.00    1         0.04  9/22/2011 11:35  9/22/2011 11:37
a.c.u3@localdomain                       0.50     0.50         1.00    1         0.04  9/22/2011 11:35  9/22/2011 11:37
a.b.u2@localdomain                       0.50     0.50         1.00    1         0.04  9/22/2011 11:35  9/22/2011 11:37
a.b.u1@localdomain                       0.50     0.50         1.00    1         0.04  9/22/2011 11:35  9/22/2011 11:37
------------------------------      --------- -------- ------------ ----  ----------- ---------------- ----------------
Number of users: 3                                                     3         0.13  9/22/2011 11:35        ???

With the "-grouprollup" option, acct group stats are rolled up the hierarchy (note that priority values are zeroed out for groups as rolled up priorities are meaningless). Non-group submitters are ordered as usual.

$ condor_userprio -all -allusers -grouporder -grouprollup
Last Priority Update:  9/22 11:41
                                    Effective   Real     Priority   Res   Total Usage       Usage            Last
User Name                           Priority  Priority    Factor    Used (wghted-hrs)    Start Time       Usage Time
------------------------------      --------- -------- ------------ ----  ----------- ---------------- ----------------
<none>                                   0.00     0.00         0.00    3         0.30 12/31/1969 17:00  9/22/2011 11:41
a                                        0.00     0.00         0.00    3         0.30 12/31/1969 17:00  9/22/2011 11:41
a.b                                      0.00     0.00         0.00    2         0.20  9/22/2011 11:35  9/22/2011 11:41
a.c                                      0.00     0.00         0.00    1         0.10  9/22/2011 11:35  9/22/2011 11:41
a.c.u3@localdomain                       0.50     0.50         1.00    1         0.10  9/22/2011 11:35  9/22/2011 11:41
a.b.u2@localdomain                       0.50     0.50         1.00    1         0.10  9/22/2011 11:35  9/22/2011 11:41
a.b.u1@localdomain                       0.50     0.50         1.00    1         0.10  9/22/2011 11:35  9/22/2011 11:41
------------------------------      --------- -------- ------------ ----  ----------- ---------------- ----------------
Number of users: 3                                                     3         0.30  9/22/2011 11:35        ???

Comment 3 Erik Erlandson 2011-09-26 19:06:20 UTC

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause:
Customers wanted to see accounting statistics for accounting groups rolled up by group hierarchy

Consequence:
Some confusion when accountant reported the unrolled statistics used internally by the negotiator and accountant.

Change:
Two new options -grouporder and -grouprollup were added to condor_userprio to allow accounting group hierarchies to be displayed and group statistics to be rolled up by hierarchy, respectively.

Result:
Customers will have the option to view accounting groups and group statistics rolled up by group hierarchy.

Comment 5 Lubos Trilety 2011-10-21 13:17:58 UTC

Tested on:
$CondorVersion: 7.6.4 Oct 14 2011 BuildID: RH-7.6.4-0.8.el5 $
$CondorPlatform: I686-RedHat_5.7 $

$CondorVersion: 7.6.4 Oct 14 2011 BuildID: RH-7.6.4-0.8.el5 $
$CondorPlatform: X86_64-RedHat_5.7 $

$CondorVersion: 7.6.4 Oct 14 2011 BuildID: RH-7.6.4-0.8.el6 $
$CondorPlatform: I686-RedHat_6.1 $

$CondorVersion: 7.6.4 Oct 14 2011 BuildID: RH-7.6.4-0.8.el6 $
$CondorPlatform: X86_64-RedHat_6.1 $

If I run 'condor_userprio (-all) -grouprollup' I get completely the same result as if I run 'condor_userprio (-all) -grouprollup -allusers -grouporder'. Is it OK?

Comment 6 Erik Erlandson 2011-10-21 16:32:39 UTC

> 
> If I run 'condor_userprio (-all) -grouprollup' I get completely the same result
> as if I run 'condor_userprio (-all) -grouprollup -allusers -grouporder'. Is it
> OK?

Yes, that behavior is consistent with the semantics for -grouprollup and -grouporder

Comment 7 Lubos Trilety 2011-10-24 09:28:46 UTC

Tested on:
$CondorVersion: 7.6.4 Oct 14 2011 BuildID: RH-7.6.4-0.8.el5 $
$CondorPlatform: I686-RedHat_5.7 $

$CondorVersion: 7.6.4 Oct 14 2011 BuildID: RH-7.6.4-0.8.el5 $
$CondorPlatform: X86_64-RedHat_5.7 $

$CondorVersion: 7.6.4 Oct 14 2011 BuildID: RH-7.6.4-0.8.el6 $
$CondorPlatform: I686-RedHat_6.1 $

$CondorVersion: 7.6.4 Oct 14 2011 BuildID: RH-7.6.4-0.8.el6 $
$CondorPlatform: X86_64-RedHat_6.1 $

New options work correctly.

>>> VERIFIED

Comment 8 Tomas Capek 2011-11-17 13:01:50 UTC

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,11 +1 @@
-Cause:
+With this update, two new options, "-grouporder" and "-grouprollup", have been added to the condor_userprio utility to allow accounting group hierarchies to be displayed and group statistics to be rolled up by hierarchy, respectively.-Customers wanted to see accounting statistics for accounting groups rolled up by group hierarchy
-
-Consequence:
-Some confusion when accountant reported the unrolled statistics used internally by the negotiator and accountant.
-
-Change:
-Two new options -grouporder and -grouprollup were added to condor_userprio to allow accounting group hierarchies to be displayed and group statistics to be rolled up by hierarchy, respectively.
-
-Result:
-Customers will have the option to view accounting groups and group statistics rolled up by group hierarchy.

Comment 9 errata-xmlrpc 2012-01-23 17:26:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-0045.html