Bug 733511 - ODS: new condor_stats-like command-line tool
ODS: new condor_stats-like command-line tool
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: condor-plumage (Show other bugs)
Development
Unspecified Unspecified
high Severity unspecified
: 2.1
: ---
Assigned To: Pete MacKinnon
Lubos Trilety
: FutureFeature, TechPreview
Depends On: 742007
Blocks: 743350 755648
  Show dependency treegraph
 
Reported: 2011-08-25 16:28 EDT by Pete MacKinnon
Modified: 2012-02-07 03:35 EST (History)
4 users (show)

See Also:
Fixed In Version: condor-7.6.5-0.1
Doc Type: Technology Preview
Doc Text:
Previously, existing statistic collection facilities (ViewCollector) used flat files that were only usable through the C++ condor_stats tool. With this update, a new general query tool to access data generated from new Grid ODS facilities and examine statistics from the ODS database store has been developed in Python. Now, users have the means to start working with Grid statistics data in a dynamic language tool. This Technology Preview is only supported on RHEL6 at this time. Technology Preview features are not currently supported under Red Hat Enterprise Linux subscription services, may not be functionally complete, and are generally not suitable for production use. However, these features are included as a customer convenience and to provide the technologies with wider exposure. Customers may find these features useful in non-production environments, and can provide feedback and functionality suggestions prior to their transition to fully supported status. Erratas will be provided for high-priority security issues. During its development additional components of a Technology Preview feature may become available to the public for testing. It is the intention of Red Hat to fully support Technology Preview features in a future release.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-01-23 12:28:49 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Pete MacKinnon 2011-08-25 16:28:51 EDT
ODS will generate raw data in the form of ClassAd documents. These need to be rolled up into the similar reports that can be drawn from condor_stats. The new CLI will likely be a python script that directly queries the ODS (exact collections to be decided).

The new python script will in turn serve as a template for integrating these stats into cumin.
Comment 1 Pete MacKinnon 2011-09-30 08:19:53 EDT
Verification based on info in contrib README using condor_q and condor_stats.
Comment 2 Pete MacKinnon 2011-10-04 16:23:23 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause: Desire for general query tool to examine statistics from new ODS database store.
Consequence: Existing statistic collection facilities (ViewCollector) use flat files that are only usable through C++ condor_stats tool.
Change: Query tool was developed in Python to access data generated from new Grid ODS facilities.
Result: Users have a means to start working with Grid statistics data in a dynamic language tool.
Comment 5 Lubos Trilety 2011-10-13 08:22:15 EDT
Tested with:
condor-plumage-7.6.4-0.7

-- path to plumage_stats.py is not in $PATH, cannot be run without full path

-- missing pymongo in dependencies
# /usr/share/condor/plumage_stats.py
Traceback (most recent call last):
  File "/usr/share/condor/plumage_stats.py", line 24, in <module>
    import pymongo
ImportError: No module named pymongo

-- missing python-dateutil in dependencies
# /usr/share/condor/plumage_stats.py
Traceback (most recent call last):
  File "/usr/share/condor/plumage_stats.py", line 29, in <module>
    from dateutil import *
ImportError: No module named dateutil

-- missing DAEMON_LIST in configuration, VIEW_SERVER didn't start

-- not created POOL_HISTORY_DIR
# cat /var/log/condor/VieServerLog
...
10/13/11 13:33:15 ERROR "POOL_HISTORY_DIR (/var/lib/condor/ViewHist) does not exist.
" at line 184 in file /builddir/build/BUILD/condor-7.6.3/src/condor_collector.V6/view_server.cpp
Stack dump for process 12636 at timestamp 1318505595 (14 frames)
condor_collector(dprintf_dump_stack+0x44)[0x8113134]
condor_collector[0x8138027]
[0xd32400]
[0xd32416]
/lib/libc.so.6(gsignal+0x51)[0x3b9af1]
/lib/libc.so.6(abort+0x17a)[0x3bb3ca]
condor_collector(_EXCEPT_+0xb2)[0x811d022]
condor_collector(_ZN10ViewServer6ConfigEv+0x282)[0x80a55d2]
condor_collector(_ZN15CollectorDaemon4InitEv+0x6b)[0x809bd4b]
condor_collector(_ZN10ViewServer4InitEv+0x62)[0x80a58b2]
condor_collector(_Z9main_initiPPc+0x35)[0x80acd55]
condor_collector(main+0x13b5)[0x80c9a35]
/lib/libc.so.6(__libc_start_main+0xe6)[0x3a5ce6]
condor_collector[0x809aa91]

-- bad path to plugin in configuration
# cat /var/log/condor/VieServerLog
...
10/13/11 13:36:51 Failed to load plugin: /usr/libexec/condor/ODSCollectorPlugin-plugin.so reason: /usr/libexec/condor/ODSCollectorPlugin-plugin.so: cannot open shared object file: No such file or directory
...

it wants plugin in libexec  directory via configuration
- VIEW_SERVER.PLUGINS = $(VIEW_SERVER.PLUGINS) $(LIBEXEC)/ODSCollectorPlugin-plugin.so
but in the package it is at
- /usr/lib/condor/plugins/ODSCollectorPlugin-plugin.so


-- maybe I am doing something wrong but I didn't get any data
1. submit some jobs
2. see user and resource list
# /usr/share/condor/plumage_stats.py --rl
slot10@host
slot11@host
slot12@host
...
# /usr/share/condor/plumage_stats.py --ul
a.a.user1@host
a.a.user2@host
b.user@host
condor_user@host
3. query some resource or user (there are no data)
# /usr/share/condor/plumage_stats.py --u a.a.user1@host -v
from:	2011-10-13 14:55:54.155729
to:	2011-10-13 15:55:54.155751
# /usr/share/condor/plumage_stats.py --r slot1@host

>>> ASSIGNED
Comment 7 Lubos Trilety 2011-10-17 10:36:20 EDT
-- not created POOL_HISTORY_DIR (created file not directory)
# cat /var/log/condor/VieServerLog
...
10/17/11 17:09:45 ERROR "POOL_HISTORY_DIR (/var/lib/condor/ViewHist) does not exist.
" at line 184 in file /builddir/build/BUILD/condor-7.6.3/src/condor_collector.V6/view_server.cpp
Stack dump for process 3862 at timestamp 1318864185 (14 frames)
condor_collector(dprintf_dump_stack+0x44)[0x815c924]
condor_collector[0x8129ba7]
[0x535400]
[0x535416]
/lib/libc.so.6(gsignal+0x51)[0xbaeaf1]
/lib/libc.so.6(abort+0x17a)[0xbb03ca]
condor_collector(_EXCEPT_+0xb2)[0x8117d92]
condor_collector(_ZN10ViewServer6ConfigEv+0x282)[0x80a90a2]
condor_collector(_ZN15CollectorDaemon4InitEv+0x6b)[0x809bd4b]
condor_collector(_ZN10ViewServer4InitEv+0x62)[0x80a9382]
condor_collector(_Z9main_initiPPc+0x35)[0x80a46c5]
condor_collector(main+0x13b5)[0x80affe5]
/lib/libc.so.6(__libc_start_main+0xe6)[0xb9ace6]

# ls -l /var/lib/condor/ViewHist
-rw-r--r--. 1 root root 0 Oct 15 00:45 /var/lib/condor/ViewHist

-- the script count with locale time, but in mongodb there's time in utc
-- if I have localized time +1 hour and more, I didn't get any result if I run script without --f parameter
-- script should be also working with utc time
1. submit some jobs
2. see user list
# /usr/share/condor/plumage_stats.py --ul
a.a.user1@host
a.a.user2@host
b.user@host
condor_user@host
3. query some user (there are no data)
# /usr/share/condor/plumage/plumage_stats.py --u a.a.user1 -v
from:	2011-10-17 17:17:48.588095
to:	2011-10-17 18:17:48.588115
[root@lt-rhel6_32 ~]# /usr/share/condor/plumage/plumage_stats.py --u a.a.user1 -v --f='2011-10-17 16:17'
from:	2011-10-17 16:17
to:	2011-10-17 18:17:55.083687
a.a.user1@lt-rhel6_32.mrg-qe-12.lab.eng.brq.redhat.com 	2011-10-17 16:17:32.506000 	0 	0 	0

>>> ASSIGNED
Comment 8 Pete MacKinnon 2011-10-18 18:38:37 EDT
Spec file will be updated to correct missing ViewHist directory.
plumage_stats.py script updated to make appropriate UTC adjustments - tested on CEST server.
Comment 9 Pete MacKinnon 2011-10-24 10:18:35 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,4 +1,10 @@
 Cause: Desire for general query tool to examine statistics from new ODS database store.
 Consequence: Existing statistic collection facilities (ViewCollector) use flat files that are only usable through C++ condor_stats tool.
 Change: Query tool was developed in Python to access data generated from new Grid ODS facilities.
-Result: Users have a means to start working with Grid statistics data in a dynamic language tool.+Result: Users have a means to start working with Grid statistics data in a dynamic language tool.
+
+Technology Preview features are not currently supported under Red Hat Enterprise Linux subscription services, may not be functionally complete, and are generally not suitable for production use. However, these features are included as a customer convenience and to provide the technologies with wider exposure.
+
+Customers may find these features useful in non-production environments, and can provide feedback and functionality suggestions prior to their transition to fully supported status. Erratas will be provided for high-priority security issues.
+
+During its development additional components of a Technology Preview feature may become available to the public for testing. It is the intention of Red Hat to fully support Technology Preview features in a future release.
Comment 10 Pete MacKinnon 2011-10-24 10:19:38 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -3,6 +3,8 @@
 Change: Query tool was developed in Python to access data generated from new Grid ODS facilities.
 Result: Users have a means to start working with Grid statistics data in a dynamic language tool.
 
+This Technology Preview is only supported on RHEL6 at this time.
+
 Technology Preview features are not currently supported under Red Hat Enterprise Linux subscription services, may not be functionally complete, and are generally not suitable for production use. However, these features are included as a customer convenience and to provide the technologies with wider exposure.
 
 Customers may find these features useful in non-production environments, and can provide feedback and functionality suggestions prior to their transition to fully supported status. Erratas will be provided for high-priority security issues.
Comment 12 Tomas Capek 2011-11-17 11:50:27 EST
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,7 +1,4 @@
-Cause: Desire for general query tool to examine statistics from new ODS database store.
-Consequence: Existing statistic collection facilities (ViewCollector) use flat files that are only usable through C++ condor_stats tool.
-Change: Query tool was developed in Python to access data generated from new Grid ODS facilities.
-Result: Users have a means to start working with Grid statistics data in a dynamic language tool.
+Previously, existing statistic collection facilities (ViewCollector) used flat files that were only usable through the C++ condor_stats tool. With this update, a new general query tool to access data generated from new Grid ODS facilities and examine statistics from the ODS database store has been developed in Python. Now, users have the means to start working with Grid statistics data in a dynamic language tool.
 
 This Technology Preview is only supported on RHEL6 at this time.
Comment 13 Lubos Trilety 2011-11-23 09:05:08 EST
Tested with:
condor-plumage-7.6.5-0.7

Tested on:
RHEL6 i386, x86_64

New condor_stats-like command-line tool (plumage_stats) is available and
functional.

>>> VERIFIED
Comment 14 errata-xmlrpc 2012-01-23 12:28:49 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-0045.html

Note You need to log in before you can comment on or make changes to this bug.